2023.2.1 skyrockets GPU usage

 

Hi there.

After updating to 2023.2.1 my GPU is pretty much fried.
Axy is reporting 99% usage on our RTX 3080 (which 'translates ' to 44% usage in Windows Taskmanager),
with a framerate of 12.5 FPS, even after increasing Axys In/Out Latency to 10 frames (used to be 3).

All outputs are set to 1080p25, sync is set to one of those outputs as well.
(So Axy shouldn't render at any odd 60hz Monitor refresh rate)
In fact, even starting up Axy already consumes 25% GPU without *any* compound loaded...

The exact same project I'm running here used to consume 60-70% GPU on 2023.1.0

As much as I desperately need the Audio-Offset-Fix, and as much as I would love to finally incorporate the CP3D in our pipeline, I will have to make a decision today if I should keep 2023.2.1 or revert back to 2023.1.0 until the performance issues are fixed.

To add to this, Audio Playback (like music tracks or the like) is absolutely horrible and stuttering.

If there are any other tips or tricks on how to reduce GPU Load in 2023.2.1 I'd be happy to hear about them!

Also, UE5.2 seems to have changed a lot in terms of lighting, the entire scene is heavily overexposed, but that's probably another story.

Cheers & all the best.

Eric.


   EricWest

 
Profile Image
Eifert@Aximmetry
  -  

Hi Eric,

We are investigating the issue. 
Until that, we suggest using GPU HDMI output without the sync.
Note people have the same issue here: https://my.aximmetry.com/post/3135-202320-202321-gpu-hdmi-output-cannot-kee

Warmest regards,

 
Profile Image
EricWest
  -  

Hi Eifert.

Thanks for the info. Unfortunately none of those tricks seem to cut the mustard.
These are our current settings:

Cheers & all the best.

Eric.


 
Profile Image
Eifert@Aximmetry
  -  

Hi Eric,

Do you experience halved frame rate or increased performance issues even when the NDI video output is synced? Like in your screenshot:

Do you have any of the hardware in your computer in Genlock? If so, you could try it without Genlock.

Warmest regards,

 
Profile Image
EricWest
  -  

Hi Eifert.

I tried with and without sync on both the NDI out and the Decklink out.
Basically no noticeable chances at all...
For the time being, I did a rollback to 2023.1.0 (as much as it hurts not having access to the new features!)
But we're back to stable 68% GPU Load (rather than 99% with dropping framerates)

None of the hardware devices receive an external Genlock.

Cheers & all the best.

Eric.


 
Profile Image
Motionworx
  -  

Hi, we have also experienced odd GPU behaviour. With sync enabled, the GPU gets maxed out. Without sync, the GPU performance is still strange. It has a seesaw effect, jumping between 40% and 70% to well over a hundred percent. Version 2023.1.0, same virtual set, hardware and conditions, performed consistently between 40% and 44%. This, combined with the NDI issues we are experiencing, means we'll have to revert as well. 


 
Profile Image
LiveGuyUSA
  -  

Using SYNC with my GPU outputs maxes the GPU as well on the latest version.

 
Profile Image
Eifert@Aximmetry
  -  

Hi,

If you are using NDI for input, we made a quick fix for it.
You need to copy the following file to the same folder as Aximmetry.Composer.exe: https://www.myairbridge.com/en/#!/transfer/8MNGRjl5Fs0/eu1/ML 
So for example if DE was installed to its default location, then you should place the file as this: C:\Program Files\Aximmetry DE\bin\Processing.NDI.Lib.x64.dll

Warmest regards,

 
Profile Image
Eifert@Aximmetry
  -  

Hi,

A beta version is now available that addresses the high GPU load issue. To get access to the beta version, reach out to sales@aximmetry.com. Please refer to this forum post in the mail or let us know that you would like to test the beta version because of the high GPU load fix.

Warmest regards,

 
Profile Image
Eifert@Aximmetry
  -  

Hi,

Aximmetry Version 2023.2.2 has been released today. The new version contains fixes for various monitor output issues and performance load.

You can find the change log here: https://aximmetry.com/learn/software-version-history/#%E2%96%BA-latest-2023-2-2

Warmest regards,

 
Profile Image
deandd
  -  

In version 2.2 I still appear to be having high GPU and CPU load, with or without sync.

Performance coming out of UE5 without the Aximmetry Composer opened is stable. Once I open the Composer, my CPU and GPU load jump up to unusable levels. 

 
Profile Image
TwentyStudios
  -  

@deandd: when comparing performance between running the scene in the UE5 editor and Aximmetry you need to make sure you’re rendering the scene in the same resolution. Unreal will automatically lower Screen Percentage in the editor to keep från rates reasonable and if you’re not running the scene in full screen on a monitor that matches your Aximmetry project resolution you’re not comparing like for like. That being said, there does seem to be something strange going on with the performance load on your system. 

 
Profile Image
Eifert@Aximmetry
  -  

Hi Deandd,

You should make sure you don't have any Unreal Editor open if you are using a cooked project.
Otherwise, we will need a bit more information to locate the source of the problem. Does the high GPU and CPU load happen right at the start or it increases over time? Does it happen in both Cooked and Live sync modes? Do you have GPU and CPU load in a blank (empty) Unreal project?

Warmest regards,

 
Profile Image
deandd
  -  

To provide a bit more information: 

I have tested in a vanilla (non-Aximmetry) Unreal install on the same scene I have, outputting at 4k 30fps to my Blackmagic card. I have no performance issues in this setup.

Performance is perfect on 2023.1. Seemingly, whatever kind of bug that is causing this was introduced in the newer versions.

On both 2023.2.1 and 2023.2.2 I start to experience this issue. Without Unreal open, output through the Blackmagic card I have connected is severely delayed, around 5-8 seconds. While Unreal is open in live sync mode, or while cooked mode is enabled (Unreal Editor closed) I have severe performance issues that do not seem to match up with the quality of my scene. The high GPU and CPU load fluctuates from anywhere from 100% to 1000%. Dropping the resolution scale to 1% in Unreal resulted in no change in performance. I have the same GPU and CPU load in a blank Unreal project. I have tried disabling sync (as mentioned in this thread) but it was unable to fix the performance issue.

 
Profile Image
TwentyStudios
  -  

Have you updated to the latest Decklink and NVidia drivers?

 
Profile Image
Eifert@Aximmetry
  -  

Hi Deandd,

If you have seconds of delay in Aximmetry through the Blackmagic card, then there is something wrong that is not related to Unreal. You should for example have about 8 frames of delay (0.25 seconds at 30 FPS) with the following settings in Edit->Preferences:

Maybe you have an old Decklink driver as TwentyStudios said.

You could also try resetting your Aximmetry settings in the Startup window. You might have some error-prone settings, like having inputs and outputs on the same capture card index slot.
Don't forget to Backup first:



There is almost no way to set rendering resolution in Unreal Editor, other than putting its viewport on full screen on a monitor that is set to your desired resolution.
For example, changing the Screen Percentage in Unreal Editor won't really change the resolution:

Also, there are about 3 other places in the Editor where you can set or affect the screen percentage.


In Aximmetry, you can directly change the Unreal rendering resolution by changing the Out Size pin in your Unreal Project module:

When leaving 0:0 at the Out Size pin, it will render in your Preferences->Rendering->Resolution.


Couple more things you could check:

Check if the issue is Unreal 5.2 related. Aximmetry 2023.1.0 uses Unreal 5.1, while Aximmetry 2023.2.0+ uses Unreal 5.2. If you tested in vanilla Unreal 5.1, then you should also test your project in vanilla Unreal 5.2 and see if the performance load increases.

Check if turning ON Low Latency Mode on the Unreal Project module fixes the performance:

Low Latency Mode was added in Aximmetry 2023.2.0.

You could test the performance in scenes rendered by Aximmetry.
For example in: [Common]:Compounds\Scene\Default_3D_Scene.xcomp and [Studio]:News Room\News Room - VirtualCam_3-Cam.xcomp

And that in the startup, you have your fastest video card selected instead of an integrated video card:

Warmest regards,


 
Profile Image
TwentyStudios
  -  

@Eifert: What are the implications for using “Low Latency Mode” or not? Couldn’t find any documentation on it….

 
Profile Image
deandd
  -  

Resetting my Aximmetry settings and rebuilding them resolved the performance issues I had. Thanks for the help :)

 
Profile Image
Eifert@Aximmetry
  -  

Hi Deandd,

If you made a backup of Aximmetry settings, we would be incredibly grateful if you could kindly send it to sales@aximmetry.com. We might be able to figure out from it which setting caused the issue.

Warmest regards,

 
Profile Image
Eifert@Aximmetry
  -  

Hi TwentyStudios,

There is no documentation about Low Latency Mode yet, but there is a documentation page under work that will include detailed information about Low Latency Mode.
In short, turning on Low Latency Mode gains about 1 frame less delay, while increasing a little bit the CPU load in your hardware. So after turning it on, you might be able to decrease the In-to-out Latency by one frame.

Warmest regards,