GPU and CPU overload for no reason

 

Hi there.

In many of our environments after reopening them, the CPU and GPU load are going crazy with no reason. Everything worked before, but for no reason Aximmetry is overloading the GPU and CPU. 
We have RTX 3090 with i9.




And after reloading other project and coming back to this one. The Overload disappeared.  This is very random.  The project in Aximmetry is in 4K 30fps. 
We had this issues in a production and the client was very angry..


   AVRJapan

 
Profile Image
TwentyStudios
  -  

We’re seeing the same thing lately. Happens very randomly, but could be related to NDI. I suspect this started happening after a system or driver update. I’ve seen reports from other users also experiencing the same thing, so there is definitely something strange going on.

 
Profile Image
DeataProduction
  -  

Can report having same issue here, recorded hours of footage at 4k@30fps with no issues or frame drops during the entire time.

Tried reopening the project today to record a few extra bits and pieces and the moment i pressed record my GPU & CPU usage went to almost 1000%

and my fps crashed to 2-4 frames per second, the behavior continued even when i stopped the recording.

Also seems to be totally random, at other times the project just refuses to not grind to a halt even when not recording and it being the same project with exactly the same settings as a few days earlier when it worked completely fine.

Also been noticing that even when not as severe to grind to a halt Aximmetry still sometimes struggles badly with GPU overload and has trouble keeping framerates but when checking GPU usage with HWmonitor its barely hitting 40% while cpu is not even at 20%

Aximmetry shows both as 300-500%


PC Specs being:

Ryzen Threadripper 1950x 4ghz, 128GB 3500mhz DDR4, RTX 3090 And Recording to a m.2 NVME 2tb 5500mb/s 

Video Coming in to aximmetry From a Canon R5 using a Elgato Camlink so the issue also exist when not using NDI

 
Profile Image
Eifert@Aximmetry
  -  

Hi all,

We are investigating this performance issue.

And we apologize for any inconvenience this issue may have caused.

Warmest regards,

 
Profile Image
Eifert@Aximmetry
  -  

Hi all,

Please answer the following questions if your experience this performance issue to help us diagnose the problem:

1. Does it happen with all projects or only specific ones?
2. Does it happen with non-4k projects?
3. Did you use the Unreal engine in the project?
4. Was there any significant change in the system before the issue first occurred? Maybe an Aximmetry update, Windows update, GPU driver update, or hardware change?
5. What is your Aximemtry version?
6. What is your Windows version? (You can find it in Control Panel / Settings / About / Windows Specifications)
7. What is your GPU driver version?
8. What kind of CPU and GPU are you using?

Thank you!

Warmest regards,

 
Profile Image
HumanContact
  -  

I have had this happen a couple times as well. Answers below:

1. Does it happen with all projects or only specific ones?

  • It's only happened twice but on separate projects so seems random.

2. Does it happen with non-4k projects?

  • Yes, 1080 @ 30fps

3. Did you use the Unreal engine in the project?

  • Yes

4. Was there any significant change in the system before the issue first occurred? Maybe an Aximmetry update, Windows update, GPU driver update, or hardware change?

  • Aximmetry update  2021.3.1 > 2021.3.2, possibly a cumulative windows 10 update and Nvidia GPU Studio Driver update

5. What is your Aximemtry version?

  •  2021.3.2

6. What is your Windows version? (You can find it in Control Panel / Settings / About / Windows Specifications)

  • 21H1 - Build 19043.1320

7. What is your GPU driver version?

  • Studio Driver - 472.39

8. What kind of CPU and GPU are you using?

  • AMD Ryzen 9 3900X 12-Core Processor @ 3.80 GHz
  • 2080 Ti

 
Profile Image
cp2carello@gmail.com
  -  

I'm having similar issues. Is there a status update to this issue? thanks

 
Profile Image
Eifert@Aximmetry
  -  

Hi,

Can you describe how the performance overload happens in your case?
There are many things that can affect Aximmetry's performance, you might have a different problem than the people here.

Warmest regards,

 
Profile Image
Eric Soulard
  -  

Hi,

I experimented the same issue with Aximmetry 2021 4.2, running on a HP Z8 G4 workstation (double 6136 XEON, 64 GB DDR4 2666, 3090 RTX, Decklink 8K Pro, NVME drives). Using an Unreal scene with was rated at 200 FPS under Unreal, we ended up setting the Aximmetry rendering settings to 1080p25 during production. Aximmetry was reporting in that case a GPU load around 50% but it was still missing a frame or so from time to time. it was OK for the type of the production but far from expected. Talking with one of the Unreal designer I worked with on this project, he also complained about Aximmetry slowness compared to other Unreal based solution.

Best regards

 
Profile Image
Eifert@Aximmetry
  -  

Hi Eric,

Can you describe how did you do the recording and the detecting of the missing frames?
It will help us to recreate the issue.

Note that you can lose frames when doing real-time recording in Aximmetry and a 3rd party software causes a sudden increase in hardware usage.

About the slowness, cooked Unreal projects in Aximmetry should achieve close to the same frame rate as the executable version of your project. You can get an executable (*.exe) from your project by packaging the Unreal project for Windows, just like how people build games with Unreal. Of course, this is all true if you use the same camera settings and frame size resolution.
Also, a rendered B Mask can cause a visible decrease in that frame rate. B Mask won't be rendered if Allow Virtuals is turned on and Light Wraps are turned off in Aximmetry's Camera compound.

Warmest regards,

 
Profile Image
TwentyStudios
  -  

@Eifert: Even with B Mask off, there is a substantial performance hit when opening the project in Aximmetry compared to running a standalone executable. This is at least true when running the project in 4K. To be clear, this is just opening the project in Aximmetry without using any billboards, keying, light wrap or other things that use up performance. Just opening the project and outputting it to the Decklink. Are you doing tests in 4K? Maybe there is a performance ottleneck that you’re not aware of?

 
Profile Image
Eric Soulard
  -  

Hi Eifert,

We only had one station recording one input. The other were running a little faster (40 % load against 50%). Recording was done on a nvme drive using H264 codec.

I can't tell if allow virtuals were enabled. Lightwrap was.

best regards

 
Profile Image
Eifert@Aximmetry
  -  

Hi TwentyStudios,

In 4k resolution, opening an Unreal project and connecting out its Out output pin resulted in about 38 fps. While it achieved 42 fps when the project was packaged for Windows with normal Unreal Editor. All this was done on relatively weak hardware, Laptop RTX 3060. On better hardware, the difference should be even less at this many fps.
Other than setting up a similar Cine Camera Actor as in Aximmetry's camera, these changes have been made to the level blueprint in both projects (except the CineCameraActor2):

Do you experience similar results?


Warmest regards,

 
Profile Image
Eifert@Aximmetry
  -  

Hi Eric,

You were recording the input with one computer and one other computer was recording the project in Aximmetry?
In the end, you compared the two recording's frame count?
I am sorry for the many questions, but a detailed description can help us recreate this exact situation.

Allow Virtuals or Lightwrap does not matter in the case of the recording. I mentioned these settings because they increase the hardware usage compared to "other Unreal based solution".

Warmest regards,

 
Profile Image
Eric Soulard
  -  

Hi,

We only had one workstation recording one input. The three other ones were just playing out. The video switching was operated from a Newtek Tricaster and the final output of this video switcher was recorded directly on an Hyperdeck video server.

Three workstations were receiving tracking data associated to the cameras connected to their inputs (one camera per chassis).

Best regards

 
Profile Image
Eifert@Aximmetry
  -  

Hi Eric,

Thank you for the detailed description and we are sorry for the inconvenience this issue may have caused.

We will investigate this issue and we will get back to you if we find something.

Warmest regards,

 
Profile Image
Accenture Labs
  -  

Hello,

I'm working at Accenture in Sophia Antipolis and we have the same issue. 

CPU and GPU overload are going crazy in Aximmetry software, perhaps in the task manager CPU, Memory and GPU are under 10% usage.

The Unreal model used runs fine on some laptop (no NDI), and it looks like it's getting crazy on our production workstation where we have 2 Blackmagic 8k pro.

Our production workstation:

AMD Ryzen Threadripper PRO 3995WX (64 cores)

256GB RAM DDR5

2 NVIDIA RTX 3090


Our pre prod laptop

Alienware X15

Intel Core i9

32GB RAM

NVIDIA RTX 3080


Do you have any update on this topic ?

Best Regards,


Gautier

 
Profile Image
TwentyStudios
  -  

@Accenture Labs: A common mistake when comparing performance in the UE4 Editor and Aximmetry is that the project is rendered at a lower resolution in the editor. Your Aximmetry project might be set to render in 4K while the UE4 editor is rendering the viewport at a fraction of that.

Another thing that might cause issues in your particular setup is that you’re running two GPUs and two capture cards. That will use a lot of PCIE bandwidth which can cause overloads that don’t show up in the Task Manager. Aximmetry or UE4 can’t use two GPUs for real-time rendering, so try disabling one of them and see if that improves things. Same with the Decklink. In our tests with two Decklink cards on an Intel i9 it can handle a total of 6 4K30p signals. After that we start to see erratic performance behavior. 

 
Profile Image
Accenture Labs
  -  

Thanks,

I'll try disabling 1 GPU. We are Ingesting 3 Cams 4k60fps and 1 video feed 1080p60, but need additional card to output a 1080p video signal.

I'll let you know how it goes.

Best Regards,

Gautier

 
Profile Image
TwentyStudios
  -  

@Accenture Labs: 3 4K60p will put a significant load on the PCIE bus and will cause a lot of memory transfers that can show up as frame drops and strange performance issues while not showing anything in the Taskmanager. You would probably be better off running several separate workstations with more modest performance than trying to do it all on a single machine. 

 
Profile Image
Accenture Labs
  -  

thanks @TwentyStudios, this helps a lot. I'll do some tests on my end and will let you know on this.

Best Regards,

Gautier

 
Profile Image
hasanfurkan
  -  




 
Profile Image
Ahmed@Aximmetry
  -  

Hello Hasan, 

please post your issue in another post as this thread is already too long and your issue is unrelated.

Best regards,