It’s hard to tell exactly what’s wrong from your description. Could you capture a video showing the issue? 

First thing to check is that you’re actually getting tracking data from the RealSense into Aximmetry. Next thing is to verify that you’ve connected all the required pins from the Aximmetry tracked compound into the Unreal scene node. 

I am receiving the tracking data (I didn't get a message that it wasn't receiving it, which I do when it's not connected)

I can see that you haven’t connected the pins correctly. The Control Data pin should go to the Control Data pin and not B1 Data. The same applies for the pins below it. 

Hi,

It would be best if you also use the mixed camera compound since that was added to your Unreal project (Green camera (virtual + tracked, 1-3 billboards)):

Or add to the Unreal project the camera that is tracked only (Green camera (tracked, 1-3 billboards)).

Warmest regards,

Hi, 

Thanks so much to the both of you, I couldn't figure out why in videos other people had less pins on the nodes, but now I understand why!

Currently things work really well, except that there is quite a bit of lag/jitter when moving the camera. We tried using the 'detect tracking delay', but its as if it's somehow skipping frames.

It was having some problems with the camera frame rate and project  frame rate not being the same, but after changing that, it looked like this still.

I read somewhere that we should turn on 'manual lens', and that does fix the jitter, but it gives a fish eye effect and the setup inside the studio node moves along with the camera, so that also wasn't the solution.

It's a little hard to describe all this, so I have video footage on this page: 

https://drive.google.com/drive/folders/14a-QLC3pocC91Y76tN8_Fwq3Fq0PoCYu?usp=sharing

Thank you so much in advance to anyone who might know what's going on!!

Hi,

Understanding the function of the Manual Lens setting is essential. It is not related to jitter issues. When the Manual Lens is enabled, the focal length and focus are set by the Manual Zoom and Manual Focus values:

If your setup lacks Zoom tracking and you use only RealSense, activating Manual Lens and adjusting the Manual Zoom is necessary to align the virtual camera's focal length with that of your physical camera. The focal length and FOV (Field of View) values for Manual Zoom can be viewed by enabling the ON SCREEN Info:

Then the FOV/focal length values are displayed at the bottom of the preview video output of the camera compound:

If you have zoom tracking and you receive lens data too from your tracking system that results in fish-eye distortion, then you can turn off the Lens Distortion in the SCENE panel rather than using the Manual Lens:

You should turn on Tracking Dejitter only if absolutely necessary.
Enabling Tracking Dejitter will lock the camera to a fixed position while the camera doesn't move more than Dejitter Pos Thresh and doesn't rotate more than Dejitter Rot Thresh.

The same applies to Camera Delay. You should only set a negative camera delay in very rare cases when the tracking system has a long processing time that results in having an even greater delay than the camera input. In this case what we recommend is:  first try to use a negative Tracking Delay.  If it is not possible (you get "Cannot keep latency" for the tracking device) then use a positive Camera Delay  (or alternatively increase the in-to-out latency, thus keeping the sync between the camera inputs in the case you use multiple ones).

If you have anything red in the ProcessLoad panel, then you may experience all kinds of lags (jitter) because your computer is not fast enough to process everything.

Lastly, the teddy bear toy is not the best choice for Detect Tracking Delay due to its lack of uniform red color, flat surface, and hard edges. Further details on "Detect Tracking Delay" can be found here.

Warmest regards

Thank you very much for the reply Eifert!

It definitely helped, but the jitter is still there. It's almost like it's dropping camera frames.

I added a video in the google doc link: https://drive.google.com/drive/folders/1zpJbHKwLvRxYR58CQGFsHE2oLo-dEDv_?usp=sharing

Also one in slow-motion that shows that sometimes, there are frames where the virtual camera does move but the real-life one is still on the frame it had before. This also happens when we want to do that Detect Tracking Delay. The bear wasn't the best example to record, but even with a small red object, the X is a little jittery and doesn't know what to focus on. 

The framerate from the project and camera are both 30fps, and we use a BlackMagic card to connect the camera to the PC. The CPU and GPU are both also in the green, so we really don't know what could cause this.

For now, we have been able to just remove these 'frozen' frames from the recordings and it moves smoothly, but it would be a bit of a pain to have to do that every time.

Thanks so much for the help in advance!!

(Not sure if it's worth noting, but this effect only occurs when there's an object in the green screen scene. If it's just the virtual camera, it moves smoothly and if the camera stands still and the actor moves in front of the green scree, it's also fine. We did have some issues with the virtual scene randomly moving by itself without the camera moving, but starting up the project again fixed that)

Hi,

It seems you are encountering two distinct issues.

Firstly, the problem with missing frames in your video input compared to the virtual camera's render is highlighted in the JitterSlow.mp4 video. This issue may still stem from differing frame rates. Although you've mentioned that both the camera and the project are set to 30 fps, you should also verify that Aximmetry's INPUT's Camera Mode is set to AUTO:

Additionally, a frequent oversight occurs when one records using the Video Recorder module; many mistakenly set the Frame Rate to something other than Realtime. If you use the Video Recorder module, you should ensure that it is set to Realtime during live recordings like yours:

More on it here: https://aximmetry.com/learn/virtual-production-workflow/preparation-of-the-production-environment-phase-i/inner-workings-of-aximmetry/rendering-settings/#video-recorder-module

Also, keep an eye out for any "missing frame" or "cannot keep latency" messages in the log while recording.

Secondly, the tracking jitter issue depicted in the Jitter1.mp4 video may represent the highest accuracy achievable with Realsense at that particular zoom level. Given my limited experience with Realsense, take this with caution.
Nonetheless, it's worth noting that zooming in tends to magnify tracking inaccuracies in any system, though professional tracking systems generally offer superior precision to Realsense.
The potential for degraded tracking quality with Realsense could also be linked to the camera not being securely affixed or not positioned horizontally. Moreover, a lack of depth variation in the camera's field of view, such as facing a flat wall, could diminish tracking accuracy, as Realsense relies on depth data for positional tracking.

Warmest regards,