LED/Content wall, combined with AR

Content Creation 3D Scene Editing 3D Animation 2D Compositing 2D Animation Augmented Reality Unreal


Hello there.

I am sorry in advance for the strange questions, but iv been wondering how could it be possible to achieve a synchronization between a 3d element projected upon a LED screen, and a AR element which appears as it is coming out from the LED screen into the real world?

The timing and perspective of the appearence of the AR element, and the disappearence of the virtual element on the LED screen, must be perfect.

This would be very interesting since then i could have people in front of the screen and have the element pop out beside them. 

Thank you in advance!


   Blackfish Studio

Profile Image
7/8/2022 12:47:46 PM

Hi Andrei,

If you are using Unreal then you will have to do it in a multimachine setup: https://aximmetry.com/learn/tutorials/for-studio-operators/using-led-walls-and-ar-together/ In this case, you can use OSC to send triggers between the machines. (OSC Out and OSC Method modules)

If the poping out geometry is not moving/changing when it pops out, then you can probably get away with rendering it to the Led Wall and in AR at the same time for a limited time.
If it moves then you could do some kind of animation/effect that will hide any incorrect timing.
Otherwise, you will need to set a delay and start the animations with that delay included.

If you are using a single machine, you can even measure the delay quite easily with a setup similar to this:

This will render the frame numbers on the led wall. Zoom on it with the camera. Set it in the Video Recorder to Video Format PNG and start recording. You can stop recording after about a second.
The recording should have two numbers. Try to change the Delayer's Frames until the two numbers are equal like this: