Hi,
In a non billboard method, what would be the options to offline render foreground elements of the virtual scene separately? In a scenario where I render files out from Aximmetry and make the final composition in Fusion, it would be important to be able to either render any foreground elements separately or to render a mask/matte of the foreground assets.
In vanilla UE this would be easy, I could just assign any assets to a separate fg layer and render just that layer, is this doable via Aximmetry?
How about render layers like cryptomatte, stencil layers or using something related to object id's?
Any of those solutions would be great as they would not require any modifications to the UE scene that would not also work for rendering the main render/background.
If those doesn't work, it might be easy enough to create a second version of the UE scene where the foreground elements had some kind of matte shader that would allow to easily create a mask out of them in Fusion.
I understand I could export tracking data to fbx, then render in UE and then render that output again in Aximmetry through lens distortion, but that is a little hassle so hope there is a better way.
Optimal would be if it was possible to simultaneously render the fg layer when rendering all the other files like the basic UE bg, 3D cleanplate, any Aximmetry crops/masks etc..But if it needs a separate round of rendering it's not a too big of a deal either.
Thank you,
Emil
Hi Emil,
As of now, you cannot use Unreal Engine's non-real-time rendering capabilities directly from Aximmetry. You have to export the tracking data as an FBX file and use it in, for example, the Movie Render Queue. More details on how to do so can be found here: https://my.aximmetry.com/post/897-camera-tracking-data-fbx-import-fails-on
Note that adding support for Movie Render Queue within Aximmetry is on our request list.
Additionally, you can output lens distortion from Aximmetry as an STMap, which you can use in Fusion to apply lens distortion. To access this feature, locate the STMap compound at [Common]:Compounds\Utilities\STMap_Generator.xcomp.
Here are a couple of ways you can set it up:
Using Only Aximmetry
If you only want to use Aximmetry, then you can record the background and a mask for the billboard to indicate what is in front of or behind the billboard. You could use this mask in Aximmetry to record two videos—one for the foreground and one for the background. Alternatively, you can record the mask and use it in Fusion if possible.
What first you should know, that when when Allow Virtuals is turned off, the camera compound in Aximmetry renders the billboard instead of Unreal.
For this, the Unreal module outputs a mask video (B Mask pin).
For example, the Out video with it's B Mask:
This mask is in the red channel of the video, but you can easily process it to any other color(s). You can use modules like Inverter, Alpha Merger, or Color Composer, or a combination of these to do so. Also, this mask only covers the size (Scale) of your billboard.
You will want to record this Mask, Billboard, and Unreal Rendered Image at a specific point inside the tracked camera compound. This way, you don't have to deal with cropping and other processes. To do this, open the camera compound for editing with the "Edit Compound In-Place" option.
Note, that this compound is part of Aximmetry's package, so you should handle it accordingly: https://aximmetry.com/learn/virtual-production-workflow/preparation-of-the-production-environment-phase-i/scripting-in-aximmetry/flow-editor/compound/#working-with-compounds-from-aximmetry-library
Double click on the camera compound to enter it, and inside find the location where the Masker module uses the mask to hide portions of the Input (Billboard) and where the LightWarp_Layer compound puts the subsequent video over Unreal's rendered image. If you use Aximmetry's Light Warp, you will also probably need to edit this LightWarp_Layer compound.
Use the videos in the connections marked by blue rectangles in the picture below to your liking and create different layers as needed. For example, by using a Masker module not on the Input's video but on the Unreal rendered background video.
One more thing: if on the billboard the Render To Depth is turned on, a billboard will still be rendered by Unreal in the output of the Unreal module.
This is necessary for certain reflection types: https://aximmetry.com/learn/virtual-production-workflow/preparation-of-the-production-environment-phase-i/green-screen-production/aximmetry-de-scene-setup-green-screen/#reflections-in-unreal-engine
However, in most cases, this isn’t a problem because the billboard rendered by Aximmetry will completely overlap it.
Using only Aximmetry with Green + AR camera compound
If you have a specific need to render only some foreground elements separately, the above solution will probably not work. In that case, you could use the Green + AR camera (tracked, 1-3 billboards) compound, where the AR part would only render the foreground objects. More on this compound and AR in Unreal here: https://aximmetry.com/learn/virtual-production-workflow/preparation-of-the-production-environment-phase-i/ar-production/aximmetry-de-scene-setup-ar/#ar-in-green-and-led-wall-projects
This would actually enable you to modify the foreground rendered (AR) objects even in live productions.
Lastly, all of this can be done in post-production using non-real-time rendering. This allows you to render at higher quality settings and resolutions as desired. This is mentioned in the page about recording here: https://aximmetry.com/learn/virtual-production-workflow/preparation-of-the-production-environment-phase-i/setting-up-inputs-outputs-for-virtual-production/video/recording/how-to-record-camera-tracking-data/#offline-rendering
Warmest regards,