actor mask, depthmap, render pass with Unreal

Studio Operators Content Creation Editions Composer 3D Scene Editing 3D Scene Export 2D Compositing Post Effects Video Playback Image Sequences Virtual Camera Camera Tracking Unreal

 

1. What is the best way to get a mask for the actor/billboard (based on the 3D camera) from an Unreal Project node?
2. What is the best way to get a depth map for the scene from an Unreal Project node?
2. What is the best way to get a render pass without the actor/billboard (but all the shadows still intact as if the actor/billboard would be there) from an Unreal Project node?



   jim4586

 
Profile Image
Eifert@Aximmetry
9/17/2021 3:49:16 PM
  -  

Hi Jim,

Masking is handled by the Aximmetry Camera which you add to your Unreal level. This will appear as a Mask output pin on your Unreal Project module (node) in your compound. Aximmetry will automatically use this mask when necessary.
https://aximmetry.com/learn/tutorials/for-aximmetry-de-users/using-tracked-cameras-and-augmented-reality-with-an-unreal-scene/#aximmetry-setup


For passing depth map to Aximmetry, you would need a Set Aximmetry Video node in your Unreal Blueprint. You would pass to that the Unreal's depth map using Unreal's Scene Capture Component with Capture Scene parameter set to "SceneDepth in R".
Please note that even if you have the depth map combining Aximmetry render with the Unreal render is not trivial because of the anti-aliasing.

In Virtual Cameras, you can turn off Allow Virtuals at Camera & Render Setup on the Cameras control board. This will make so, that only Aximmetry renders the actor's billboard, and Unreal will only render the shadows, reflections, and ambient occlusion. You will see that the Out output pin of your Unreal level in Aximmetry won't have the actor's billboard.

Warmest regards,

 
Profile Image
jim4586
9/17/2021 5:11:10 PM
  -  

To obtain the mask,

Is B Mask always set? We use virtual camera with allow virtuals but B Mask has no output from Unreal.

We use this camera component in Unreal right now:


 
Profile Image
Eifert@Aximmetry
9/20/2021 2:27:23 PM
  -  

Hi Jim,

B Mask won't be rendered if Aximmetry has no need for it, like when Aximmetry is not rendering any billboards.
If you really want to get the depth map and not render anything with Aximmetry, you might want to turn on Light Warp Cam 1 and set its depth and strength to zero:

Warmest regards,

 
Profile Image
jim4586
9/20/2021 6:16:47 PM
  -  

So how can we get the alpha mask of the billboard out when B Mask isn't set? Can we force Aximmetry to output it regardless?

Tried to use custom depth to set a mask for the billboard, but then the actor mask is binary and has no soft transitions/alpha values so that doesn't really work either.

We are trying to render all the passes from Unreal at once like we used to.

 
Profile Image
Eifert@Aximmetry
9/23/2021 10:10:55 AM
  -  

Hi Jim,

The settings in the previous comment with the "Light Warp Cam 1" should produce a B Mask without having the light warp rendered. This B Mask will have transparency if needed. Note the B Mask does not include the billboard/actor, the B Mask is used for rendering the billboard with Aximmetry's renderer and to determine how much of it should be rendered/masked.

I am not sure what "custom depth" setting you are changing.

Warmest regards,

 
Profile Image
jim4586
9/24/2021 3:37:25 AM
  -  

The B mask is not what we need then.

unreal has a property custom depth - check it out. It allows rendering a custom value to identify an object. However, we are not obtaining alpha mask that way, only binary.

How can we get an alpha mask highlighting the keyed actor in the rendered Unreal scene?

 
Profile Image
Eifert@Aximmetry
10/5/2021 2:38:13 PM
  -  

Hi Jim,

You could get the keyed actor and its position similar way as Aximmetry's camera does it in Unreal. After that, you could render it in your own way. You can find the 'Get Aximmetry' references to the transformation and video for example at Aximmetry_VirtualCam_3-Bill/Blueprints/Aximmetry_Virtual_Camera_1-3_Billboards Event Graph: (other Cameras have similar blueprint structure)

Warmest regards,



 
Profile Image
TwentyStudios
10/5/2021 7:33:43 PM
  -  

@jim4586:So You samt to get the alpha mask from the Billboard Actor? If you turn off Allow Virtuals, the Billboard is entirely composited in Aximmetry, only shadows and reflections are actually rendered in the UE4 scene. That means you should be able to get the mask directly from the billboard if you go into the virtual cam compound and look around a bit. Once you find it you could even add an output pin with the alpha to the compound so you don’t have to go into it every time to find it. 

 
Profile Image
jim4586
10/5/2021 11:56:10 PM
  -  

We wanted it all in rendering pass so we can easily render it (rather than having to switch settings).

We did find the following pin in the billboard compound, which seems to output a RGBA stream (with alpha mask) of the actor from the camera perspective. Could this pin be exposed by default out of your compounds?

Maybe others would be interested in it too, and it seems like you already have the right alpha mask in Aximmetry. Or is there something not right in terms of using the out pint from Masker? That way one does not need to mess with Unreal at all for the alpha mask and it renders in the same pass.


 
Profile Image
Eifert@Aximmetry
10/12/2021 1:40:00 PM
  -  

Hi Jim,

I am sorry, I thought you needed something like that in Unreal.

Note, that if you turn off the light warp, it will not render a mask at that pin (at Masker).

I am not exactly sure what for you are using this mask.
But if you can give us more details on how/why you are using this mask, then it could help us make this more exposed in future releases.

Warmest regards,

 
Profile Image
jim4586
8/10/2022 7:27:35 AM
  -  

@Eifert,

we are looking for render passes to tweak the footage in post production after the fact. Unreal has the movie render pass queue for this purpose, but how can we do this easily with Aximmetry?

passes of interest are:

- depth map with/without talent

- key

- undistorted footage of source

- distorted footage with original background but color correction applied

- object ID (cryptomatte) pass

- motion vector pass

- normals with/without talent

- stmap for undistort

- stmap for distort

- render without certain objects / deep compositing…

https://docs.unrealengine.com/4.26/en-US/AnimatingObjects/Sequencer/Workflow/RenderAndExport/HighQualityMediaExport/RenderPasses/

 
Profile Image
Eifert@Aximmetry
9/9/2022 11:58:57 AM
  -  

Hi Jim,

You would want to record the tracking data and use that in the movie render pass like it is described here: https://my.aximmetry.com/post/1686-can-i-use-movie-render-queue-to-generate

It is probably not possible to run the movie render pass with Aximmetry billboards in it. So if you want talent in your movie render pass, then you will have to composite it after the movie render pass is finished.
For example, by allowing virtuals off, Aximmetry could render the billboard into your movie render pass. You could load the movie render pass images into Aximmetry, and run the same camera footage and movement/tracking. Connect to the camera compound Rendered input pin the matching movie render pass images while using the Unreal compound to render the B Mask input pin.
Of course, Aximmetry could not do all the rendering settings for the billboard that is possible in the Unreal movie render pass, but depending on your project you might not need it.

Warmest regards,

 
Profile Image
jim4586
9/9/2022 4:48:55 PM
  -  

Hi Eifert,

can this be automated in Aximmetry? The whole idea of render passes is automation so that one gets all the different versions one needs for compositing but only has to set it up and render once. See if we have to work through 100 different clips/takes, then it’s really cumbersome to switch actor billboard on / switch it off, change something in Unreal to get a depthmap, motion vectors, normals, or then change in Aximmetry to get the stmap to undistort or stmap to distort etc. and each time we would have to trigger render. That’s then 100 takes x 8 render passes = 800 renders we would have to setup by hand, configure and click render each time.

if it’s just each clip once, it’s 100 renders.

 
Profile Image
Eifert@Aximmetry
9/12/2022 4:27:38 PM
  -  

Hi Jim,

Well, there is no out of box solution for this. Even the Movie Render Queue was in beta until UE5.

You will probably have to manually create from the recording data FBX files a new level sequence for each of the 100 takes.

Loading in the images of the Movie Render Queue in Aximmetry and rendering billboards into them probably can be automated. Including running through each of the 100 takes.

Note, that in my previous post I said that Aximmetry will render and composite the billboard into the generated images by the Movie Render Queue. The Unreal Movie Render Queue can not render the Aximmetry billboards, currently. This also means that you won't have reflections and shadows from the billboard.

I added better integration with Movie Render Queue to our request list. We will consider adding it in future releases.

Warmest regards,

 
Profile Image
jim4586
9/12/2022 5:44:43 PM
  -  

FBX recording data does not work for tracked cameras since it misses the camera distortion. We are talking about tracked cameras, not billboard rendering.

You should have your own render pass selector in Aximmetry so that it’s fast to render out different passes in one go when rendering a scene from sequencer.