Hi Jim,

Masking is handled by the Aximmetry Camera which you add to your Unreal level. This will appear as a Mask output pin on your Unreal Project module (node) in your compound. Aximmetry will automatically use this mask when necessary.
https://aximmetry.com/learn/tutorials/for-aximmetry-de-users/using-tracked-cameras-and-augmented-reality-with-an-unreal-scene/#aximmetry-setup

For passing depth map to Aximmetry, you would need a Set Aximmetry Video node in your Unreal Blueprint. You would pass to that the Unreal's depth map using Unreal's Scene Capture Component with Capture Scene parameter set to "SceneDepth in R".
Please note that even if you have the depth map combining Aximmetry render with the Unreal render is not trivial because of the anti-aliasing.

In Virtual Cameras, you can turn off Allow Virtuals at Camera & Render Setup on the Cameras control board. This will make so, that only Aximmetry renders the actor's billboard, and Unreal will only render the shadows, reflections, and ambient occlusion. You will see that the Out output pin of your Unreal level in Aximmetry won't have the actor's billboard.

Warmest regards,

To obtain the mask,

Is B Mask always set? We use virtual camera with allow virtuals but B Mask has no output from Unreal.

We use this camera component in Unreal right now:

Hi Jim,

B Mask won't be rendered if Aximmetry has no need for it, like when Aximmetry is not rendering any billboards.
If you really want to get the depth map and not render anything with Aximmetry, you might want to turn on Light Warp Cam 1 and set its depth and strength to zero:

Warmest regards,

So how can we get the alpha mask of the billboard out when B Mask isn't set? Can we force Aximmetry to output it regardless?

Tried to use custom depth to set a mask for the billboard, but then the actor mask is binary and has no soft transitions/alpha values so that doesn't really work either.

We are trying to render all the passes from Unreal at once like we used to.

Hi Jim,

The settings in the previous comment with the "Light Warp Cam 1" should produce a B Mask without having the light warp rendered. This B Mask will have transparency if needed. Note the B Mask does not include the billboard/actor, the B Mask is used for rendering the billboard with Aximmetry's renderer and to determine how much of it should be rendered/masked.

I am not sure what "custom depth" setting you are changing.

Warmest regards,

The B mask is not what we need then.

unreal has a property custom depth - check it out. It allows rendering a custom value to identify an object. However, we are not obtaining alpha mask that way, only binary.

How can we get an alpha mask highlighting the keyed actor in the rendered Unreal scene?

Hi Jim,

You could get the keyed actor and its position similar way as Aximmetry's camera does it in Unreal. After that, you could render it in your own way. You can find the 'Get Aximmetry' references to the transformation and video for example at Aximmetry_VirtualCam_3-Bill/Blueprints/Aximmetry_Virtual_Camera_1-3_Billboards Event Graph: (other Cameras have similar blueprint structure)

Warmest regards,

@jim4586:So You samt to get the alpha mask from the Billboard Actor? If you turn off Allow Virtuals, the Billboard is entirely composited in Aximmetry, only shadows and reflections are actually rendered in the UE4 scene. That means you should be able to get the mask directly from the billboard if you go into the virtual cam compound and look around a bit. Once you find it you could even add an output pin with the alpha to the compound so you don’t have to go into it every time to find it. 

We wanted it all in rendering pass so we can easily render it (rather than having to switch settings).

We did find the following pin in the billboard compound, which seems to output a RGBA stream (with alpha mask) of the actor from the camera perspective. Could this pin be exposed by default out of your compounds?

Maybe others would be interested in it too, and it seems like you already have the right alpha mask in Aximmetry. Or is there something not right in terms of using the out pint from Masker? That way one does not need to mess with Unreal at all for the alpha mask and it renders in the same pass.

Hi Jim,

I am sorry, I thought you needed something like that in Unreal.

Note, that if you turn off the light warp, it will not render a mask at that pin (at Masker).

I am not exactly sure what for you are using this mask.
But if you can give us more details on how/why you are using this mask, then it could help us make this more exposed in future releases.

Warmest regards,

@Eifert,

we are looking for render passes to tweak the footage in post production after the fact. Unreal has the movie render pass queue for this purpose, but how can we do this easily with Aximmetry?

passes of interest are:

- depth map with/without talent

- key

- undistorted footage of source

- distorted footage with original background but color correction applied

- object ID (cryptomatte) pass

- motion vector pass

- normals with/without talent

- stmap for undistort

- stmap for distort

- render without certain objects / deep compositing…

https://docs.unrealengine.com/4.26/en-US/AnimatingObjects/Sequencer/Workflow/RenderAndExport/HighQualityMediaExport/RenderPasses/

Hi Jim,

You would want to record the tracking data and use that in the movie render pass like it is described here: https://my.aximmetry.com/post/1686-can-i-use-movie-render-queue-to-generate

It is probably not possible to run the movie render pass with Aximmetry billboards in it. So if you want talent in your movie render pass, then you will have to composite it after the movie render pass is finished.
For example, by allowing virtuals off, Aximmetry could render the billboard into your movie render pass. You could load the movie render pass images into Aximmetry, and run the same camera footage and movement/tracking. Connect to the camera compound Rendered input pin the matching movie render pass images while using the Unreal compound to render the B Mask input pin.
Of course, Aximmetry could not do all the rendering settings for the billboard that is possible in the Unreal movie render pass, but depending on your project you might not need it.

Warmest regards,

Hi Eifert,

can this be automated in Aximmetry? The whole idea of render passes is automation so that one gets all the different versions one needs for compositing but only has to set it up and render once. See if we have to work through 100 different clips/takes, then it’s really cumbersome to switch actor billboard on / switch it off, change something in Unreal to get a depthmap, motion vectors, normals, or then change in Aximmetry to get the stmap to undistort or stmap to distort etc. and each time we would have to trigger render. That’s then 100 takes x 8 render passes = 800 renders we would have to setup by hand, configure and click render each time.

if it’s just each clip once, it’s 100 renders.

Hi Jim,

Well, there is no out of box solution for this. Even the Movie Render Queue was in beta until UE5.

You will probably have to manually create from the recording data FBX files a new level sequence for each of the 100 takes.

Loading in the images of the Movie Render Queue in Aximmetry and rendering billboards into them probably can be automated. Including running through each of the 100 takes.

Note, that in my previous post I said that Aximmetry will render and composite the billboard into the generated images by the Movie Render Queue. The Unreal Movie Render Queue can not render the Aximmetry billboards, currently. This also means that you won't have reflections and shadows from the billboard.

I added better integration with Movie Render Queue to our request list. We will consider adding it in future releases.

Warmest regards,

FBX recording data does not work for tracked cameras since it misses the camera distortion. We are talking about tracked cameras, not billboard rendering.

You should have your own render pass selector in Aximmetry so that it’s fast to render out different passes in one go when rendering a scene from sequencer.

Hi Eifert,

Was there any updates for rendering different render passes from Aximmetry? Is there a way to create these passes automatically from the standard compounds? Because we don't want to have to modify the compounds every time again when updating...

Hi Jim,

Sadly, there wasn't any change regarding this.

Note that, we have something like this on our request list, so we will consider adding it in future releases.

Warmest regards,

Hi,

We're excited to share that our latest release, Aximmetry Version 2023.3.1, which has just been launched, now includes an STMap generator.

To access this feature, locate the STMap compound at [Common]:Compounds\Utilities\STMap_Generator.xcomp.
Here are a couple of ways you can set it up:

• For live tracking input - simply use a Camera Tracking module:
  
  
  
• For playback of tracking recording from tracked camera compounds - Use the Record Data collection pin's Lens Distortion key to get the Lens Data:
  
  

Thank you for your patience and ongoing support.

Warmest regards,

Thanks @Eifert, what about supporting different render passes, has there been any progress?

@eifert For stmap, is there an option to also export undistort stmap and undistorted footage?

Hi Jim,

For the various render passes in Unreal Movie Render, it's still necessary to import the tracking data in FBX format into Unreal. However, Aximmetry has recently simplified this process a bit by also exporting focus data alongside it.

Yes, you can export an undistorted STMap. To do so, simply use the Undistort pin:

Warmest regards,