3D cleanplate scanning for sim | Forum

3D cleanplate scanning for simple set props?

Hi!

We're totally blown away by the power of the 3D cleanplate. It is absolutely stunning how it actually works, not just in theory, and allows to effortlessly get s great key even when the green screen is lit horribly unevenly and low light and talents are being lit with color lights etc. Totally changed everything! No need to care much about lighting the green screen anymore, we just focus on lighting the talent to make the comp work. Almost a fix to everything! But, the simple green screen cleanplate can't be used for everything.

Now at a shoot we just had we needed to use set props like a big green box that the talent could sit/stand on and as we couldn't use the cleanplate to scan it we had to deal with all the ancient keying issues from forgotten era before july 2023 ( :D ).

Now my question is, is there anyway to use the 3D cleanplate generator to scan a separate 3D model of a prop? How could this be done? I could easily bring an accurate 3D mesh to Aximmetry that has been prepared like the green screen 3D model needs to be and I could match its position perfectly with the real world prop.

Am I right to assume the prop can not be part of the green screen model as it would create a very complex shape? I undestand the model has to be one solid mesh with no overlapping geo so just adding a 3D mesh to the floor of the green screen model doesn't work, right. Wouldn't necessarily be a too difficult trick to fix that overlapping geometry by just removing it all and fixing the topology. A bit tedious perhaps, but even this could be often 100x faster than deal with insane keying problems in post.

Other option I thought was if it was possible to just create a simple 3D model of the prop separately and use that separately to scan just that. Depending on a shot, in our green box example having just three sides of the box would cover everything perfectly and solve insane amount of issues. This kinds 3D model wouldn't be more complex than the green screen 3D model even if it was different of course.

This would bring so much more possibilities for complex shoots. If this was somehow possible, is it possible to use two separate 3D cleanplates with different scans in one project, one for the green screen and one for the prop?

If combining the two cleanplates in one project was not possible, it would still be possible to take advantage of the prop cleanplate keying in post by simply rendering matte of that specific area separately in a separate project. This obviously IF a 3D model of a prop can be made to work with the 3D cleanplate generator. Comparing to a lot of trickery often needed in complex projects in post, this would still be a time saver and extremely interesting to us!

Thank you and wish you all a great week,

Emil

Nestruction Studios

Eifert@Aximmetry

Hi Emil,

Thank you for your kind words regarding the 3D clean plate.

Yes, overlapping geometry (concave geometry) within the 3D clean plate is currently not supported.

There is no simple solution for this yet. However, we added support for concave geometry in the 3D clean plate to our request list and we will consider adding it in future releases.

In the meantime, one approach you could explore involves using more than one keyer in the camera compound. This method is similar to the solution outlined here.
But rather than merely blending the two keyers, you would need to create a 3D mask for your large green box prop. To accomplish this, you can use the Mask Mixer module. For its Mask pin, you will need to generate the 3D mask. This can be easily done by duplicating the Studio Mask compound and then creating a parent Scene Node for it with a 3D box model (you can use the Box module in Aximmetry for this purpose). This box will serve as the mask and, consequently, should employ a shader with a solid black color. For a smoother transition between the two keyers, consider adding a Blurrer module to the mask's image output.
It should look similar to this:

The two keyers can be two 3D clean plate keyers or any other type of keyer.

Should you have any further questions or require additional guidance on this, please don't hesitate to reach out.

Warmest regards,

Nestruction Studios

Ok, thank you. It would be awesome to besides the 3D green screen model get another slot for a 3D prop or whatever it could be called. One additional slot for importing a separate 3D mesh for additional 3D cleanplate scan. But I'd understand if that would overall cause just more confusion than relief 😅

How about the option of just rendering these prop cleanplates separately in a different empty project? As the keyer just inputs a video for the cleanplate I doubt it would care whether the cleanplate input is coming from the 3D cleanplate generator of the same project live or if it's a video player that is playing back a pre-rendered 3D cleanplate recording made in another project. Rendering these would be pretty easy if it works.

But can the cleanplate generator accept a mesh for scanning that is like just 2 or 3 sides of a cube? This would cover a lot of ground for props. Kinda totally different type of mesh than a cyclorama but no overlapping geo etc. I can probaply test this out my self within a few weeks but just curious if you're faster with an answer :)

Emil

Eifert@Aximmetry

Hi Emil,

You can add another Clean Plate generator within the same camera compound; there's no need to start a separate project for this. To do this, you'll need to edit in place the Keyer compound found inside the camera compound. You can then duplicate the CLEAN PLATE 3D GENERATOR within that keyer compound. To blend the different Clean Plates together, use the Mask Mixer module, as I mentioned in my previous post. This blended output should then be connected to the Advanced keyer compound's Clean Plate pin. Also, connect its Parent pin to a Scene Node module, which should, in turn, be connected to your prop models.

In post-production, you should be rendering in non-realtime. This method eliminates the need for pre-rendering (clean plate video) to save on processing time. For more information, read the offline rendering paragraph here: https://aximmetry.com/learn/virtual-production-workflow/preparation-of-the-production-environment-phase-i/setting-up-inputs-outputs-for-virtual-production/video/recording/how-to-record-camera-tracking-data/#offline-rendering

It's probably possible to generate two or three sides of a cube with the clean plate generator and utilize them in the solution described above. However, I'm slightly skeptical about how effectively this will work in real-life scenarios. Each time you move or rotate the prop, even slightly, you might need to rescan it due to the changes in the lighting on it.

Warmest regards,

Nestruction Studios

Woooow it works perfectly!! Holy nuggets this is brilliant!

I did a test with a mentioned three sided box object and it works.

For whatever reason I got stuck into thinking I´d need to have a
input pin in the tracked cam compound for the 3D mesh of the additional clean plate, but
like you mentioned the parent pin in the clean plate generator compound is right there ready
for any additional mesh to be scanned. Amazing! Like it always seems to be, the tools are always
there to build almost anything.

Duplicating the generator and its function for a separate mesh was fairly easy, just expose the pins and
connect them the same way as the first one, just the shader needed to be brought from inside the second clean plate generator compound.
For combining the clean plates of the green screen cyclorama and the separate prop I just grabbed the output from inside the generator as the main out pin had a solid color background instead of transparency. I´m sure there are many different ways to do it, this was one and seems to work.

Very happy to see it´s so easy and incredibly effective. Wish i had had this two weeks ago hah!

For real world scenarios I see great potential and it would have solved a lot of issues already in a shoot last month. A lot of structures where the actors need to be able to stand/sit/lay on or interact needs to be sturdy anyway, and as I´ve been using 3D masks for same props I noticed the props did not move a centimeter the whole shoot day. We´ve even screwed a drum set to the floor and it actually worked, didn´t move one bit!

I´m gladly surprised by how this works cus it is as easy to move the additional 3D meshes for clean plate as it is for any other 3D objects for 3D masks for example so it is super easy to match the locations for a 3D mesh and a physical studio prop. It makes such a big difference that it will allow to take jobs where difficult lighting scenarios and physical props would be needed and also to get better results faster even in basic scenarios with physical props. Very excited about it!

Emil

3D cleanplate scanning for simple set props?

Cookie consent