Per default, the  live video is rendered inside the UE4 scene. This means it’s affected by the whole UE4 rendering pipeline, where things like fog and post processing effects like Bloom, chromatic aberration and color correction will affect the video. If you’re scene is correctly set up (custom depth stencil pass) it shouldn’t be affected by TAA and there is an Invere Tonemapper option to counter the effects of the UE4 Tonemapper, but you will often have to tweak both the scene and the incoming video to get a good quality.

Because of this there is an option in the Aximmetry Cameras panel to turn off Allow Virtuals. This will instead composite the live video inside Aximmetry, like a traditional post composite. This means the colors and image quality will be unaffected by UE4 and will get you a much sharper image, but you will have to work a bit harder with the Lightwrap and color correction to integrate the live foootage realistically in the scene.

You can also get proper lighting to your actor without rendering your footage trough unreal but its a bit of a hack.

You would need to basically send unreal a masked white plane and then in Aximmetry using multiply module comp the irradiance received by the white plane over your actor.

You would also obviously lose reflections of your actor as they would appear white with this method.

@Deata: I don’t think that would work since the white plane would affect bloom in the PostProcessVolume. Better to use a black mask and add blend the footage over that. 

In any case, I would love for Aximmetry to comment on this. The video quality with AllowVirtuals on is definitely much worse than it should be. Maybe they could implement the mask+Blend as an alternative? Or dig deeper in the UE4 source to figure out a better way to render the video.

@twentystudios ahh yeah I completely forgot about bloom, but couldn't you exclude the plane with a custom stencil

@DeataProduction: I think you would loose a lot of the benefits of compositing in 3D space without the billboard interaction with bloom. Theoretically, a black alpha mask hole rendered in UE4 could still receive bloom, fog and other effects in 3D space and then you would multiply the (unmultiplied) RGB billboard on top of that inside Aximmetry. The problem is that the soft edges wouldn’t align perfectly due to scaling and motion blur from the alpha mask would also be problematic, unless exactly the same motion blur was generated for the billboard in Aximmetry. 

@twentystudios yeah currently it seems there is no real way to get all the benefits without drawbacks, as with a black mask you would lose all lighting information 

@DeataProduction: You actually wouldn’t loose all lighting information, since the black mask itself could be lit. When multiplied with the the clean keyed billboard in Aximmetry it would still keep some of the light interaction. Don’t know if the results would actually be useable, but might be worth a try. 

Hi all,

Some example screenshots would be nice to get into more details.
Note, if you don't want to show your own talents/actors in your screenshot, Aximmetry comes with some green screen videos located at Common\Videos\Green.

I think most of the problems mentioned here are coming from lighting.
Sadly, we don't have much documentation on Unreal lights and on real-world light set up for virtual reality.

About the Mask+Blend idea, you can achieve a similar effect by changing the shader of the billboard to use emissive more than the base color. Of course, this will have its own problems. But if you want to experiment:

Warmest regards,

@Eifert: Sorry, but this issue has absolutely nothing to do with lighting in UE4 or the real world. When the billboard is set to Unlit it receives no lights from UE4 at all. 

Screenshots shouldn’t be necessary as you can just open one of the included Aximmetry UE4 projects and switch Allow Virtuals on/off to see the dramatic difference in sharpness on the billboard. 

This is one of the most discussed issues with Aximmetry, and it’s not an issue that competing software like Pixotope or Zero Density has, so I really hope you can take a deeper look into this. 

Hi TwentyStudios,

I am not exactly sure what DeataProduction and caohoangphucan problems are. Screenshots would help me a lot. Like probably you could immediately tell from a screenshot if they are using lit or unlit billboard.

For the "sharpness issue", you can try changing the Rendering to Linear in Aximmetry properties to see if that makes Allow Virtuals on/off more similar in the Aximmetry included projects.

Warmest regards,

@Eifert: I’m not talking about color shifts, bloom, lit/unlit or other things that might affect the billboard. In my case my issue with a just the clear and very noticeable loss of image sharpness (as in image resolution) with Allow Virtuals turned on. With Allow Virtuals off the mage is nice and crisp, with Allow Virtuals on there is a remarkable loss of fine details in the image, like it’s been smoothed with a mipmap. I understand that the colors might look different and that the billboard might be affected by other things in the PostProccessVolume, but I just want it sharp and not smudged.

@eifert:

I finally took some screen shots and captured a video to illustrate the issue. I've cropped in on the footage heavily so that it's even clearer, but even without cropping this issue is very noticeable. It's exactly the same frame in both examples, but there is a clear loss of image detail with Allow Virtuals on. This is an included Aximmetry UE4 sample project using the latest version.

I also captured a video that shows that the image is jittering with Allow Virtuals turned on, while it's totally stable with it turned off.

https://app.box.com/s/xu682brwm269pn1jkbv8tuyyif67f64r

Virtuals Off:

https://app.box.com/s/a1o94zpf0ebrt5ez7b6lhn26izq1yzf3

Virtuals On:

https://app.box.com/s/0ryy0zmjfgclzsmsv3zdwofmozenm02w

Hi,

If I am right, you recorded the footage while your billboard was zoomed out, and then you cropped that footage probably from 3840x2160 to 960x540.

So, I did some similar experiments.
The jitter is caused by the TAA, setting the Threshold to 1 stops it:

Maybe there could be an option to dynamically apply TAA based on distance.

I reproduced the sharpness difference in your image. But at this level of detail, Aximmetry is a very different render engine than Unreal. Never the less we are always looking for ways to make Allow Virtuals better.

While at close-ups the difference is this much:

I think, most new users will have more trouble applying the Unreal lights on their billboards than the sharpness issue.

Also, you don't need to link the images, the images that are pasted into the comments will be downloaded at original resolution. But anyway, thank you for the detailed images, it helps a lot, at least for me.

Warmest regards and happy Christmas,

@eifert: Setting the TAA to 1 doesn’t do anything to increase the sharpness, it just gets rid of the jitter while causing smudging instead. 

The issue doesn’t appear as much in your sample clip since the source footage contains less details (lower resolution?).

Again, I don’t think this has anything to do with users applying lights to the billboard. There are a lot of post from experienced users here and on Facebook groups that complain about sharpness specifically. We already know that this is something that can be fixed since other UE4 based software like Zero Density and Pixotope doesn’t have this issue, so I really think this should be considered a bug in the current implementation. There are many limitations with disabling Allow Virtuals, so while it fixes the the sharpness issue it’s just a bandaid at best. 

I hope you will take a serious look at this.

I agree with @TwentyStudios, there is a blurry layer on top of my talents! I reduced it a bit by setting up:

1) TAA Threshold to 1

2) Render To Depth to OFF (loosing shadowing)

3) Forward Shading enabled in Unreal

When I set "AllowVirtual" to ON I have this degrading blurry effect (on left here):

Look particularly at hands at written note!
Aximmetry you must fix this, it's unacceptable at this Broadcast DE price! Unless we all are doing some misunderstanding in our setups with Unreal?

My GPU runs at 48%, CPU at 49%

Hi TwentyStudio,

In my sample clip, the billboard was a bit bigger than the full size of the screen.

Your issue is quite specific to TAA and it is most visible when the billboard is small. We are working on better ways to handle Unreal's TAA. One result of that work is the TAA Threshold setting in the billboard of Aximmetry.
However, the TAA Threshold does not fully turn off the TAA on the billboard even if it is set to 1.
You could try in the example where you made the screenshots, to turn off the Anti-Aliasing (Unreal Editor Project Settings) and you will probably see that it fixes the blur:

Note for anyone wanting to fully turn off Anti-Aliasing like above, It will greatly reduce the image quality! You can make it a bit better by rendering it in UHD (2160p) and resizing the video to FHD (1080p) in Aximmetry.

Warmest regards,

Hi Ferlaur,

Unreal for Aximmetry designed solely with Deferred shading. It is very likely that multiple things won't work with Forward Shading.

Also, it looks like if forward shading is enabled, it flips the TAA Threshold and TAA Threshold 1 will mean 0.

Warmest regards,

@Eifert: Turning off the TAA isn’t a solution that anyone should consider. What is needed is for Aximmetry to modify the rendering pipeline to fully disable TAA for the billboard without causing any pixel jitter, blur or any other strangeness. We know that it can be done because it has in competing UE4 VP software.

Hi,

We increased the priority of the TAA blur issue.
And we are actively working on it.

Warmest regards,

Hi,

The latest Aximmetry release (2022.1.0) improved on the TAA blur issue:
"Improved billboard sharpness in Allow Virtuals mode." https://aximmetry.com/learn/software-version-history/#%E2%96%BA-latest-2022-1-0 

This also corrects the loss of detail when the billboard occupies just a little part of the virtual world like it was discussed in this forum thread.

Warmest regards,