BMD Lens Tracking: JSON input into Aximmetry Broadcast DE

 

I have an Installation that runs 3 BMD Cameras into Aximmetry Broadcast DE with EZtrack Lighthouse 6DOF and Zoom/Focus Tracking by SWAN and electromechanical Lens encoders.

ATM I can't track focus the way I'd like to because the lenses (Canon CN-E 18-80 IS KAS) have an internal focus drive that does not move the focus gear ring on the outside of the lens when actuated by the BMD Focus demand attached to the camera. Current workaround is using some old mechanical Follow Focus units, but I´d like to get rid of them as it precludes using autofocus and is generally clunky to use.

The Cameras can be polled for the focus position of the lenses over ethernet as per this document:

https://documents.blackmagicdesign.com/DeveloperManuals/BlackmagicCameraControl.pdf

As I understand it sending GET /lens/focus at regular intervals will yield a response in the Form of a JSON number value "focus" in the format .0000-1.000  (10 bit floating point).

So I would need to regularly poll this value as often as possible (maybe 1-2 times a second, focus tracking the background is not *that* time critical), buffer it, interpolate the values with some sort of smoothing and finally feed it into Aximmetry/UE.  

I´m not really a programmer at this level, so the question is: is this feasible at all? Does Aximmetry offer this kind of interface and data handling?




   Stefan Reck

 
Profile Image
Eifert@Aximmetry
  -  

Hi,

I don't see any reason why this wouldn't be feasible. However, if you plan to use it with different Zoom levels, it can get quite complicated. Unfortunately, we can't test it because we currently don't have a BDM camera that supports the Camera Control REST API.

You can use the HTTP Request module to access the API.
You can read about the module here: https://aximmetry.com/learn/virtual-production-workflow/preparation-of-the-production-environment-phase-i/setting-up-inputs-outputs-for-virtual-production/external-controllers/using-websocket-and-http-in-aximmetry/#http-request

Then you can convert the received JSON format to Aximmetry's internal Collection data type with the JSON Text module.
BMD Lens Tracking: JSON input into Aximmetry Broadcast DE
You can learn more about the Collection data type here: https://aximmetry.com/learn/virtual-production-workflow/preparation-of-the-production-environment-phase-i/scripting-in-aximmetry/flow-editor/collection-for-databases/ 


In the end, you would need something like this:
BMD Lens Tracking: JSON input into Aximmetry Broadcast DE

First, you would need to determine the lowest working Auto Interval of the HTTP Request module for the API. The lower the interval, the less smoothing will be necessary.

Use the Collection Scalar module to extract the focus data from the Collection data type (JSON).

The Delay and Smooth Scalar modules will handle interpolation and delay the focus data to the rest of the camera's input. First, identify the delay between the video input and the focus data. Change the focus rapidly on your camera and compare when the focus value changes versus when the focus change is visible in the INPUT of the camera compound.
The focus data will probably arrive before the Video Input. Use the Delay module's Delay pin to increase the delay of the focus until it matches the INPUT video. You can connect the Delay module's output to a Scalar Peeker module to see when its value changes.

If the focus data arrives after the Video Input even without the Delay module, then increase the video delay by adjusting the In-to-out Latency:  https://aximmetry.com/learn/virtual-production-workflow/preparation-of-the-production-environment-phase-i/inner-workings-of-aximmetry/in-to-out-latency/#in-to-out-latency-1
This might not be enough, then further increase the delay using the Camera Delay parameter in the INPUT panel:
https://aximmetry.com/learn/virtual-production-workflow/preparation-of-the-production-environment-phase-i/green-screen-production/tracked-camera-workflow/inputs-tracked-camera/#delays
Note, that you will also need to change the tracking delay accordingly.

Once you have a good idea of the actual delay between the video and focus data, set the Delay pin of the Smooth Scalar module according to the Auto Interval of the HTTP Request. Subtract this delay from the Delay module's delay pin, as the Smooth Scalar module adds additional delay to the focus data.


Now the hardest part, calibrating the focus. For one zoom level, you could use the Curve module, which functions similarly to the Sequencer: https://aximmetry.com/learn/virtual-production-workflow/preparation-of-the-production-environment-phase-i/scripting-in-aximmetry/sequencing/sequencer-and-sequence-editor/

According to the Camera Control REST API documentation, you will receive focus data values between 0 and 1.
Add keys in the Curve module between 0 and 1 (approximately 4-6 keys), where the key values will represent the focus values used by Unreal.
BMD Lens Tracking: JSON input into Aximmetry Broadcast DE
To find these values, you can use the method demonstrated in this video for focus calibration in the Basic Calibrator (but instead of using the Basic Calibrator, you will have to use the Curve module): https://aximmetry.com/learn/virtual-production-workflow/preparation-of-the-production-environment-phase-i/tracking/basic-calibrator/

However, this will only calibrate your focus for one Zoom level!
If you plan to change Zoom levels, you will need to perform multiple focus calibrations with multiple Curve modules at different Zoom levels, and then interpolate between them. The accurate interpolation between Zoom levels is more complex than using Smooth modules, so for more accuracy, you could consider feeding the focus data back to the Basic Calibrator to create a calibration profile. This would end up in a quite complex setup, but if you plan to do so, I can look into the specifics of how you could achieve this.


Whether you succeed or encounter challenges, we are curious about your results.

Warmest regards,

 
Profile Image
Stefan Reck
  -  

Sounds good, thank you for the explanation. Do I even need to calibrate focus for different zoom levels? The lenses are completely parfocal...

 
Profile Image
TwentyStudios
  -  

@Stefan Reck: If the lens is Parfocal you probably won’t have to do multiple calibrations.

If you figure out a solid workflow and build a simple compound for this I’m sure other people would be interested as well. You could potentially sell it on axicomps.com to recoup some of the invested time. 

 
Profile Image
Stefan Reck
  -  

The guys over at BMD have also had a look at this, seems like it should actually be doable...:

https://forum.blackmagicdesign.com/viewtopic.php?f=2&t=205267#p1067720https://forum.blackmagicdesign.com/viewtopic.php?f=2&t=205267#p1067720





 
Profile Image
Eifert@Aximmetry
  -  

Hi,

Upon further reflection, the API provides the ability to retrieve the timecode, which can give you a good idea of the delay between the video and the API.
For example, you could determine the delay as shown below, using an HTTP Request module to get the timecode with an URL that end with ...api/v1/transports/0/timecode

Note, that I am not exactly sure what format the timecode is received from the API. Therefore, you may need to use one of these modules:

Of course, you could use this timecode to automatically sync the focus with the video. However, since you will be receiving the timecode and the focus value in two different packets (as I didn't see an option to get them in the same packet in the API), this synchronization won't always be perfectly accurate. There will probably be a -/+1 frame difference.

Warmest regards,

 
Profile Image
Stefan Reck
  -  

Thanks again for the workflow suggestion. I have tested the basic parts of it (cameras reliably return values if hammered with requests...) and will do a full up test shortly.

As I have not one but two tracked cameras I will obviously need to duplicate this. So how do I make the system switch the focus signal source whenever I switch cameras?

 
Profile Image
Eifert@Aximmetry
  -  

Hi,

You can get which camera is selected on the SELECT CAMERA panel using a Transmit Integer module, with the From Tunnel set to SELECTED CAMERA. Which then you can use with a Switch Scalar module.
BMD Lens Tracking: JSON input into Aximmetry Broadcast DE

To efficiently duplicate your logic, you can group elements into a compound. Learn more about grouping and compounds here: https://aximmetry.com/learn/virtual-production-workflow/preparation-of-the-production-environment-phase-i/scripting-in-aximmetry/flow-editor/compound/ If you use linked compounds, any future changes will be applied to all duplicates.

Warmest regards,

 
Profile Image
Stefan Reck
  -  

Thanks, that looks fairly straightforward. I might expand that a little to also give me tallies on an external multiview monitor, but that depends on whether I can send UDP commands *from* a specific port from within Aximmetry. See here: https://my.aximmetry.com/post/4129-sending-udp-data-from-a-specific-portq18

Going back to lens tracking: Do I have to set in the curve module the format the output needs to be in (probably a distance in meters) or is that already pre-set in there? What would be the correct extreme values here? Minimum focus distance would be either 0.5m or 1.2m depending on the lens, but what about infinity? 

Doing the actual calibration shouldn't be too hard then. With the lenses being parfocal a standard TV test chart on a wheeled stand and a laser range finder strapped to the camera will quickly give me a table of raw readout values vs. actual focus distance... 

 
Profile Image
Stefan Reck
  -  

btw. , here's the focus data table for both lenses, in case anybody has a similar setup:


Focus Data BMD URSA Broadcast G2 - Canon CN-E18-80mm T4.4 L IS KAS S

0.0000000 Minimum

0.10205078125  0.50 m

0.28515625    0.60 m

0.3997802734375 0.70 m

0.4510498046875 0.75 m

0.4608154296 0.80 m

0.51171875    0.85 m

0.5477294921875 0.90 m

0.59912109375 1.00 m

0.632568359375 1.10 m

0.6630859375   1.20 m

0.6881103515625 1.30 m

0.7034912109375 1.40 m

0.7227783203125 1.50 m

0.736328125  1.60 m

0.7557373046875 1.70 m

0.780517578125 2.0 m

0.8359375 3.0 m

0.862060546875 4.0 m

0.875732421875 5.0 m

0.8778076171875 6.00 m

0.8785400390625  7.00 m

0.89208984375  8.00 m

0.89208984375  9.0 m

0.892822265625 10.0 m

0.8997802734375 11.0 m

1.0000000 infinity



Focus Data BMD URSA Broadcast G2 - Canon CN-E70-200mm T4.4 L IS KAS S

0.000000000000  Minimum

0.0555419921875 1.20 m

0.135986328125  1.30 m

0.2545166015625 1.50 m

0.298583984375 1.60 m

0.3702392578125 1.80 m 

0.4600830078125 2.00 m

0.642822265625 3.00 m

0.7154541015625 4.00 m

0.8433837890625 6.00 m

0.873291015625 8.00 m

0.880126953125 10.00 m

1.000000000 infinity



 
Profile Image
Stefan Reck
  -  

So the test worked, the signals go where they should, readout is stable and I can focus and defocus objects in UE at will with the BMD focus demands on the cameras. Cameras are polled 10x per second which is probably excessive, but it pretty much obviates the need for the delay in front of the smoothing section.

There are some issues before this can go to production though:

- I really need to know the Cam_Focus value Aximmetry assumes to be infinity to get the translation curve right.

- I have another, virtual camera in this Compound (3+3 Mixed Unreal). Where do I get the focus value of this to feed it into the switch scalar?

- As you can see from the table above the translation from the raw BMD focus values to physical focal distance is not linear at all, it looks more like an exponential curve. This makes it impossible to accurately model the translation in the curve editor, as the x-axis resolution in the space of just 1 second (Values 0 to 1) is simply not fine enough. Is there a better way to do this? Maybe apply a simple mathematical formula (x10 or even x100, or a logarithm) to the raw data to make this possible? How? 


 
Profile Image
Eifert@Aximmetry
  -  

Hi,

Focus curve and data table

Unreal expects the focus data in meters, and Aximmetry does not perform any conversion on it once you set it in the collection. Therefore, you should provide these meter values in the curve. Note that "Infinity" focus actually means no focus—your camera receives all the light as parallel lines, so no focus effect occurs. In Unreal, the 0 focus value has the same effect. So you should avoid setting the "Minimum" focus value to 0. Instead, consider setting the minimum focus value to a small number like 0.01. For infinity, choose a high value that provides a similar effect to your studio camera when it is just below infinity. Ensure this value is greater than the furthest object in your scene so that when your studio camera is set to infinity, the virtual world also maintains no focus.


Mixed camera compound

In mixed camera compounds, the Cam Focus Distance pin transmits focus data exclusively for the virtual camera.
When using a tracked camera, the Control Data collection pin transmits the focus data.
It is easy to miss the Control Data pin on the compound:
BMD Lens Tracking: JSON input into Aximmetry Broadcast DE
 


Interpolate

The curve editor is as fine as float values can be:

It can be somewhat confusing since it displays Time and Frames on the X-axis by default.
However, you can customize what is displayed there by navigating to File > Properties - Timing:
BMD Lens Tracking: JSON input into Aximmetry Broadcast DE


You can also use a formula in Aximmetry instead of the curve module. For example, some websites provide online (polynomial) curve fitting.
You can easily paste the generated formula into Aximmetry using an expression compound:
BMD Lens Tracking: JSON input into Aximmetry Broadcast DE
There is an example of an Expressions compound available at: [Tutorials]: Scripting\Expression.xcomp

Warmest regards,

 
Profile Image
Stefan Reck
  -  
Thanks, so I won't have to worry about the virtual camera at all. For the time being I used a Multiplication x100 to expand the x-axis in the curve editor, that makes manually placing the keyframes possible again. New time I'm in the studio I'll try changing the time format to something that enables me to directly enter values from 0 to 100 in the x-axis. I´ll also try to use an exponential curve; that would give me the smoothest translation, at the cost of some accuracy towards the infinity end of the values as I will have to cut off the output somewhere.
 
Profile Image
Stefan Reck
  -  

So, I'm getting there slowly, but the curve module is probably not really the right tool here, so maybe there is an alternative...?

Short explanation: The raw focus input values range from 0 to 1. Multiplying by 1000 gets them into a range that is usable with a practical amount of zoom on the x-axis of the curve editor. However switching the time and frame display as explained above does *not* actually change the input format, it will always be processed as HH:MM:SS:FF. Not knowing this I changed it to display frames, defined about 45 keyframes to make a reasonably accurate curve and found it did not work right. Everything above a raw input value of 0.03 went straight to infinity (maximum output value). So with the curve set I changed the multiplier to 33.10 (equal to 1000 frames at 30fps) and it seems to work - or does it? The input values are in straight decimal, but the format the curve module is expecting is obviously not...?

 
Profile Image
Eifert@Aximmetry
  -  

Hi,

I think your main issue is that you can only set whole values in the Time field of the keys.
If you change the setting to Proportion Time Format, you can adjust the Time field down to three decimal places:
BMD Lens Tracking: JSON input into Aximmetry Broadcast DE

This 3 decimal limitation applies only to the Time field in the UI. By dragging the key with the mouse in the timeline of the Sequence Editor, you can set even smaller decimals, although these smaller values will not show up in the Time field but will appear in the output.

The input value does not have any limitation; it can be any floating-point value.

You could try setting up the keys and test the input values in the Dashboard panel. For this, you need to expose the Curve module's input to the left side, so it would appear on the Dashboard and use a Scalar Peeker to see the output values:
BMD Lens Tracking: JSON input into Aximmetry Broadcast DE


Additionally, you should be able to work within these value ranges in the curve editor without zoom issues. Use CTRL + middle mouse to zoom and SHIFT + middle mouse to pan up and down.
You can maximize the Curve Editor by clicking this button:
BMD Lens Tracking: JSON input into Aximmetry Broadcast DE

Warmest regards,