Replaying a real-time session with high-res green-screen source + camera switching

Studio Operators Sequencer Video Recording Camera Tracking Unreal

 
We want to be able to perform and record a real time session and replay the exact camera moves we did and use the HQ recorded camera source. So essentially change 1 thing in the Unreal Scene (like a logo) and ‘record again’ with the same green screen source and all the previous camera moves we did.

This is what I’ve done so far:

We're using Teranex AV’s which are generating the time code for the Panasonic PTZ’s and embedding this on SDI. I’m recording the high quality green screen per camera source on Hyperdeck 12G's. I’m then also recording the tracking data (xdata) with the external time code. When I transfer these files to the computer, rename them to match the xdata tracking data file, I can ‘play’ in the sequencer and I can see the Unreal background moving and zooming in perfect sync with my green screen footage. So far, so good.

But I’m missing the actual changing of the camera’s we did. As I was under the impression that this also is being recorded. But instead I can now press ‘play’ and do a new switch of the camera’s, but that’s not what I want.

I’ve seen that there is also a tracking option next to the record module that records the program output and that results in a FBX file. Never worked with that before but as far as I understand it, that should record the camera’s switched etc. So I've opened it in Blender and indeed I see the key frames and the 'switching' of the camera. So the data is there.

I've also tried and imported the FBX in Aximmetry (in the importer options choose 'Camera's), but not sure how I can use this info further. 

So, is it possible what I want?
Can I replay the session from Aximmetry WITH the camera changes we did? I understand if we created a playlist, we could re-run the playlist, but at the time of recording I have no idea which camera will be live.






   cm-aximmetry

 
Profile Image
ericmarodon
  -  

Hi,

Have you tried the new playlist module?

It's very easy to use and you can program your camera switches once and recall them as many times as you want.

I've been using it for something very similar to what you suggests.

Best,

Eric


 
Profile Image
cm-aximmetry
  -  

Hi Eric, 

Thanks for your quick response, like I mentioned in my post, I know of the new playlist module. But I am/was looking for a way to replay a recorded live scene with the exact same camera changes/switches, without having to re-create them using the playlist module (or start with that). As we switch live, we have no exact idea which camera/shot will be used etc.

@Aximmetry, I've come up with a solution that works, but would like to hear your thoughts. Also for other users that face the same issue.

This is what I've done:

  1. Captured the input recording tracking data (.xdata) per Camera source and the FBX tracking file during the live recording
  2. Captured the HQ green-screen camera source with time code on SSD via Hyperdeck 12G recorders
  3. Opened the FBX file in Blender, changed the out-point to include the last key frame
  4. Discovered that on a camera change, the Y-position of the camera in the key frame changed
  5. Exported FBX from Blender to DAE (Collada) file, but included the key frames in the anim-section
  6. Imported the DAE file with 'import options' to include camera
  7. Entered the DAE file compound and exposed the Play/Stop/Restart input pins from the Sequencer
  8. Also exposed the 'World Pos' to the right (output)
  9. Hooked up a 'Vector Split' module because I only needed the Y value
  10. Added a greater / less logic + Switch Integer module, basically saying: If Y-pos = here, then Cam 4, otherwise cam 5
  11. Hooked that up to 'Playlist Select Cam'
  12. Added some triggers that could trigger both the normal and this sequencer + a delay option
  13. Set Camera's to Playlist mode on 'Camera Page'
  14. Make some initial adjustments on the sequencer to match the recordings with the final output recording (so start is the same)
  15. And bingo… When I play back the HQ recording, the camera now also switches on the exact correct moment



@Aximmetry... Although this is working in our case (fixed camera's), and you only have to set up the initial sync, I'm wondering if this can be done better?

Still learning Aximmetry, and happy that it's working, but the process is quite frustrating because of the lack of documentation and the endless possibilities. 


 
Profile Image
Eifert@Aximmetry
  -  

Hi,

Sadly, there is no ready-made solution for this. However, we have similar items on our request list, so we will consider adding it in future releases.

Alternatively, you could make your own save and load logic in the Flow Editor. Like, you could use the JSON Exporter and JSON File modules to load and save. And the System Params module to get the master timecode for timing.

Also, I am not exactly sure why the Y-position change happens, so note that your solution might not work in the long run.

Warmest regards,

 
Profile Image
cm-aximmetry
  -  

Hi Eifert,


Thanks for your reply. 
The Y-position is changing because we are using 2 fixed PTZ's. So I think it is expected behavior. Please see the attached images.

Also, could you explain more about the JSON part and the system Param Timecode part, and how I would set this up?




 
Profile Image
Eifert@Aximmetry
  -  

Hi,

Ahh, now I understand, the PTZ cameras explain why your Y-position is constant.


To save camera switches, at first, you should set the Timecode Master to be one of your input videos:

This timecode will be read by the System Params' Master Timecode pin. And it will be used both for recording camera switches and playing them back:

Note, in the next major Aximmetry release, you will be able to generate a timecode master within the Flow Editor using one of the "... To Timecode" modules.


Then you can use the following logic to save and replay. It will save the timecode as a key at every camera change, and the ID of the camera will be the key's value.

This way of saving things is far from an ideal solution, but it should work for simple things.

Note, the replay uses the External Control Mode, which you need to turn on for the replay to work: https://aximmetry.com/learn/virtual-production-workflow/preparation-of-the-production-environment-phase-i/green-screen-production/virtual-camera-workflow/cameras-control-board/#external-control-mode 

Warmest regards,

 
Profile Image
cm-aximmetry
  -  

Thanks, for the extensive example, appreciated! I'm going to try to implement this today.


 
Profile Image
cm-aximmetry
  -  

Hi Eifert,

I understand the logic for 95% here, as you are combining the time code with the currently selected cam, and on-change, writing that to a JSON. The saving part works perfectly, as it adds a line to the JSON every time I switch camera's.

But on replaying via the sequencer, it still displays the Master Time code of the incoming live signal, not from the recorded video. That's why I think that part isn't working yet.

Some extra info:

- PTZ 1 (cam 4) is sending time code and is set to Time code Master
- I've set the External Control Mode ofc (had to do the same with my FBX workaround)
- I've also tried using the restart trigger for the sequencer, but no effect here.
- There is a default? cam0 line in the JSON, I think when loading the project?

Any idea why the playback part (loading logic) is failing?


 
Profile Image
Eifert@Aximmetry
  -  

Hi,

What you are experiencing is actually a bug in the compound. Which will be fixed in the next major version.
You can easily work around this bug by making sure no other input is set to Master Timecode. And add a Sequence Video module to the Sequencer. Then set this Sequence Video's video file to the recorded input video you used as the master timecode while recording. And set the Sequence Video to Timecode Master:

There are going to be actually several Timecode improvements in the next major Aximmetry version.

And you should press the reset right after pressing recording. Otherwise, the initial state of the Selected Camera won't be recorded.

Sadly, there is no Transmit module that sends out that the recording started. You could use a playlist to start recording and triggering reset in sequential order: https://aximmetry.com/learn/virtual-production-workflow/preparation-of-the-production-environment-phase-i/scripting-in-aximmetry/sequencing/playlists/

Warmest regards,

 
Profile Image
cm-aximmetry
  -  

Yes, that trick indeed fixed that I am now seeing the time code of the source file. And when I play/pause in the sequencer, I see the correct time code.

However, still no switching of camera's. Looking at the JSON I should have passed a switch moment. Not sure why nothing is happening though. When I peek on the Collection of the JSON file (or JSON out) I don't see any info, feels like that is the issue.




 
Profile Image
Eifert@Aximmetry
  -  

Hi,

Yes, the problem will be with the JSON file module. Note, you have to give the file name with its extension, like Y:\test.json. Cause without an extension like this Y:\test, it will only work at exporting with the JSON Exporter module, but it won't work at importing with the JSON file module as that module requires file extension.

Warmest regards,

 
Profile Image
cm-aximmetry
  -  

Ah,

You are totally correct, as I saw the output file I didn't think about this again. I'll try this and report back.

Thanks.

 
Profile Image
cm-aximmetry
  -  

Hi Eifert,


It's indeed working now. However, it has one interesting behavior in the current setup.
That is, that every time you play back the video from sequencer, it's re-writing to the original JSON file and changing the timings. As the current logic is "write on change".

I think I'll have to implement some logic like"If in Playback mode, don't write to the JSON".

But apart from that and the time code bugs (with working workaround), it's working as intended.

 
Profile Image
Eifert@Aximmetry
  -  

Hi,

Ahh yes, I forgot to say that. The easiest way to separate import and export is to have an if module before the JSON Exporter module's Export pin.
Note, you can connect the Trigger data type pin to the If module (vector), and the trigger data will be automatically converted to vector and back to trigger without loss of information.

Warmest regards,