Several Multi-Machines configuration problem

 

Hello,

I'm experiencing several problem in setup a multi-machine enviroment for a LED wall volume. To be more specific the satellite machines keep forgetting the setting, expecially for example the IP for Optitrack data stream, so it keep turning back to local IP. Sometimes the shared path also disappear too.

Also the channel matrix in the editor pc randomly invert the satellite machines, so it must be manually inverted to get the render in the right place. 

Another annoying thing is that I can't change camera calibration profile from editor pc, because the frustum doesn't update on satellite machines, so it have to be done in the setting for each macine before starting it up.

Can you also suggest me how to properly configure the shared path? For example if I write in this format "\\editor\project" (with double slash in front) it can't find the assets, so it must be written in this format "X:\editor\project" so I do have to assign a specific letter to a folder (map network drive) and this is a problem especially for those assets stored in drive C: of the main folder (like the libraries). So it seems like Aximmetry needs to have all paths with the same drive letter.

 Thank you.

   Merovingio

 
Profile Image
Merovingio
  -  

To add some context: these are the two channels that continue to invert randomly with each Aximmetry restart. Those are the satellite machines.



 
Profile Image
Eifert@Aximmetry
  -  

Hi,

I am not sure how the satellite machines would be able to forget settings like the IP of Optitrack's tracking.
Note, if you made Camera Presets: https://aximmetry.com/learn/virtual-production-workflow/preparation-of-the-production-environment-phase-i/multi-machine-environment/multi-machine-in-a-large-studio-environment/#defining-camera-presets-in-aximmetry then you have to give the same preset for all machines. This is because in LED wall setups, all video inputs are received by the control machine, and all machines must receive the same tracking data.
In other camera compounds, you need to set different Camera Presets per machine as they have their own separate video, tracking, and virtual camera perspective to render.


I am not exactly sure what you mean by invert. Do you mean these switch positions?

It could happen because they have the same monitor ID (...Monitor[1]...). Or Nvidia switches up the NDisplay1 and NDisplay2 numbering. When it is inverted, NDisplay1 and NDisplay2 are written in the same order as before being inverted?
Note, that you don't necessarily need to use NDisplay, you could probably do the same by using a Sticker module in the Flow Editor.


To ensure the camera calibration profile is read on all machines, in the shared projects folder create a folder named DeviceSetups manually. For example: \\Desktop-rmoet5re\MyProjects\DeviceSetups. More on it here: https://aximmetry.com/learn/virtual-production-workflow/preparation-of-the-production-environment-phase-i/multi-machine-environment/multi-machine-in-a-large-studio-environment/#ensuring-access-of-shared-device-configuration

To access the shared folder, your path must start with the name of the computer after the double slash. And then the name of the shared folder, for example: \\Desktop-rmoet5re\MySharedProjects
You can find the shared folder, by first finding its computer in the Network when adding the Root folder:

Also, you can use network drives: https://support.microsoft.com/en-us/windows/map-a-network-drive-in-windows-29ce55d1-34e3-a7e2-4801-131475f9557d#ID0EBD=Windows_10 Then the local path (X:\... instead of \\) should be sufficient.

Warmest regards,

 
Profile Image
Merovingio
  -  

I have some questions, before dive in this answer:

- Do the shared folder needs to have a specific name, or can be anything?

- Can I have two separate project folder (for example for use is "Project_Aximmetry" and "Project_Unreal")?

- It seems to me that not only the shared folders nedds to be configured in the same way for every machine, but also listed in the same order, is this correct?

- The "DeviceSetups" folder must be in "Project_Aximmetry" folder?

- Looks like when I click the browser in the "render setting" on the remote machine, to setup a path, it miss some of the network resources I see from outside the application, in the Windows browser. I don't know if this is Aximmetry related, but maybe it can explain why it randomly loose Optitrack IP?

We are quite stuck right now, something is unstable and I think is a mix Aximmetry Setting vs Windows Settings.

 
Profile Image
Eifert@Aximmetry
  -  

Hi,

Shared folders can have any name.

You can have two separate root project folders or as many as you like. But the simplest is if they are all under one folder including Aximmetry's libraries. And you set that folder as the root project folder in Aximmetry.

I don't think listing them in the same order matters.
Only when adding cameras in your Unreal Editor, you should have the first root folder which contains the Aximmetry libraries. But that doesn't matter in a multi-machine setup.

The DeviceSetups folder can be in a folder with any name. The only thing that matters is that it is a Root Projects folder.
But just in case, you should try to put it into the folder that is the first Project Root folder in Aximmetry.

That Network folder of Windows sometimes doesn't list all the computers. This is normal and shouldn't affect anything once you add the right folders.
If your Network folder doesn't list the computer that contains the shared folders, then you can type in your computer's name with \\ and it should list out all the shared folders:

If this doesn't work, then you should check this too if it works in the File Explorer instead of Aximmetry. If it works, then you should try running Aximmety as an administrator:

Warmest regards,

 
Profile Image
Merovingio
  -  

By the way, may I also ask you where the "remote render config" file are stored in? Which is the path?

 
Profile Image
TwentyStudios
  -  

@Eifert: I thought that the channel matrix assignment being switched randomly bug had been fixed in the latest update? If not, this should really be addressed ASAP. We had a really rough time with this on a LED wall production. 

 
Profile Image
Merovingio
  -  

Pretty sure we have tried any of the configuration above. We'll try again with a single shared folder, with everything inside, but I'm not sure this is the solution.

We had the system up and running, and suddenly with the same setting it starts loosing Optitrack IP on remote machines (it looks like it doesn't really save the settings), while sometimes the remote machine are just not present in the channel matrix (but the remote window still see it, send command but the output is black). Also randomly the system say that one of the machine resources are not shared. We have a brand new studio stucked in this situation.

Maybe some Windows/net settings? Ports? Firewall is down. Net resources are present. Some specific folder setting? We had a Perforce installation on the same machine, can it somehow interfere with Aximmetry? I'm just trying to think out of the box.

Please, we have a brand new studio stuck on this communication/setting problem.

@TwentyStudios I think we have the latest update, so maybe it is a bug and it is not yet fixed, I presume.

 
Profile Image
Eifert@Aximmetry
  -  

Hi,

Looking at this again, are these two different computers, and they are named as NDisplay1 and NDisplay2?

If so, you should make sure that computers have the same Index assigned as in previous startups in the control machine:


Also, you should make sure that you only have 1 active network card in your computers. And you are not actually connected to the same network from one computer through 2 different ways.
This could actually be the source of all the networking problems you are experiencing.
@TwentyStudios if you were referring to this, then there is no good solution to this other than not connecting a computer's two network cards into the same network. This can cause problems in other software too.


I am not familiar with Perforce or what any of their software can do.



You can export (Backup) and import (Restore) config in the Startup window using the Manage Setttings... option:

Note: these files are not really readable.



In LED Walls, we actually suggest using the Unified multi-machine setup:
https://aximmetry.com/learn/virtual-production-workflow/preparation-of-the-production-environment-phase-i/multi-machine-environment/multi-machine-setup/#channel-matrix

It makes setting up the outputs easier, as all computers will have the same indexes for outputs. The same as you would just use one computer for all the LED Walls.

Warmest regards,

 
Profile Image
Merovingio
  -  

Thank you for the reply.

As for the loosing of the Optitrack IP: I noticed that in the DeviceSetups folder there is an Optitrack.xml file, where it is stored the IP of the Adapter, which is the one it loose everytime. But of course any connected machine have a different Adapter, but since every machine reads from the same shared folder I suspect it writes one of the adapter and the second machine can0t find that ip and switch to local. 

Can this be the case? Hope it make sense.

 
Profile Image
Merovingio
  -  

I can confirm the problem of the lost of the Adapter IP was indeed because the shared folder "DeviceSetups" contain one optitrack.xml file, and every machine writes on this very file. Now i deleted the shared folder, and switched back to the original _DeviceSetups folder. But now I don't understand how remote machine can handle different settings, like this Adapter IP for example using the shared folder?

 
Profile Image
Eifert@Aximmetry
  -  

Hi,

You shouldn't edit any files.
You should use mapped settings on all your machines and when you select Mapped: #... then it will select that mapping in the remote machine.
More on Mapping here: https://aximmetry.com/learn/virtual-production-workflow/preparation-of-the-production-environment-phase-i/setting-up-inputs-outputs-for-virtual-production/mapping-devices/
In your case, you should set up mapping in your computers that point to a different Optitrack IP but with the same mapped ID.
So for example:
Computer 1-> Manage Devices: Mapped: #1 - Optitrack adapter IP 127.0.0.1
Computer 2-> Manage Devices: Mapped: #1 - Optitrack adapter IP 128.0.0.2
And in the control machine, you select Mapped: #1in the INPUT.

Warmest regards,