If you want to move the camera freely 6DOF you need to calibrate using the 6DOF calibrator. If the tracking data from the jib isn’t accurate (consistent), you won’t be able to calibrate it accurately with the 6DOF method either, since it relies on accurate tracking for this part of the calibration.

It looks like your pan/tilt calibration is already very accurate, but without reliable 6DOF data from the jib you will never be able to get good results, no matter how much you calibrate it. 

Hi,

If the jib can send the PTZ position (camera rotation) data and jib position data (arm's position) separately to Aximmetry, then:

You should make a PTZ calibration with the PTZ position data. And set the PTZ rotation center height to the height difference between the Jib position data and the actual height of the PTZ rotation center. And do the calibration without causing changes in the Jib position data.
In the Tracked camera compound set the Tracking Device to your PTZ position data and the Zoom Device to Jib position data.

In this case, the Zoom delay will apply to the Jib's position data.

If the jib adds together the PTZ and jib position and sends it to Aximmetry, then:

You should try making 6DOF calibration work. You might be able to place the calibration board in a position where the jib can reach every calibration position.

Otherwise, you could do the PTZ calibration and then in Aximmetry edit the camera compound so that it uses the jib's position. It won't be perfect, but it will be probably better than what is in your video. I can show you how to edit the camera compound for this.
But first, you should make sure that the tracking data you get from jib is accurate enough as @TwentyStudios said. You can write out the tracking data like this in Aximmetry's Flow Editor:

You should check if the position changes are the same as in the real world.

Warmest regards,