[Python] How to average camera extrinsics each calculated from a target detected at different...

Let's say I have a given number n of cameras with known intrinsics. Each camera is synchronized, and each camera detected t frames with a calibration target that was rotated and moved around, seen simultaneously by every camera.

For each frame, I have some objp, imgp, where imgp are detected points of the calibration target. I am using them to compute extrinsics [R|t] (with cv2.solvePnP.)

How do I average together all this data? If I understand correctly, I can't simply do quaternion averaging and translation averaging on the raw results of cv2.solvePnP, since - as the target is rotated and moved around - the frame of reference will be different.

I tried changing the frame of reference first, like shown below:

def get_extrinsics_fixed_to_global_reference(R_ref, t_ref, R_other, t_other):

# Get the extrinsics of the other camera in the reference camera's coordinate system
R = R_other @ R_ref.T
t = t_other - R @ t_ref

return R, t


but I got garbage results. I am currently using a setup with n = 2 cameras.

Edit: To clarify, my idea was to essentially compare the resulting transformations between the coordinate systems of cam_0 and cam_k for k > 0. For each set of frames (corresponding to a calibration target tilted in a specific way) I would get a set of said transformations (cam_0 -> cam_k), and they should be in theory the same - this is where I would average out the rotation matrices to then finally get extrinsics. Note: I already know intrinsics for each camera.

Edit 2: I think I fixed it, below is a visualization of what I meant:

Continue reading...