[Python] Behavior of camera-intrinsics when downscaling image. Use-case: Megapose...

I'm currently working on a full pipeline to estimate the pose of industrial metal objects on a conveyor using Megapose. For that I'm using a camera with a resolution of 1280x1024px. The pipeline consists

1. Acquiring an image in full resolution when an external trigger is applied
2. Downscaling the image to 640x512
3. Automatic detection of the object and masking + bbox (on downscaled image)
4. Estimation of the object pose (on downscaled image with bbox coordinates & .obj mesh as well as the camera intrinsics K-matrix)

For estimation I have the following code

cam = CameraData(K=K, resolution=(h, w))
obs = ObservationTensor.from_numpy(rgb=img, depth=None, K=cam.K).cuda()

OBJ_LABEL = "85589"
OBJ_MESH_PATH = "./85589/85589.obj"

obj_data = ObjectData(
label=OBJ_LABEL,
bbox_modal=np.array([x, y, x + bw, y + bh], dtype=np.float32)
)
detections = make_detections_from_object_data([obj_data]).cuda()

OBJ_RIGID = [
RigidObject(label=OBJ_LABEL,
mesh_path=Path(OBJ_MESH_PATH),
mesh_units="mm")
]
obj_dataset = RigidObjectDataset(OBJ_RIGID)

pose_estimator = load_named_model(
"megapose-1.0-RGB-multi-hypothesis",
object_dataset=obj_dataset
).cuda()

#with torch.no_grad():
data_TCO_final, data = pose_estimator.run_inference_pipeline(
observation=obs,
detections=detections,
n_refiner_iterations=3,
n_pose_hypotheses=10,
)

T = data_TCO_final.poses[0].cpu().numpy()


The rgb=img is the downscaled one (by using img = cv2.resize(img_full, (640, 512), interpolation=cv2.INTER_AREA))

At the end I visualize the outcome by drawing the estimatet coordinatesystem of the object onto the image. And here it gets interesting:

When I use the camera intrinsics, which I calibrated on 1280x1024 images, the outcome looks super nice. The Z-axes starts at the origin of the object defined by the cad-model and points in the correct direction. X- and Y-Axes are also fine (since I have rotation-symmetric objects the outcome differs from estimation to estimate but the plane of both fits the tilting of the object). The T-Matrix however is even in translation way off and does not work with the real-world settings (e.g. z-trans is around 15cm but it should be 70cm). Sure, I used the "wrong"-intrinsic parameters since they were for the high-quality image. Megapose is able to compensate that by thinking, that the Model is closer than it is in the real world.

Next step: scaling the K-matrix according to the image-scale. When I apply the factor of 0.5 to fx, fy, cy and cy the matrix should correspond correctly to the downscaled image (or am I wrong?). If I run the estimation with these scaled parameters not only the T-Matrix is way off but also the drawen axes do not fit the object at all (see pictures below).

I'm not quite sure what is happening here. Maybe I missed something? Maybe there is another way of downscaling images inside megapose itself? I sadly cannot test on full resolution with original intrinsics since GPU runs out of memory (even with only 1 hypothesis and 1 refiner iteration).

For visualization: with the "wrong" intrinsic parameters (original ones from 1280x1024). For reference, the axes should be 1mm long. The object is 26mm length...

with the downscaled intrinsic parameters:

I appreciate any ideas!

Edit: for reference, these are the K-matrices for the full and downscaled images

Full:

K = np.array([
[9569.51159, 0.0, 654.607481],
[ 0.0, 9561.13907, 519.54065],
[ 0.0, 0.0, 1.0]
], dtype=np.float64)


Downscaled:

K = np.array([
[4786.600316022559, 0.0, 324.6020298466702],
[0.0, 4784.247516721188, 235.07637840007217],
[0.0, 0.0, 1.0]
], dtype=np.float64)

Continue reading...