[Python] Understanding why solvePnPRansac could retrieve bad pose estimate and how to correct...

I am trying to get pose estimation of the front face of the box (image below) using only the minimum number of points (i.e 4) by using opencv cv2.solvePnPRansac.

Here are my inputs:

• The 2D pixel point set is chosen by hand and are labelled in white.
• For the 3D coordinates, I have supposed that the dimension of the box is 3x4x6 (whatever unit) and the origin is the point at top-left (pixel_coord = [1065, 602]).
• The distortion coefficient are simply np.zeros((5,1)).astype('float32')
• The Camera intrinsic parameter are given on this github notebook.

I tried to verify the outputed rvec & tvec with cv2.projectPoints. The new 2D reprojected points are plotted on the figure below and labelled in red.

As it can be seen, a first set of 3 points reproject perfectly (white and red label color are superimposed) but the 4th point's re-projection is very bad..

My questions are:

Is this behavior "normal" ? I also tried using more than four 3D-2D corresponding points, but results are getting worse! Is there something wrong with my method ? (or the code ?) Is there some iterative work to do ?

import os
import numpy as np
import matplotlib.pyplot as plt
import cv2

# some utils plot function:
def plot_points(ax, pts_list, c):
for pt in pts_list:
pt_x = pt[0]
pt_y = pt[1]
pt_label = (str(int(pt_x)) + ' , ' + str(int(pt_y)))
ax.scatter(pt_x, pt_y, s=80, facecolors='none', edgecolors=c)
ax.text(pt_x+5, pt_y+5, pt_label, fontsize=15, color=c)

def project_planes(ax, src, c):
x_src = [val[0] for val in src] + [src[0][0]]
y_src = [val[1] for val in src] + [src[0][1]]
ax.plot(x_src, y_src, '--', c=c)

# read image
image1 = cv2.imread('image1.jpg') # the image can be found on the github link above.
image1 = cv2.cvtColor(image1, cv2.COLOR_BGR2RGB)

# camera matrix
mtx = np.array([
[2448.0, 0, 1253.0],
[0, 2438.0, 986.0],
[0, 0, 1.0]
]).astype('float32')

# distortion coefficient
dst = np.zeros((5,1)).astype('float32')

# Object Dimension: 3x4x6
box3D_visible = np.array([
[0, 0, 0],
[0, 4, 0],
[3, 4, 0],
[3, 0, 0],
# [0, 4, 6],
# [0, 0, 6],
])

box2D_visible = np.array([
[1065, 602], # [1080,598],
[1074, 1342], # [1094,1340],
[1774,1086],
[1840,478],
# [504, 900],
# [424, 356]
])

# Pose estimate
_, rvec, tvec, _ = cv2.solvePnPRansac(
box3D_visible[:,:,np.newaxis].astype('float32'),
box2D_visible[:,:,np.newaxis].astype('float32'),
mtx,
dst
)

#Reprojection
box2D_reproj = cv2.projectPoints(
box3D_visible[:,:,np.newaxis].astype('float32'),
rvec,
tvec,
mtx,
dst
)[0][:,0,:]

# Plots
fig, ax = plt.subplots()
plt.imshow(image1)
#
box_face1 = box2D_visible[:4]
project_planes(ax, box_face1, 'w')
plot_points(ax, box_face1, 'w')
#
project_planes(ax, box2D_reproj, 'r')
plot_points(ax, box2D_reproj, 'r')
plt.show()



I want to get the reprojection of the full parallelepiped (box skeleton) onto this image.



Continue reading...