[Python] 3D reconstruction from multiple depth maps

I’m working on reconstructing a 3D object from multiple depth maps taken at different angles. The object is rotating on a turntable, and the hyperspectral camera is stationary. The depth data is stored in a 570x570 2D array for each photo, and I’m using rotation matrices to align the points. However, I’m running into issues with overlap and incorrect rotations when trying to merge the depth maps. The code to extract the depth information for every pixel from the camera works, but I’m struggling with properly aligning and merging the depth maps. How can I fix the rotation and merging of depth maps to get a correct 3D model? Note: I'm working in Python. Should I use something else than rotation matrices?

This is the point where I've got now. I'm able to convert the datacubes (that the hyperspectral camera generates after taking a picture), into 2D depth maps. The next step in my code will be to merge the depth maps into a 3D-shape, but that's where I'm stuck. Code:

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

# Settings
num_pictures = 10 # This is a choice of the user.
rotation_angle = 360 / num_pictures # Angle difference per picture
image_size = 570 # 2D-array is 570x570
camera_distance = 162 # Depth distance from camera to rotation-axis / center of object

# Simulate a object_distance_map (dummy data for tests)
object_distance_map = np.random.uniform(100, 300, (image_size, image_size))
object_distance_map -= camera_distance # Correctie toepassen

# Assenstelsel define
x_range = np.linspace(-image_size // 2, image_size // 2, image_size)
y_range = np.linspace(0, image_size, image_size)

X, Y = np.meshgrid(x_range, y_range)
Z = object_distance_map # Depth information

# 3D-puntenwolk initiëren
points_3d = np.column_stack((X.flatten(), Y.flatten(), Z.flatten()))

# Function to create a rotation matrix around the y-axis
def rotation_matrix_y(theta):
theta_rad = np.radians(theta)
return np.array([
[np.cos(theta_rad), 0, np.sin(theta_rad)],
[0, 1, 0],
[-np.sin(theta_rad), 0, np.cos(theta_rad)]
])

# Transform 3D points per rotation angle
all_points = []
for i in range(num_pictures):
theta = i * rotation_angle
R = rotation_matrix_y(theta)
rotated_points = points_3d @ R.T
all_points.append(rotated_points)

# Combine all points
all_points = np.vstack(all_points)

# Visualisation
fig = plt.figure(figsize=(10, 10))
ax = fig.add_subplot(111, projection='3d')
ax.scatter(all_points[:, 0], all_points[:, 1], all_points[:, 2], s=0.5, alpha=0.5)
ax.set_xlabel("X-axis")
ax.set_ylabel("Y-axis") # y-axis points upwarts
ax.set_zlabel("Z-axis") # z-axis should be depth-axis
ax.set_title("3D-reconstruction of object")
plt.show()


Note: object_distance_map is the 570x570 array, which contains the depth-value between camera and a certain pixel of 1 image. If we take 10 pictures, we have 10 maps as input.

Thanks for your time!

Continue reading...