[Python] How to adapt SAM2 for real-time object tracking using live video input?

I'm working on adapting SAM2 for real-time object tracking using a live video stream (e.g., from a webcam). SAM2 can currently track an object across a pre-recorded video when given an initial marker point.

Here’s what I have done so far:

I process the video stream every 2 frames to reduce computation.

For each step, I retain the previous frame and the current frame, and use SAM2 to perform tracking between the two.

I convert the output mask from SAM2 into a bounding point or object marker, and use that as the new input for the next tracking iteration.

Issues I'm facing: Tracking accuracy is poor — the object drifts or loses alignment quickly.

Performance is insufficient — even with frame skipping, SAM2's processing time per frame is too slow for real-time usage.

My questions: Is there a better approach to adapt SAM2 for continuous live tracking?

How can I optimize performance for real-time usage (e.g., reduce model size, simplify inputs)?

Are there alternative methods to maintain tracking accuracy across frames using SAM2's output (e.g., refining the mask-to-point conversion)?

Would combining SAM2 with a lightweight optical flow or motion estimation module help maintain consistency between frames?

img_pre = None
inference_state = None
prompts = {}

save_dir = "current_frames"
os.makedirs(save_dir, exist_ok=True)


prev_frame_path = os.path.join(save_dir, "0.jpg")
curr_frame_path = os.path.join(save_dir, "1.jpg")

for frame_idx, img_path in enumerate(image_paths):
img_bgr = cv2.imread(img_path)
if img_bgr is None:
print(f"fail reading: {img_path}")
continue



if img_pre is None:
cv2.imwrite(prev_frame_path, img_bgr)

inference_state = predictor.init_state(video_path=save_dir)
predictor.reset_state(inference_state)

ann_frame_idx = 0
prompts = {}


plt.figure(figsize=(9, 6))
plt.title(f"Frame {ann_frame_idx}")
plt.imshow(cv2.cvtColor(img_bgr, cv2.COLOR_BGR2RGB))


for i, box in enumerate(filtered_boxes_np):
obj_id = i
points = get_two_pt(box)
labels = np.array([1, 1], np.int32)


prompts[obj_id] = (points, labels)


_, out_obj_ids, out_mask_logits = predictor.add_new_points_or_box(
inference_state=inference_state,
frame_idx=ann_frame_idx,
obj_id=obj_id,
points=points,
labels=labels,
)



for j, out_obj_id in enumerate(out_obj_ids):

show_mask((out_mask_logits[j] > 0.0).cpu().numpy(), plt.gca(), obj_id=out_obj_id)

plt.axis('off')
plt.tight_layout()
plt.show()

else:
temp_bbox = []

cv2.imwrite(prev_frame_path, img_pre)
cv2.imwrite(curr_frame_path, img_bgr)

frame_names = [
p for p in os.listdir(save_dir)
if os.path.splitext(p)[-1] in [".jpg", ".jpeg", ".JPG", ".JPEG"]
]
inference_state = predictor.init_state(video_path=save_dir)
predictor.reset_state(inference_state)

ann_frame_idx = 0
prompts = {}



for i, box in enumerate(filtered_boxes_np):
obj_id = i
points = get_two_pt(box)
labels = np.array([1, 1], np.int32)


prompts[obj_id] = (points, labels)


_, out_obj_ids, out_mask_logits = predictor.add_new_points_or_box(
inference_state=inference_state,
frame_idx=ann_frame_idx,
obj_id=obj_id,
points=points,
labels=labels,
)


for j, out_obj_id in enumerate(out_obj_ids):
show_mask((out_mask_logits[j] > 0.0).cpu().numpy(), plt.gca(), obj_id=out_obj_id)




video_segments = {} # video_segments contains the per-frame segmentation results
for out_frame_idx, out_obj_ids, out_mask_logits in predictor.propagate_in_video(inference_state):
video_segments[out_frame_idx] = {
out_obj_id: (out_mask_logits > 0.0).cpu().numpy()
for i, out_obj_id in enumerate(out_obj_ids)
}

# render the segmentation results every few frames
vis_frame_stride = 1
#plt.close("all")
for out_frame_idx in range(0, len(frame_names), vis_frame_stride):
plt.figure(figsize=(6, 4))
plt.title(f"frame {out_frame_idx}")
plt.imshow(Image.open(os.path.join(save_dir, frame_names[out_frame_idx])))
for out_obj_id, out_mask in video_segments[out_frame_idx].items():
bbox, centroid = get_bbox_and_centroid(out_mask)
if out_frame_idx == len(frame_names)-1:
x1, y1, x2, y2 = bbox
temp_bbox.append(np.array([x1, y1, x2, y2]))

plt.scatter(bbox[0], bbox[1], c='blue', s=40, marker='o')
plt.scatter(bbox[2], bbox[3], c='green', s=40, marker='o')
plt.scatter(centroid[0], centroid[1], c='red', s=40, marker='o')
show_mask(out_mask, plt.gca(), obj_id=out_obj_id)

filtered_boxes_np = temp_bbox

img_pre = img_bgr


My goal is to track an object based on a user click (e.g., mouse input on a specific point of the object) — I do not want to rely on object detection models like YOLO or other class-based detectors for object selection. The user should be able to directly select the target in the video feed for tracking to begin.

Continue reading...