I am running the code from LeRobot and I am trying to control the robot arm via teleoperation and at the same time display the camera feed. After setting the config this can easily be done via: python lerobot/scripts/control_robot.py --robot.type=so100 -control.type=teleoperate However, when i do this the program gets stuck at displaying the camera feed. More specifically it gets stuck at the cv2.imshow(): if display_cameras and not is_headless(): image_keys = [key for key in observation if "image" in key] for key in image_keys: img = observation[key].numpy() print(key, img.shape, img.dtype) cv2.imwrite(f"{key}.png", cv2.cvtColor(observation[key].numpy(), cv2.COLOR_RGB2BGR)) cv2.imshow(key, cv2.cvtColor(observation[key].numpy(), cv2.COLOR_RGB2BGR)) cv2.waitKey(1) If I save the image as a .png it works perfectly and I am able to execute the rest of the code in parallel. However, when I try to display the image it gets stuck and never goes beyond the cv2.imshow(). As suggested online, I have tried to include cv2.waitKey(0) and cv2.destroyAllWindows() inside the loop, but this had no effect. When I try to use opencv to display an image in a separate script it works, but when I do it here it gets stuck. I am using an Intel realsense as a camera, but the camera works perfectly since I can get the output of it and save it. I am using a conda environment as suggested in the instructions with python 3.10.16 and opencv-python 4.11.0.86 on Ubuntu 22.04. Any help is appreaciated, thanks Most people online say that the problem is in calling the 'cv2.waitKey(0)` function. However, including this inside the loop doesn't have any effect. I have made sure to be in a non-headless environment, but as I said OpenCV is able to display the images in other scripts. I suspect it could be a problem related to threading. As this is not my code, I will probably need to open an issue on GitHub. Continue reading...
