CLIP Issue I am preparing an AR gaze dataset and want to label the object a user is looking at. I have fisheye video frames (also tried undistorted). The global scene label from CLIP is reasonable (e.g., “hallway”), but when I try to make CLIP notice the gaze point by blending a subtle red Gaussian glow at the gaze location, the gaze object often gets misclassified (e.g., “curtains” instead of “door”). What I need A robust way to predict the object at the gaze point using CLIP (or a CLIP-friendly pipeline). Why is the red-dot/heatmap approach failing, and what’s a better pattern to follow? What I tried Red dot blending (Gaussian glow at gaze): same mislabels even after undistortion. Direct ROI crop around the gaze (without modifying pixels) and classify with CLIP: still sometimes “curtains” for a door. Continue reading...
