I’m working on test automation using Robot Framework, and I’ve run into a challenge. My application runs inside a NoVNC environment, and the entire UI is rendered inside an HTML element. That means I don’t have access to individual DOM elements, since everything is drawn as a single image on the canvas. Right now, I’m using OpenCV to visually detect UI elements (buttons, icons, etc.). Once I find them, I get their coordinates and simulate clicks using tools like pyautogui, all wrapped into Robot Framework keywords. The problem: OpenCV detection isn’t always reliable. Small visual differences (resolution, scaling, color changes, etc.) can cause the element matching to fail. Element coordinates shift when the window is resized or when resolution changes, which makes the tests brittle. My question: Is there a more reliable way to detect or locate elements inside a canvas in this kind of setup (NoVNC)? Any alternative to OpenCV or tips to make this kind of image based detection more stable? Continue reading...
