[Python] Image matching fails with low confidence using pyautogui when script is triggered...

I'm working on automating GUI testing using Robot Framework and PyAutoGui. I tried both pyautogui.locateOnScreen() and cv2.matchTemplate() to detect UI elements by matching a reference image inside a screen region.

The reference image is visibly present inside the screenshot (confirmed manually), but both approaches either fail or return very low confidence (~0.21), and no click is triggered.

• I'm running the Python script using Robot Framework’s Run Process keyword.

${result}= Run Process python OCR.py GPIOOutput ${reference_image} stdout=stdout.log stderr=stderr.log

• The image path is passed via sys.argv.

if __name__ == "__main__":
if len(sys.argv) < 3:
print("[ERROR] Usage: python OCR.py <word> <confidence>")
else:
word = sys.argv[1]
reference_image = sys.argv[2]
result = find_and_click_text(word, reference_image)
print(result)


Here's what works:

• If the path is hardcoded inside the .py script → locateOnScreen() works perfectly.
• If I pass the image path via sys.argv, and print it → it looks identical.
• Image.open(reference_image) works — file exists and loads fine.

Here's what fails:

• pyautogui.locateOnScreen(reference_image) → fails silently or gives low confidence (~0.2)
• Even if I pass the loaded image object (Image.open(path)) to locateOnScreen() → it fails
• OpenCV's cv2.matchTemplate() also gives low confidence (~0.21) even though the reference image clearly appears in the screenshot.

.py File Output:

[DEBUG] Reference image path received: C:\PFE\Test_Automation\CubeMX\PA6_reference.png
[DEBUG] sys.argv: ['OCR.py', 'GPIOOutput', 'C:\\PFE\\Test_Automation\\CubeMX\\PA6_reference.png']
[DEBUG] File exists at: C:/PFE/Test_Automation/CubeMX/PA6_reference.png
[ERROR] Traceback (most recent call last):
File "C:\Python312\Lib\site-packages\pyautogui\__init__.py", line 172, in wrapper
return wrappedFunction(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python312\Lib\site-packages\pyautogui\__init__.py", line 210, in locateOnScreen
return pyscreeze.locateOnScreen(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python312\Lib\site-packages\pyscreeze\__init__.py", line 405, in locateOnScreen
retVal = locate(image, screenshotIm, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python312\Lib\site-packages\pyscreeze\__init__.py", line 383, in locate
points = tuple(locateAll(needleImage, haystackImage, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python312\Lib\site-packages\pyscreeze\__init__.py", line 257, in _locateAll_opencv
raise ImageNotFoundException('Could not locate the image (highest confidence = %.3f)' % result.max())
pyscreeze.ImageNotFoundException: Could not locate the image (highest confidence = 0.274)


What I’ve Confirmed:

• The problem isn't solved even when I limit the search area using pyautogui.screenshot(region=(...))
• ImageChops.difference() confirms the reference and cropped screen area are pixel-perfect
• The issue happens only when the script is run from Robot Framework using Run Process
• The reference image isn't accessed after running the script.

Screenshots: The first image is my screen and the second is the reference image.




Could this be due to:

• pyautogui behaving differently when run as a subprocess?
• Robot Framework receiving more than one argument?

Thanks in advance — I’m happy to share any additional information (code + screenshots) if needed.

Continue reading...