[Python] Another OpenCV python question concerning masking of contours with bitwise operations

Pretty awful with OpenCV and have struggled with this part of my project. I've created a solver for the game Wordament, and from there have learned customtkinter and a lot of GUI stuff to make it a true Win10/11 app. I reached a place where I needed to get frustrated over python again so I decided to implement an OCR ability to fill in the initial entry array of my main window/solver rather than type it out. So that is the why and what.

Right now I am beating my head against all the possible ways to make it easier for EasyOCR or pytesseract to accurately read my input grid. This has been brutal and I have spent a good two months on this off and on, working on refining and thresholding and optimizing my image for processing. I've yet to re-train my own EasyOCR model specifically for Segoe-UI Sans (the font in question), but I am now trying to just mask out the digits on each tile, which gives OCR a lot of trouble. I also want to mask out the entire background (the space between the tiles) eventually and end up with a 4x4 grid of white letters on a black background. Here is my starting point, ignore the rectangle drawn on it, and here is my mask result:

(EDIT just realized this template crop isn't the matching one with the same digits, but I believe the idea is the same either way. Let me know if you need the actual matching grid template 4x4)





And here is my function, which will work outside the class if you take out the OOP self calls, but its very primitive and straightforward to read:

def template_matching_crop(self):
if not self.scrot_path or not os.path.isfile(self.scrot_path):
raise ValueError("Invalid screenshot path")

# Read the screenshot image
img = cv.imread(self.scrot_path)
if img is None:
raise ValueError("Failed to read screenshot")

# Convert the image to grayscale for matching, etc. etc.
img_gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY)
template = cv.imread(self.template_path, 0)
if template is None:
raise ValueError("Failed to read template")

res = cv.matchTemplate(img_gray, template, cv.TM_CCOEFF_NORMED)
_, max_val, _, max_loc = cv.minMaxLoc(res)

top_left = max_loc
h, w = template.shape
bottom_right = (top_left[0] + w, top_left[1] + h)

# ROI
cropped_region = img[top_left[1]:bottom_right[1], top_left[0]:bottom_right[0]]

# Convert the cropped region to grayscale, blur it, thresh
cropped_gray = cv.cvtColor(cropped_region, cv.COLOR_BGR2GRAY)

median_blur = cv.medianBlur(cropped_gray, 5)
_, thresh = cv.threshold(median_blur, 201, 255, cv.THRESH_BINARY)

# Find contours in the thresh
contours, _ = cv.findContours(thresh, cv.RETR_TREE, cv.CHAIN_APPROX_SIMPLE)

# Create a mask to blank out small contours, initialized to white (255)
mask = np.ones_like(cropped_gray) * 255

for contour in contours:
# Get the bounding box and area of conts
x, y, w, h = cv.boundingRect(contour)

area = cv.contourArea(contour)

# If the area is less than arbitrary N, mask it
if area < 180:
cv.drawContours(mask, [contour], 0, 0, -1)
plt.imshow(mask)

# This might be stupid
masked_cropped_region = cv.bitwise_or(cropped_gray, mask)

# Binarize the final image so the background (and mask) is white (255) and foreground is black (0)
_, cropped_region = cv.threshold(masked_cropped_region, 201, 255, cv.THRESH_BINARY)
final_image = cv.bitwise_not(cropped_region)
plt.imshow(final_image)

# Save the binarized cropped region, make my function whole within class, headbutt
processed_path = os.path.join(ASSETS + os.sep + "ocr" + os.sep + "cropped_region.png")
cv.imwrite(processed_path, final_image)
plt.show() #DEBUG
return cropped_region, processed_path


What I want is to mask over those little digits in the most efficient way, and combine my mask with my base image and end up with that 4x4 letter grid, sans digits. I don't know why I'm struggling with this part, but every attempt so far has landed me with completely blank boundingrects (the light blue), or just the digits or just a completely masked maxarea rectangle.

Continue reading...