[Python] How to properly handle splitting text when extracting table data using Google Cloud...

I'm using Google Cloud Vision API's DOCUMENT_TEXT_DETECTION to extract text from an image that contains a table. My goal is to extract the table and save it as a CSV file. However, the output I'm getting has issues where some fields are split incorrectly, especially when there are spaces or multi-part words. Here's a comparison of the expected vs actual results:

Expected Result Actual Result:

As you can see, the fields like "ゲームタイトル" and cells like "パブリッシャー 1" and so on are being split into two separate columns such as "ゲーム", "タイトル" and "パブリッシャー", "1" when they should be in a single column. And the cells like "ゲーム番号1" are missing SPACE between text and number. This happens especially when there is a space or a long word. I tried a table with shorter text and it didn't have any problems.

I have a function that processes the bounding boxes returned by the Vision API and groups text into rows and columns based on proximity (x_threshold and y_threshold).

Here is a simplified version of my code for grouping the text into cells:

import requests
import io
import base64
import os
import pandas as pd
import csv

API_KEY = ''
image_path = ''

def prepare_request(image_path: str):
with io.open(image_path, 'rb') as image_file:
content = base64.b64encode(image_file.read()).decode('utf-8')

payload = {
"requests": [{
"image": {
"content": content
},
"features": [{
"type": "DOCUMENT_TEXT_DETECTION"
}]
}]
}
return payload

def run_cloud_vision(image_path: str):
request = prepare_request(image_path)

url = f"https://vision.googleapis.com/v1/images:annotate?key={API_KEY}"

response = requests.post(url, json=request)
result = response.json()

return result # Return JSON

def get_center(bounding_box):
vertices = bounding_box['vertices']
x = (vertices[0].get('x', 0) + vertices[2].get('x', 0)) / 2
y = (vertices[0].get('y', 0) + vertices[2].get('y', 0)) / 2
return x, y

def group_text_into_cells(result, x_threshold=50, y_threshold=20):
if 'responses' in result and 'textAnnotations' in result['responses'][0]:
annotations = result['responses'][0]['textAnnotations'][1:] # Skip the first element (it's the full text)

# Create a list to store each cell's information (x, y, text)
cells = []

for annotation in annotations:
x, y = get_center(annotation['boundingPoly'])
# text = annotation['description'].strip() # Trim whitespace
text = annotation['description']
cells.append((x, y, text))

# Sort cells by their y-coordinate (rows) and then by x-coordinate (columns)
cells.sort(key=lambda cell: (cell[1], cell[0])) # Sort by y first, then x

# Group cells into rows based on y-coordinate proximity
rows = []
current_row = []
last_y = None

for cell in cells:
x, y, text = cell
if last_y is None or abs(y - last_y) > y_threshold:
if current_row:
rows.append(current_row)
current_row = [cell]
else:
current_row.append(cell)
last_y = y

if current_row:
rows.append(current_row)

# Now, for each row, sort by x-coordinate and use x_threshold to create columns
table = []
for row in rows:
row.sort(key=lambda cell: cell[0]) # Sort by x-coordinate for columns

# Group text into columns based on x proximity
current_row_texts = []
current_column_text = ""
last_x = None

for cell in row:
x, y, text = cell
if last_x is None or abs(x - last_x) > x_threshold:
if current_column_text:
current_row_texts.append(current_column_text) # Add the previous column's text
current_column_text = text # Start a new column
else:
current_column_text += text # Append to the current column
last_x = x

if current_column_text:
current_row_texts.append(current_column_text) # Add the last column's text
table.append(current_row_texts)
return table
return []

def save_results_to_file(table):
if table:
df = pd.DataFrame(table)

csv_file_path = 'results.csv'
df.to_csv(csv_file_path, index=False, header=False, quoting=csv.QUOTE_NONE, escapechar='\\')

print(f"Results saved to {csv_file_path}")
else:
print("No text detected or no table structure found.")

if __name__ == '__main__':
result = run_cloud_vision(image_path)
table = group_text_into_cells(result)
save_results_to_file(table)


What I've tried is that I've adjusted the x_threshold and y_threshold values to group text correctly, but the problem still occurs, especially when dealing with longer or compound words. I suspect the issue may be related to how the Vision API splits text or how my code handles concatenating words when there is a small gap between them.

So my questions are:

1. How can I better handle the spacing between words so that longer terms like "ゲームタイトル" are not split incorrectly across multiple columns?
2. Are there any specific best practices for grouping text returned by the Vision API to ensure it doesn't get split unnecessarily?

Continue reading...