[Python] Group separate strings of an OCR-Result based on coordinates in the image

I use easyocr to read the key figures from an image (display output of measuring instrument). Because of different proportions of characters on the picture, some characters/strings, that are meant to be one unit, like value and unit (e.g "230 Volt"), are recognised as separate strings ("230", "Volt"). Another example are multiline strings, where each line is recognised as separate string. To illustrate it I prepared a picture. Its a little bit exaggerated but I hope it´s easy to understand.

Example picture to illustrate the problem

What I want to do


I try to find the elements that are on the same line or column (and very close to each other) and concatenate these strings.

Example data to handle (Output of easyocr)


(Coordinates are from top-left corner to bottom-left corner in clockwise direction)

([[239, 31], [563, 31], [563, 195], [239, 195]], '230', 0.7262734770774841)
([[591, 147], [661, 147], [661, 183], [591, 183]], 'Volt', 0.983400155647826)
([[801, 171], [1039, 171], [1039, 239], [801, 239]], 'This is a', 0.9870205241250117)
([[802, 256], [1232, 256], [1232, 328], [802, 328]], 'sentence with', 0.9997852752308181)
([[805, 341], [1065, 341], [1065, 427], [805, 427]], 'multiple', 0.9999849956753041)
([[212, 427], [311, 427], [311, 479], [212, 479]], 'Text', 0.9999873638153076)
([[362, 428], [474, 428], [474, 476], [362, 476]], 'More', 0.9999922513961792)
([[505, 413], [643, 413], [643, 479], [505, 479]], 'Text', 0.9999755620956421)
([[798, 428], [1136, 428], [1136, 500], [798, 500]], 'linebreaks.', 0.8525006562415545)
([[317, 601], [479, 601], [479, 669], [317, 669]], 'More', 0.9999911785125732)
([[529, 603], [665, 603], [665, 669], [529, 669]], 'Text', 0.9757571413464591)
([[699, 603], [841, 603], [841, 669], [699, 669]], 'with', 0.9999924302101135)
([[950, 608], [1182, 608], [1182, 683], [950, 683]], 'spaces.', 0.8026406194725301)

Output as DataFrame


I tried to handle it as Dataframe and split the values to x and y for each point. I though this view will help me. But i am still stucked

Text Score tl_x tl_y tr_x tr_y bl_x bl_y br_x br_y
0 230 0.726273 239 31 563 31 239 195 563 195
1 Volt 0.983400 591 147 661 147 591 183 661 183
2 This is a 0.987021 801 171 1039 171 801 239 1039 239
3 sentence with 0.999785 802 256 1232 256 802 328 1232 328
4 multiple 0.999985 805 341 1065 341 805 427 1065 427
5 Text 0.999987 212 427 311 427 212 479 311 479
6 More 0.999992 362 428 474 428 362 476 474 476
7 Text 0.999976 505 413 643 413 505 479 643 479
8 linebreaks. 0.852501 798 428 1136 428 798 500 1136 500
9 More 0.999991 317 601 479 601 317 669 479 669
10 Text 0.975757 529 603 665 603 529 669 665 669
11 with 0.999992 699 603 841 603 699 669 841 669
12 spaces. 0.802641 950 608 1182 608 950 683 1182 683

What output I want to achieve:


I`m happy with a list of concatinated strings

["230 Volt", "This is a sentence with multiple linebreaks.","More text with spaces", ...]

What I tried

• I tried to sort the column-values with exact or almost the same values to bins with pd.cut(). And access the strings over the bin-name, but I have no idea how to code it, so its not hardcoded and can be run automatically on different pictures.
• Also tried to use np.isclose() with relative tolerance to values to group them together
• tried looping with a lot of if-conditions, but nothing works.

Help


I am sure there is a very simple solution to this, just I am not good enough at programming yet to see it.

What would be the best approach to find the closest neighbours (on the same line/column) and group them together?

Continue reading...