[Python] getting the text and tokens using layoutlmv3

I trained a layoutlmv3 model creating a label dataset using label studio. i was able to test the output of the model using the following code

encoding = processor(image, words, boxes=boxes, word_labels=word_labels, return_tensors="pt")
for k,v in encoding.items():
print(k,v.shape)


And draw the boxes back on the pdf using below code

import torch

with torch.no_grad():
outputs = model(**encoding)
logits = outputs.logits
logits.shape
print (outputs)


predictions = logits.argmax(-1).squeeze().tolist()
print(predictions)
labels = encoding.labels.squeeze().tolist()
print(labels)
def unnormalize_box(bbox, width, height):
return [
width * (bbox[0] / 1000),
height * (bbox[1] / 1000),
width * (bbox[2] / 1000),
height * (bbox[3] / 1000),
]

token_boxes = encoding.bbox.squeeze().tolist()
width, height = image.size

true_predictions = [model.config.id2label[pred] for pred, label in zip(predictions, labels) if label != - 100]
true_labels = [model.config.id2label[label] for prediction, label in zip(predictions, labels) if label != -100]
true_boxes = [unnormalize_box(box, width, height) for box, label in zip(token_boxes, labels) if label != -100]

print(encoding.values)


What i am trying to understand is whether the predictions that the model gives out are just bounding boxes where i have to run teserract again on the boxes to extract text or does the model spit out the words with the associated labels.

Continue reading...