[Python] What architecture should I use to extract all text from any image (OCR use case)?

I am working on a use case where I want to extract all visible text from any kind of image. This includes medicine packages, documents, product labels, and natural scenes. The goal is to build a model that outputs all the text present in the image.

I want to understand:

1. What model architecture is most suitable for this task?
2. Should I go with CNN + BiLSTM + CTC or consider Transformer-based models like TrOCR?
3. What are the preprocessing and dataset requirements to train such a model effectively?

I am looking to build and train this model myself (not just use an API like Tesseract or EasyOCR).

I tried,

1. 
   CRNN Model (CNN + BiLSTM + CTC loss):
   I implemented a basic CRNN using Keras. The model includes:
   
   • CNN layers for feature extraction
   • BiLSTM layers for sequence modeling
   • Dense + CTC loss for character decoding
2. 
   TrOCR by Microsoft:
   I tested a transformer-based OCR model using Hugging Face’s pretrained microsoft/trocr-base-stage1. However, it didn’t work well with noisy medicine images — accuracy was low for small text and unusual fonts.
   

What I’m expecting to know:

• Is CNN + BiLSTM + CTC still the best architecture for general-purpose OCR?
• Should I explore ViT + Transformer-based models for better performance?
• What image preprocessing and dataset formatting are ideal for training such a model?
• Is there a better way to align input image widths with sequence lengths for CTC training?

Any architecture suggestions, research paper links, or open-source code would be helpful.

Thanks in advance!

Continue reading...