[Python] How to containerize a FastAPI app using HuggingFace and spaCy without creating a huge...

I'm building a FastAPI application that processes and summarizes health reports using the following NLP tools:

HuggingFace Transformers (facebook/bart-large-cnn)

spaCy (en_core_web_sm, en_core_sci_sm)

PyTorch

Transformers

The app runs fine locally. But when I build a Docker image and preload the models, the final image size ends up being ~18GB, which seems too large for deployment.

Here’s what I’m trying to achieve:

Minimize image size (ideally under 2–3GB)

Avoid bundling model weights directly into the image

Download and cache HuggingFace and spaCy models into a volume on first run

Used python:3.11-slim as the base image

Installed dependencies from requirements.txt (including torch, transformers, and spacy)

Wrote a preload.py script that downloads models on startup

Mounted /models as a volume to store model weights

Added a .dockerignore file to avoid copying unwanted files

Questions: What’s the best practice for managing large NLP model files in a Dockerized app?

Should I use a CPU-only version of Torch to reduce size?

Is it standard to download HuggingFace and spaCy models at runtime into a mounted volume?

Are there other tools or techniques (multi-stage builds, BuildKit caching) I should be using?

Continue reading...