[Python] Python Flask App with LlamaIndex + Ollama application significantly slower in offline...

Problem


I have two nearly identical Python applications using LlamaIndex + Ollama for document Q&A:

• Online version: ~5 seconds response time
• Offline version: ~18 seconds response time FYI i am running the container after building the docker image for it on an internet-less env, so i am forcefully disabling hugging face downloads on container restart Environment

Both use the same model (llama3.2-vision), same PDF processing, same query ("Summarize"), but drastically different performance.

Setup Details

• Hardware: Same machine, same GPU
• Model: llama3.2-vision via Ollama
• Framework: LlamaIndex
• Document: Same PDF file (regular PDF, not scanned (Sturcture a.k.a does not need OCR))
• Processing: Both versions successfully extract text and generate responses

Code Comparison


Online Version (Fast - ~5s):

# Global LLM instance
llm = Ollama(model="llama3.2-vision", request_timeout=3600, temperature=0.0)
Settings.llm = llm
Settings.embed_model = "local:BAAI/bge-small-en-v1.5"





Offline Version (Slow- ~18-20s):

os.environ['HF_HUB_OFFLINE'] = '1'
os.environ['TRANSFORMERS_OFFLINE'] = '1'
os.environ['HF_HUB_DISABLE_TELEMETRY'] = '1'
os.environ['HF_HUB_DISABLE_PROGRESS_BARS'] = '1'
os.environ['HF_HUB_DISABLE_SYMLINKS_WARNING'] = '1'
os.environ['KMP_DUPLICATE_LIB_OK'] = 'TRUE'

# Set cache directories
os.environ['HF_HOME'] = '/app/.cache/huggingface'
os.environ['TRANSFORMERS_CACHE'] = '/app/.cache/huggingface'
os.environ['SENTENCE_TRANSFORMERS_HOME'] = '/app/.cache/sentence_transformers'
os.environ['XDG_CACHE_HOME'] = '/app/.cache'

....
embedding_model = HuggingFaceEmbedding(
model_name="BAAI/bge-small-en-v1.5",
cache_folder="/app/.cache/sentence_transformers",
device="cpu",
trust_remote_code=False,
embed_batch_size=1
)

# Configure LlamaIndex global settings
Settings.embed_model = embedding_model
Settings.chunk_size = 1024
Settings.chunk_overlap = 64
)


Questions

Why would creating new Ollama instances per session be 3-4x slower than reusing a global instance? Does Ollama have connection/initialization overhead that compounds with each new client?

Python 3.11 LlamaIndex 0.10.x Ollama running locally Docker container (offline version) NVIDIA GPU with CUDA

• I have tried ollama's nomnic embedding
• I pre-downloaded the models i.e hugging face embedding model and the ocr model and reference them in my python app in env variables
• added the offline env in docker file and built the image

Continue reading...