[Python] Why is FAISS document retrieval slow and inconsistent on EC2 t3.micro instance?

I'm building a document Q&A system using FAISS for vector search on an AWS EC2 t3.micro instance (1 vCPU, 1GB RAM). My FAISS index is relatively small (8.4MB .faiss + 1.4MB .pkl files), but I'm experiencing inconsistent retrieval times - sometimes fast (<1s), sometimes very slow (>60s).

The Problem

After loading the FAISS index into memory, document retrieval is inconsistent:

Fast cases: <1 second response times

Slow cases: 60+ seconds for the same query

This is happening even though the index is cached in memory after the first load, so it shouldn't be loading from disk each time.

My Environment

EC2 Instance: t3.micro (1 vCPU, 1GB RAM)

Storage: EBS General Purpose SSD (gp2)

FAISS Version: Latest via LangChain

Index Size: 8.4MB (index.faiss) + 1.4MB (index.pkl)

Embeddings: OpenAI text-embedding-ada-002

Code Snippet

# Load vector store (cached after first load)
if context_file_path not in VECTOR_STORE_CACHE:
vector_store_path = os.path.join(VECTOR_STORE_ROOT, context_file_path)
VECTOR_STORE_CACHE[context_file_path] = FAISS.load_local(
vector_store_path,
embeddings,
allow_dangerous_deserialization=True
)

# Retrieve documents
vectorstore = VECTOR_STORE_CACHE[context_file_path]
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})
docs = retriever.get_relevant_documents(query) # This can take 60+ seconds


Questions

Why would FAISS retrieval be inconsistent for the same query on the same in-memory index?

Are there specific FAISS configuration options that would improve consistency?

Would using a different index type (IVF, HNSW) help with consistent performance?

Is 1GB RAM sufficient for FAISS operations of this size?

Could this be a garbage collection issue in Python?

Log Output

Loading vector store for OL_IGSE_Biology into memory...
Vector store for OL_IGSE_Biology cached in memory.
Starting document retrieval for query: 'who are organisms?...'
✅ Document retrieval completed in 66.19 seconds. Found 3 docs.


Any insights would be greatly appreciated. I expected document retrieval to be consistently fast with an in-memory index of this size, especially after the initial load.

Continue reading...