[Python] Memory leak due to accumulating threads after add_documents with Pinecone

I have a question regarding thread behavior when embedding data into Pinecone using LangChain.

When I call the add_documents method, I notice that seven new threads are spawned—which seems expected. However, after the operation completes, these threads remain alive. With each subsequent request, additional threads are created and also remain active. Over time, this leads to an accumulation of threads and increased RAM usage, eventually causing my service to crash in production.

I’ve ensured that both the Pinecone vector store and embedding model instances are implemented as singletons, so they shouldn’t be re-instantiated per request. You will see from each request after operation add_documents more threads are spawned and never reused or killed after it. I also wrapped the function with ThreadPoolExecutor but still the same results.

Has anyone encountered this behavior or have any suggestions on how to handle thread cleanup or prevent this leak?

Any insights would be greatly appreciated!

pages = text_splitter.create_documents([text_input])
print(f"Active threads 2 : {[t.name for t in threading.enumerate()]}")
combined_training_text = ""

for doc in pages:
combined_training_text += doc.page_content + " " # Add space between texts if needed

characters_count = count_characters(combined_training_text)
print(f"Active threads 3 : {[t.name for t in threading.enumerate()]}")
pinecone_index = select_pinecone_manager(llm)
print(f"Active threads 4 : {[t.name for t in threading.enumerate()]}")
vectorstore = pinecone_index.get_vectorstore(namespace_id)
print(f"Active threads 5 : {[t.name for t in threading.enumerate()]}")
vector_ids = vectorstore.add_documents(documents=pages)
print(f"Active threads 6 : {[t.name for t in threading.enumerate()]}")
return vector_ids, characters_count, combined_training_text[:1000]

Logs: Active threads 4 = ['MainThread', 'Thread-1 (start mq)', 'ThreadPoolExecutor-0_0', 'Thread-3 (worker)', 'Thread-4 (worker)', 'Thread-5 (worker)', 'Thread-6 (worker)', 'Thread-7 (handle_workers)', 'Thread-8 (handle_tasks)', 'Thread-9 (handle_results)'] Active threads 5 = ['MainThread', 'Thread-1 (start mq)', 'ThreadPoolExecutor-0_0', 'Thread-3 (worker)', 'Thread-4 (worker)', 'Thread-5 (worker)', 'Thread-6 (worker)', 'Thread-7 (handle_workers)', 'Thread-8 (handle_tasks)', 'Thread-9 (handle_results)'] Active threads 6 = ['MainThread', 'Thread-1 (start mq)', 'ThreadPoolExecutor-0_0', 'Thread-3 (worker)', 'Thread-4 (worker)', 'Thread-5 (worker)', 'Thread-6 (worker)', 'Thread-7 (handle_workers)', 'Thread-8 (handle_tasks)', 'Thread-9 (handle_results)', 'Thread-10 (worker)', 'Thread-11 (worker)', 'Thread-12 (worker)', 'Thread-13 (worker)', 'Thread-14 (handle_workers)', 'Thread-15 (handle_tasks)', 'Thread-16 (handle_results)']
​

Continue reading...