[Python] Chroma.from_documents() hangs indefinitely when loading documents split by newline

I'm working with LangChain and trying to create a vector database from a text file containing book descriptions and ISBNs. Each line in the text file is a separate book description, so I split the loaded text by newline. Here's my code snippet:

from langchain_community.document_loaders import TextLoader
from langchain.text_splitter import CharacterTextSplitter
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_chroma import Chroma
import torch

raw_documents = TextLoader('data/tagged_description.txt', encoding='utf-8').load()
text_splitter = CharacterTextSplitter(chunk_size=0, chunk_overlap=0, separator="\n") # split by newline
documents = text_splitter.split_documents(raw_documents)

device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Using device: {device}")

huggingface_embeddings = HuggingFaceEmbeddings(
model_name="sentence-transformers/all-mpnet-base-v2",
model_kwargs={'device': device}
)

db_books = Chroma.from_documents(
documents,
embedding=huggingface_embeddings
)


The problem is that the code hangs indefinitely on the line Chroma.from_documents(). The input file contains multiple lines (5231), each representing a book description with its ISBN. Splitting by newline should produce a list of documents, but the vector store creation never completes.

How can I debug or fix this issue?

Continue reading...