[Python] langchain / Chroma Process finished with exit code -1073741819 (0xC0000005) with

I am stumped with this problem

chunks = []
for path in file_paths: # path is a string filepath to a csv
chunks.extend(self.chunk_data(path))
chunks = filter_complex_metadata(chunks)
# add all relevant documents to chunks
# creates a unique cache for this dataset
cache_dir_name = "local_cache"
if not os.path.exists(cache_dir_name):
os.makedirs(cache_dir_name)
self.vector_store = Chroma.from_documents(documents=chunks,
embedding=FastEmbedEmbeddings(model_name="BAAI/bge-small-en",
cache_dir=cache_dir_name)) # crashes here


When it gets to the end it throws the following error:


Fetching 5 files: 0%| | 0/5 [00:00<?, ?it/s]C:\Users\...\venv\Lib\site-packages\huggingface_hub\file_download.py:140: UserWarning: `huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\Users\...\cache\models--Qdrant--bge-small-en. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `HF_HUB_DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.
To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development
warnings.warn(message)
Fetching 5 files: 100%|██████████| 5/5 [00:05<00:00, 1.18s/it]

Process finished with exit code -1073741819 (0xC0000005)


The weird thing is this code was working fine before. Then I cloned a new repo, reinstalled all dependencies, and now it is not. I am afraid it might be for some reason external to the codebase, however it is at the Chroma.from_documents where it fails.

The issue is exit code 0xC0000005 is an access violation, as far as I can tell, and this is really hard to debug..

Update: I activated developer mode in windows, now it still says Process finished with exit code -1073741819 (0xC0000005) at the same point but with no warnings...

Continue reading...