1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
  2. Anuncie Aqui
    Anuncie aqui você Também: fdantas@4each.com.br

[Python] Langchain FastEmbed with ChromaDB

Discussão em 'Python' iniciado por Stack, Setembro 23, 2024.

  1. Stack

    Stack Membro Participativo

    I'm trying to follow a simple example I found of using Langchain with FastEmbed and ChromaDB. I will eventually hook this up to an off-line model as well. I believe I have set up my python environment correctly and have the correct dependencies. Just am I doing something wrong with how I'm using the embeddings and then calling Chroma.from_documents?

    from langchain_community.embeddings.fastembed import FastEmbedEmbeddings
    from langchain_community.document_loaders import PyPDFLoader
    from langchain_community.vectorstores.utils import filter_complex_metadata
    from langchain_community.vectorstores import Chroma
    from langchain.text_splitter import RecursiveCharacterTextSplitter
    import glob

    class ChatPDF:
    vector_store = None
    retriever = None
    chain = None

    def __init__(self):
    self.embeddings = FastEmbedEmbeddings()
    self.text_splitter = RecursiveCharacterTextSplitter(chunk_size=1024, chunk_overlap=100)

    def ingest(self, file_path: str):

    doc = PyPDFLoader(file_path=file_path).load()
    chunks = self.text_splitter.split_documents(doc)
    chunks = filter_complex_metadata(chunks)

    # generate vector store
    vector_store = Chroma.from_documents(documents=chunks, embedding=self.embeddings)




    new_chat = ChatPDF()
    docs_to_process = glob.glob("tmp/*.pdf")

    for pdf in docs_to_process:
    new_chat.ingest(file_path=pdf)


    When I run it however, I get the following:

    Traceback (most recent call last):
    File "/Users/c/Documents/Code/embeddings-test.py", line 35, in <module>
    new_chat.ingest(file_path=pdf)
    File "/Users/c/Documents/Code/embeddings-test.py", line 26, in ingest
    vector_store = Chroma.from_documents(documents=chunks, embedding=self.embeddings)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/c/Documents/Code/.venv/lib/python3.12/site-packages/langchain_community/vectorstores/chroma.py", line 878, in from_documents
    return cls.from_texts(
    ^^^^^^^^^^^^^^^
    File "/Users/c/Documents/Code/.venv/lib/python3.12/site-packages/langchain_community/vectorstores/chroma.py", line 842, in from_texts
    chroma_collection.add_texts(texts=texts, metadatas=metadatas, ids=ids)
    File "/Users/c/Documents/Code/.venv/lib/python3.12/site-packages/langchain_community/vectorstores/chroma.py", line 277, in add_texts
    embeddings = self._embedding_function.embed_documents(texts)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/c/Documents/Code/.venv/lib/python3.12/site-packages/langchain_community/embeddings/fastembed.py", line 117, in embed_documents
    embeddings = self._model.embed(
    ^^^^^^^^^^^^^^^^^
    AttributeError: 'NoneType' object has no attribute 'embed'

    Continue reading...

Compartilhe esta Página