1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
  2. Anuncie Aqui
    Anuncie aqui você Também: fdantas@4each.com.br

[Python] How can I split csv file read in langchain

Discussão em 'Python' iniciado por Stack, Setembro 11, 2024.

  1. Stack

    Stack Membro Participativo

    this is set up for langchain

    from langchain.text_splitter import RecursiveCharacterTextSplitter

    text_splitter=RecursiveCharacterTextSplitter(chunk_size=100,
    chunk_overlap=20,
    length_function=len)


    now I need to read a csv file

    import csv
    with open("test.csv") as f:
    # test is an iterator
    test=csv.reader(f,delimiter=",")


    this does not work because test is an iterator

    # object of type '_csv.reader' has no len()
    chunks=text_splitter.create_documents(test)


    text_splitter.create_documents accepts str. If I read a .txt file and pass it, it works. so I need to convert _csv.reader type to str. I tried

    chunks=text_splitter.create_documents("".join(test)


    I get

    ValueError: I/O operation on closed file.

    Continue reading...

Compartilhe esta Página