[Python] Reading a huge .csv file

Discussão em 'Python' iniciado por Stack, Abril 9, 2025.

  1. Stack

    Stack Membro Participativo

    I'm currently trying to read data from .csv files in Python 2.7 with up to 1 million rows, and 200 columns (files range from 100mb to 1.6gb). I can do this (very slowly) for the files with under 300,000 rows, but once I go above that I get memory errors. My code looks like this:

    def getdata(filename, criteria):
    data=[]
    for criterion in criteria:
    data.append(getstuff(filename, criteron))
    return data

    def getstuff(filename, criterion):
    import csv
    data=[]
    with open(filename, "rb") as csvfile:
    datareader=csv.reader(csvfile)
    for row in datareader:
    if row[3]=="column header":
    data.append(row)
    elif len(data)<2 and row[3]!=criterion:
    pass
    elif row[3]==criterion:
    data.append(row)
    else:
    return data


    The reason for the else clause in the getstuff function is that all the elements which fit the criterion will be listed together in the csv file, so I leave the loop when I get past them to save time.

    My questions are:


    1. How can I manage to get this to work with the bigger files?


    2. Is there any way I can make it faster?

    My computer has 8gb RAM, running 64bit Windows 7, and the processor is 3.40 GHz (not certain what information you need).

    Continue reading...

Compartilhe esta Página