I'm currently trying to read data from .csv files in Python 2.7 with up to 1 million rows, and 200 columns (files range from 100mb to 1.6gb). I can do this (very slowly) for the files with under 300,000 rows, but once I go above that I get memory errors. My code looks like this: def getdata(filename, criteria): data=[] for criterion in criteria: data.append(getstuff(filename, criteron)) return data def getstuff(filename, criterion): import csv data=[] with open(filename, "rb") as csvfile: datareader=csv.reader(csvfile) for row in datareader: if row[3]=="column header": data.append(row) elif len(data)<2 and row[3]!=criterion: pass elif row[3]==criterion: data.append(row) else: return data The reason for the else clause in the getstuff function is that all the elements which fit the criterion will be listed together in the csv file, so I leave the loop when I get past them to save time. My questions are: How can I manage to get this to work with the bigger files? Is there any way I can make it faster? My computer has 8gb RAM, running 64bit Windows 7, and the processor is 3.40 GHz (not certain what information you need). Continue reading...