[Python] pandas chunksize - concat inside and outside the loops

I have to read massive csv files (500 million lines), and I tried to read them with pandas using the chunksize method, in order to reduce memory consumption. But I didn't understand the behaviour of the concat method and the option to read all the file and reduce memory. I'm adding some pseudocode in order to explain what I did so far.

Let's say I'm reading and then concatenate a file with n lines with:

iter_csv = pd.read_csv(file.csv,chunksize=n/2)
df = pd.concat([chunk for chunk in iter_csv])


Then I have to apply a function to the dataframe to create a new column based on some values:

df['newcl'] = df.apply(function)


Everything goes fine. But now I wonder what's the difference between the above procedure and the following:

iter_csv = pd.read_csv(file.csv,chunksize=n/2)
for chunk in iter_csv:
chunk['newcl'] = chunk.apply(function)
df = pd.concat([chunk])


In terms of RAM consumption, I thought that the second method should be better because it applies the function only to the chunk and not to the whole dataframe. But the following issues occur:

1. putting the df = pd.concat([chunk]) inside the loop returns me a dataframe with a size of n/2 (the size of the chunk), and not the full one;
2. putting the df = pd.concat([chunk]) outside, after the loop returns the same n/2 dataframe length.

So my doubt is whether the first method (concatenate the dataframe just after the read_csv function) is the best one, balancing speed and RAM consumption. And I'm also wondering how may I concat the chunks using the for loop.

Thanks for your support.

Continue reading...