[Python] How to efficiently process a large CSV file with pandas when memory is limited?

I'm working with a very large CSV file (around 10GB) that doesn't fit into my computer's memory. When I try to load it into a pandas DataFrame using pd.read_csv(), I get a MemoryError.

What's the most efficient way to process this file using pandas without loading the entire thing into memory at once? I need to perform operations like:

• Calculating the sum of a specific column.
• Filtering rows based on certain conditions.
• Grouping and aggregating data.

My code:

import pandas as pd
import numpy as np

# Simulate a large CSV file
with open('large_file.csv', 'w') as f:
for i in range(10000000): # 10 million rows
f.write(f'{i},{np.random.rand()},{np.random.randint(0, 10)}\n')


# This line causes a MemoryError
df = pd.read_csv('large_file.csv', names=['id', 'value', 'category'])

# Desired operations (example)
# total_value = df['value'].sum()
# filtered_df = df[df['category'] > 5]
# grouped_df = df.groupby('category')['value'].mean()

print("Processing Complete") # Never Reaches Here.


I've tried using chunksize in pd.read_csv(), but I'm not sure how to efficiently perform the desired operations on the chunks and combine the results. Are there other techniques or libraries I should consider?

Continue reading...