[Python] How can I efficiently filter and process large CSV files with pandas?

I'm working with a CSV file that has millions of rows and several columns. I need to:

Filter the rows based on certain conditions (e.g., age > 30 and salary > 50000). Perform some operations on the filtered data (e.g., add a new column that calculates the yearly bonus as 0.1 * salary). Save the processed data to a new CSV file. However, I keep running into memory issues because the file is too large to load entirely into memory. Here's what I've tried so far:

python

import pandas as pd

# Load the CSV file
df = pd.read_csv('large_file.csv')

# Filter rows
filtered_df = df[(df['age'] > 30) & (df['salary'] > 50000)]

# Add a new column
filtered_df['bonus'] = 0.1 * filtered_df['salary']

# Save to a new CSV file
filtered_df.to_csv('processed_file.csv', index=False)


This works well for smaller files, but when the file size grows, it crashes due to memory limitations.

I've read about dask and chunk processing, but I'm unsure how to implement them in my case. Could someone provide a clear example of how to process large CSV files efficiently?

Any help would be greatly appreciated!

Continue reading...