[Python] How do I process CSV files from S3 using pandas without loading the entire file?

My current data processing flow looks like this

1. Load CSV
2. Pivot Data
3. Filter the original data based on some results from previous step
4. Repeat several times

I have this working on several CSV files in S3. But I am going to encounter files that can be over GBs in volume. I feel that it will take a lot of resources to load the entire data and process it. Is there a way I can achieve this without having to throw a lot of CPU and memory at the script?

Continue reading...