[Python] How to release memory of intermediate Python-Polars DataFrames in a large dependency...

I am performing operations on a directed acyclic graph (DAG) of DataFrames using Polars (eager API).

Here’s a simplified example of my workflow:

1. Read parquet files into df1.
2. Use df1 to create df2 and df3.
3. Use df3 to create df4.
4. Use both df2 and df3 to create df5.

In reality, I have more than 200 such levels of DataFrame transformations to reach the final leaf nodes (output DataFrames).

Problem


Each DataFrame holds millions of rows and ~100 columns, so memory usage grows rapidly. I want to release memory for intermediate DataFrames as soon as they are no longer needed.

Example:

• Once df2 and df3 are created, I no longer need df1 → release it from memory.
• Once df4 and df5 are created, I no longer need df2 and df3.

I have tried:

del df1
import gc
gc.collect()


...but the memory is not being released.

Constraints

• 
  I am aware I could run the entire DAG execution in a subprocess, so that when the process finishes, memory is released.
  
  
• 
  However, I cannot wait for the entire graph to finish — I must free memory incrementally during execution because I run out of memory before completing the graph.
  

Question


How can I explicitly release memory of intermediate Polars DataFrames while processing a large DAG of transformations like this?

Continue reading...