[Python] Polars dataframe is keep on holding memory

I am running my script in while True.

In every hour it collect data from parquet and process it. It consumes around 50% of RAM, after completing the process I use del to delete the dataframe and do gc.collect() to collect any free memory but still not working.

I am using lazy dataframe to collect data.

Here is my sample code

pa_dataset = pyarrow.dataset.dataset(source=dataset_path, format="parquet", partitioning="hive"
df1 = pl.scan_pyarrow_dataset(pa_dataset, allow_pyarrow_filter= True).filter((pl.col("timestamp") > start_date) & (pl.col("timestamp") < end_date))\
.select('sourceIPAddress', 'destinationIPAddress', \
'sourceTransportPort', \
'destinationTransportPort', \
'protocol', 'numPacketsCountIn', \
'numPacketsCountOut')\
.collect()
del df1
del pa_dataset
gc.collect()


My problem is it should release the acquired memory but it does not What should I do for that to happen?

Continue reading...