I have a dataset with more than 300 million rows and 7 columns. I want to compute rolling quantiles over lots of groups, but I run out of memory. I use the following code: ( lf.sort('time') .rolling( index_column='time', group_by=['id1', 'id2'], period="1mo", closed="left", ) .agg( pl.quantile('x', 0.1).alias('q10') ) ) There are around 17_000 different values in id1 and 20 in id2. I use sink_parquet instead of collect and the engine is "streaming". I have tried Hive partitioning the data by the time column, but that doesn't seem to make a difference. If I don't use group_by there is no memory issue, but it does take a loong time to run. I suppose I can split and save the dataset by the id columns to make it fit in memory, but it would be nice to avoid. Continue reading...
