[Python] Polars LazyFrame streaming directly to (partitioned) Parquet without collecting

I am reading some data from AWS S3 with polars.scan_pyarrow_dataset. When I am finished with my data processing, I would like to write the results back to cloud storage, in partitioned Parquet files. However, I'd like to avoid collecting all results in memory. I believe this should be possible with the Parquet format because of its support for a row groups, as well as the ability to split the data into physically separate files.

I don't see any support for this in the Polars LazyFrame API. I see that collect has streaming=True, but that still looks like it collects the final result in memory.

Is there some way to achieve this that I am not seeing?

Continue reading...