[Python] What is the correct way to get a sample from a polars.LazyFrame in Python?

I'm dealing with a huge amount of data (from CC NEWS Dataset).

The data is stored in a parquet-file before execution. I must load it with pl.scan_parquet(path) as a pl.DataFrame wouldn't fit into memory.

What I tried so far:

import polars as pl

lf = pl.scan_parquet(path)
lf.select(pl.col("col_of_interest").sample(n=sample_size,seed=0)) \
.sink_parquet("sample.parquet") # (1) crashes

# (2) when above code is run without .sink_parquet():
lf.collect() # crashes


In my case, the column of interest is the content of a crawled webpage. Without this column, the LazyFrame could easily be collected as the resulting DataFrame is much smaller in size. When executing the above code with col_of_interest, even when sample_size = 1, the program crashes (I know for sure that a sample of size 1 fits into memory). When using a different column with much smaller data instead, the program executes without any complains.

My question: What is the correct (memory efficient) way to sample from a pl.LazyFrame?

Memory efficient solutions that sample when scanning (or reading) the data, are accepted as well.

Continue reading...