[Python] Polars `scan_parquet` doesn't seem to infer Hive schema of partitioned parquet dataset

I have a dataset created using pl.sink_parquet (* see note below) that is partitioned by date and part_id:

-dataset
|
-date=2009-08-01
|
-part_id=0
|
-0.parquet
-part_id=1
|
-0.parquet
-...
-part_id=N
|
-0.parquet
-date=2009-09-01
|
-part_id=0
|
-0.parquet
-part_id=1
|
-0.parquet
-...
-part_id=N
|
-0.parquet
-...
date=2023-01-01
|
...


I thought that I could load e.g. all parquets where part_id=0 "very quickly" using pl.scan_parquet("dataset", hive_partitioning=True).filter(pl.col("part_id") == 0), but it seems that Polars actually scans the entire dataset (which in my case amounts to ~60,000 files), which takes about 40 seconds, as opposed to just inferring that this should be a simple vertical concat of the part-0 parquets. If I manually pass a list of the full paths to scan_parquet, the entire load takes ~0.5 secs.

Is there something I'm missing in that perhaps this is not how scan_parquet is supposed to work?

Note: The dataset is created using multiple calls to sink_parquet where each time the argument to the function is a Polars DataFrame that only contains a single date. I'm still using the date column in the list of columns to partition by, however, so I'm not creating the folder structure manually.

I'm using Polars 1.31.0 on Linux.

Continue reading...