[Python] Polars for Python, can I read parquet files with hive_partitioning when the directory...

I manually created directory structures and wrote parquet files rather than used the partition_by parameter in the write_parquet() function of the python polars library because

• I want full control over the parquet file naming
• I want to handle what to do in the advent of existing parquet files in the sub-directories (in my case I merge data and deduplicate)

Question: Can I use the pl.scan_parquet() function with hive_partitioning set to True and benefit from filtering on lazy dataframes and parallel reads and automated file discovery (using glob pattern) even though the files were written manually not using the partition_by parameter?

Thanks

Continue reading...