I have pretty big (up to ~300Gb) datasets stored by partitions in parquet format (compressed). I'm trying to find an efficient way to read parts (as defined by a set of filters) of the dataset into pandas. The way it's done now is result = ds.dataset(dataset_storage_root, format="parquet", partitioning='hive').scanner( columns=columns, filter=filters ).to_table().to_pandas() Although this works, it's rather slow (I assume this is due to the fact it actually reads the full dataset and only then applies filters, iterating over each row). And by rather slow I mean ~13 seconds, which is OK given the dataset size, but ridiculous given the actual amount of data I need to retrieve. Manually determining the subfolder for the data and reading just this part takes ~9ms for comparison. The downside is I need to manually add partition columns and values and handle quite a few corner cases with filtering and schemas. I would imagine there must be a way to do this via API and efficiently at that. What I've already tried to my disappointment: df_pandas = pd.read_parquet(dataset_storage_root, engine="pyarrow", filters=filters) Takes 1m 23s df_pq = pq.read_table(dataset_storage_root, filters=filters) Takes 1m 22s These take insane amount of time despite the claim is it should only read the subset specified by filters. Continue reading...
