I am loading a large Parquet file with pyarrow, and then convert it to a Pandas DataFrame. Since this can be very memory-intensive, I need to see if loading the entire file in one go can fit into the memory in a single go, or if I should split it in batches (and if so, how big the batches should be). At the moment, the best solution I could find is to read the first row, convert it to a DataFrame, and get its size, then multiply it by the amount of rows: pf = ParquetFile(...) df = next(pf.iter_batches(batch_size=1)).to_pandas() total_size = df.memory_usage(deep=True).sum() * pf.metadata.num_rows I'm relying on the fact that, in my case, the values will always be the same (float and int), so this works for me. But I don't think this is really the most efficient solution here. My wish would be to work this out of the metadata alone. Is this really the best I can do to predict how big the load operation will actually turn out to be? Continue reading...
