[Python] How do I create a Pytorch Dataset from multiple files where each file has multiple...

How do I create an dataset that reads in data from multiple files, but where each file has lots of rows or batches.

For example, I have a partitioned parquet dataset (created with pandas.to_parquet), with text or embeddings in each row.

The multiple batches per file, multiple file setup seems pretty common for text data, but for some reason all the examples for torch.utils.data.Dataset assume one observation per file (like for images) or 1 file for the entire corpus.

torchdata seems to have some tools for this, but this seems to be no longer maintained.

Is there an established module in pytorch or 3rd party one that handles this file structure?

Continue reading...