[Python] What size should an `IterableDataset` report when used in a multi-worker `DataLoader`?

Here's a simple dataset and a data loader that uses it:

import torch
from torch.utils.data import DataLoader, IterableDataset


class Dataset(IterableDataset):
def __init__(self, size: int):
super().__init__()
self.size = size

def __iter__(self):
for i in range(self.size):
yield torch.tensor(, dtype=torch.float32)

def __len__(self):
return self.size


dataloader = DataLoader(Dataset(5), batch_size=1, num_workers=4)
result = list(dataloader)
print(len(result))


This prints the result:

/usr/local/share/luxcarta/venv/lib/python3.9/site-packages/torch/utils/data/dataloader.py:718: UserWarning: Length of IterableDataset <__main__.Dataset object at 0x77699d662fa0> was reported to be 5(when accessing len(dataloader)), but 6 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
warnings.warn(warn_msg)
/usr/local/share/luxcarta/venv/lib/python3.9/site-packages/torch/utils/data/dataloader.py:718: UserWarning: Length of IterableDataset <__main__.Dataset object at 0x77699d662fa0> was reported to be 5(when accessing len(dataloader)), but 7 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
warnings.warn(warn_msg)
/usr/local/share/luxcarta/venv/lib/python3.9/site-packages/torch/utils/data/dataloader.py:718: UserWarning: Length of IterableDataset <__main__.Dataset object at 0x77699d662fa0> was reported to be 5(when accessing len(dataloader)), but 8 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
warnings.warn(warn_msg)
/usr/local/share/luxcarta/venv/lib/python3.9/site-packages/torch/utils/data/dataloader.py:718: UserWarning: Length of IterableDataset <__main__.Dataset object at 0x77699d662fa0> was reported to be 5(when accessing len(dataloader)), but 9 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
warnings.warn(warn_msg)
/usr/local/share/luxcarta/venv/lib/python3.9/site-packages/torch/utils/data/dataloader.py:718: UserWarning: Length of IterableDataset <__main__.Dataset object at 0x77699d662fa0> was reported to be 5(when accessing len(dataloader)), but 10 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
warnings.warn(warn_msg)
/usr/local/share/luxcarta/venv/lib/python3.9/site-packages/torch/utils/data/dataloader.py:718: UserWarning: Length of IterableDataset <__main__.Dataset object at 0x77699d662fa0> was reported to be 5(when accessing len(dataloader)), but 11 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
warnings.warn(warn_msg)
/usr/local/share/luxcarta/venv/lib/python3.9/site-packages/torch/utils/data/dataloader.py:718: UserWarning: Length of IterableDataset <__main__.Dataset object at 0x77699d662fa0> was reported to be 5(when accessing len(dataloader)), but 12 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
warnings.warn(warn_msg)
/usr/local/share/luxcarta/venv/lib/python3.9/site-packages/torch/utils/data/dataloader.py:718: UserWarning: Length of IterableDataset <__main__.Dataset object at 0x77699d662fa0> was reported to be 5(when accessing len(dataloader)), but 13 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
warnings.warn(warn_msg)
/usr/local/share/luxcarta/venv/lib/python3.9/site-packages/torch/utils/data/dataloader.py:718: UserWarning: Length of IterableDataset <__main__.Dataset object at 0x77699d662fa0> was reported to be 5(when accessing len(dataloader)), but 14 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
warnings.warn(warn_msg)
/usr/local/share/luxcarta/venv/lib/python3.9/site-packages/torch/utils/data/dataloader.py:718: UserWarning: Length of IterableDataset <__main__.Dataset object at 0x77699d662fa0> was reported to be 5(when accessing len(dataloader)), but 15 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
warnings.warn(warn_msg)
/usr/local/share/luxcarta/venv/lib/python3.9/site-packages/torch/utils/data/dataloader.py:718: UserWarning: Length of IterableDataset <__main__.Dataset object at 0x77699d662fa0> was reported to be 5(when accessing len(dataloader)), but 16 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
warnings.warn(warn_msg)
/usr/local/share/luxcarta/venv/lib/python3.9/site-packages/torch/utils/data/dataloader.py:718: UserWarning: Length of IterableDataset <__main__.Dataset object at 0x77699d662fa0> was reported to be 5(when accessing len(dataloader)), but 17 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
warnings.warn(warn_msg)
/usr/local/share/luxcarta/venv/lib/python3.9/site-packages/torch/utils/data/dataloader.py:718: UserWarning: Length of IterableDataset <__main__.Dataset object at 0x77699d662fa0> was reported to be 5(when accessing len(dataloader)), but 18 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
warnings.warn(warn_msg)
/usr/local/share/luxcarta/venv/lib/python3.9/site-packages/torch/utils/data/dataloader.py:718: UserWarning: Length of IterableDataset <__main__.Dataset object at 0x77699d662fa0> was reported to be 5(when accessing len(dataloader)), but 19 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
warnings.warn(warn_msg)
/usr/local/share/luxcarta/venv/lib/python3.9/site-packages/torch/utils/data/dataloader.py:718: UserWarning: Length of IterableDataset <__main__.Dataset object at 0x77699d662fa0> was reported to be 5(when accessing len(dataloader)), but 20 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
warnings.warn(warn_msg)
20


I don't get it. The dataset reports its correct size. I get that each worker will iterate until its end, but then how should I "configure" it in each worker? Should it report a length of self.size * 4 so that the DataLoader sees the length that all workers together will see? Should I modify it after the fact in each worker so that each worker gets a dataset that's 4 times smaller?

Why is the DataLoader expecting len(dataset) to be equal to number_of_samples * worker_size when it's not the dataset that handles the workers?

Continue reading...