I have an ordered dataset(shuffle=False) that is categorised into "bins". I shall present an example on smaller scale that helps to clarify. Let's say the size of dataset is 60 with bins of sizes 10,20,30. I want to train my model in the order of bins. (first with 10 then 20 and 30). I want my DataLoader to get data in batch_sizes of 8. In this case, after getting the first 8 datapoints, I don't want to get the 2 remaining from bin-1 and get 6 from next one. What I want is to get only 2 and in the next iteration, get the 8 from bin-2. In short, I want to complete training in one bin first before moving to other. Also if batch_size happens to be greater than bin size, I want to get data in solely one bin before moving to next. Can I please get some advice on how to do this? I could think of two ways: implementing a custom DataLoader(need advice on this too) or just create separate DataLoaders for each bin and while iterating with bins in the outermost loop, grab the corresponding DataLoader and do training. Will the latter method have some serious downsides? Continue reading...
