[Python] Most efficient way to use a large image dataset with Google Colab -- getting drive...

I'm using Google Colab to a classifier in PyTorch and my training dataset has ~30,000 JPEG images which I have stored in my Google Drive. Efficiently using this large amount of data with Colab and Drive has been a nightmare, primarily because my Google Drive tends to crash or "time out" when I try to read images from a folder.

These are the two approaches I've tried so far, and both have failed.

1. 
   Read images directly from Google Drive when __getitem__ is implemented, i.e., my PyTorch Dataset object looks something like:
   
   class Dataset(torch.utils.data.Dataset):
   def __init__(self, image_ids, labels):
   self.image_ids = image_ids
   self.labels = labels
   
   def __len__(self):
   return len(self.image_ids)
   
   def __getitem__(self, i):
   img_path = f'drive/MyDrive/images/{image_ids}'
   img = transforms.ToTensor()(PIL.Image.open(img_path)
   label = self.labels
   return img, label
   
   
   Thus, when __getitem__ is called, it reads the image from the folder images in my Google Drive (where all 30,000 images are stored). However, this creates the problem that when I create a DataLoader and loop over the minibatches, it gives me the error that a "Google Drive Timeout occurred", which, from research, seems to be what happens sometimes with large folders in Google Drive.
   
   
   [*]
   Create a TensorDataset: To circumvent the above issue, I thought I would create a TensorDataset. To do this, I have to first create a massive tensor of all 30000 training images, i.e., of shape (30000, 3, 128, 128) (each image is 3x128x128), which takes a little time to do. So, I can't do it each time I run my code—thus I try saving this large tensor, but this leads to memory issues in my Colab which crashes the Runtime. Plus, it's like 12 GB, so I'm sure this is not an efficient way.
   
   

How can I do this? The setup is very simple, but it's proving to be a bit annoying because Google Drive doesn't seem primed to do these things. I just have a folder of 30,000 images which I want to read as Torch tensors (in minibatches for training) into Colab. What's the best way of doing this, and/or how can I solve the problems in the approaches I discussed above?

My sense is that approach #1 is the most sensible, because it only reads into memory what needs to be read at a time. But for some reason, reading things from a Google Drive folder, which has many elements (this one has 30,000), leads to something called a "Google Drive timeout". The same process is trivial on my computer's CPU, but I need GPUs for training, so I need to be able to do this on Colab. I don't know how to solve this.

For the record, I use Colab Pro, so I have access to high-RAM runtimes.

Continue reading...