[Python] Some of the steps have been OOM killed. How to avoid this?

I am doing image classification in pytorch and use the adversarial robustness toolbox (https://adversarial-robustness-toolbox.readthedocs.io/en/latest/index.html). This framework wants me to give my entire dataset as parameters to a wrapper function. But loading the entire dataset leads to OOM errors as I use the ImageNet 2012 dataset as training data which is 155GiB but I only have 28 GB of Memory.

My idea was to not use the entire dataset at once but use a for loop and each instance of the for loop load a part of the dataset and pass it to the wrapper. However even after only loading 1/200th of data at a time into the array I pass to the wrapper eventually I run out of memory.

for a in range((len(filelist)//MEMORYLIMITER)+1):
print('Imagenet segement loaded: ' +str(a))
if ((a+1)*MEMORYLIMITER-1<len(filelist)):
x_train = np.array([np.array(Image.open(IMAGENET_PATH_TRAIN+'/'+fname)) for fname in filelist[a*MEMORYLIMITERa+1)*MEMORYLIMITER-1]])
x_train = np.transpose(x_train, (0, 3, 1, 2)).astype(np.float32)
x_train = x_train/255
print('load was successful: '+ str(a))

#pass x_train to wrapper
else:
x_train = np.array([np.array(Image.open(fname)) for fname in filelist[a*MEMORYLIMITER:]])
x_train = np.transpose(x_train, (0, 3, 1, 2)).astype(np.float32)
x_train = x_train/255
#pass x_train to wrapper


filelist is a list holding the filenames of all images MEMORYLIMIT is a int that says how many pictures there can be per 'slice' (total 1,281,167)

Is there a way to free the memory from the loaded images in python after I passed them to the wrapper?

I tried to delete the x_train array manually adding

del x_train
gc.collect()


after passing it to the wrapper but still I run out of memory.

Continue reading...