[Python] Save to disk training dataset and validation dataset separately in PyTorch

I want to save train dataset, test dataset, and validation dataset in 3 separate folders.

Doing this for training and testing is easy

# Get training and testing data
all_training_data = getattr(datasets, config['Pytorch_Dataset']['dataset'])(
root= path_to_data + "/data_all_train_" + config['Pytorch_Dataset']['dataset'],
train=True,
download=True, # If present it will not download the data again
transform=ToTensor()
)
test_data = getattr(datasets, config['Pytorch_Dataset']['dataset'])(
root= path_to_data + "/data_test_" + config['Pytorch_Dataset']['dataset'],
train=False,
download=True, # If present it will not download the data again
transform=ToTensor()
)


This code makes use of torchvision.datasets to load and save to disk the dataset specified in config['Pytorch_Dataset']['dataset'] (e.g. MNIST). However there is no option to load a validation set this way, there is no validation=True option.

I could split the train dataset into train and validation with torch.utils.data.random.split, but there are two main problems with this approach:

• I don't want to save the folder data_all_train, I want to save only 2 folders, one with the true training part and one with the validation part
• I would like PyTorch to understand if data_train and data_validation are present, and in this case it should not download again data_all_train, even if not present

Continue reading...