[Python] Why does setting torch.backends.cudnn.deterministic = True make my TCN model...

I encountered a very interesting issue when training a deep learning model using Python and PyTorch. I designed a simple TCN as follows:

import torch
import torch.nn as nn
class TCN(nn.Module):

def __init__(self, input_size, output_size):
super(TCN, self).__init__()

self.convres = nn.Conv1d(input_size, output_size, kernel_size=1) ## 1x1 convolution
self.reluRes = nn.ReLU()

self.conv1 = nn.Conv1d(input_size, output_size, kernel_size=3, dilation=1, padding=1)
self.relu1 = nn.ReLU()

self.conv2 = nn.Conv1d(output_size, output_size, kernel_size=3, dilation=2, padding=2)
self.relu2 = nn.ReLU()

self.pool = nn.AdaptiveAvgPool1d(1)
self.fc = nn.Linear(output_size, 1)

def forward(self, x):
# Input shape: BatchSize x Seq_Len x FeatureSize
# Transpose to: BatchSize x FeatureSize x Seq_Len
x = x.permute(0, 2, 1)
residual = self.reluRes(self.convres(x))
x = self.relu1(self.conv1(x))
x = self.relu2(self.conv2(x))
x = x + residual
x = self.pool(x)
x = x.view(x.size(0), -1)
x = self.fc(x)
return x


I ensured reproducibility by setting the random seeds as follows:

import torch
import numpy as np
import random

def setup_seed(seed):
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
np.random.seed(seed)
random.seed(seed)
torch.backends.cudnn.deterministic = True


The problem: When I started training the model, I noticed an extremely slow training speed. Even after one or two hours, the first epoch's loss had not yet been printed. However, my GPU is an RTX 4090, so I expected the training to be very fast given that the model structure is quite simple.

To investigate, I did some testing:

1. I commented out one line in the forward pass:

x = self.relu2(self.conv2(x))

After doing this, the training suddenly became extremely fast (just a few minutes per epoch).

1. I realized the real problem was the following line:

torch.backends.cudnn.deterministic = True

When I set torch.backends.cudnn.deterministic = True, the training was excruciatingly slow. When I set it to False, the model trained very fast.

1. Another strange thing: If I kept torch.backends.cudnn.deterministic = True but commented out the second convolution (conv2), the training speed was normal again.

My question:

1. 
   Why does setting torch.backends.cudnn.deterministic = True cause the TCN model to train extremely slow? Even though the model only contains standard Conv1d layers, I did not expect such a dramatic performance drop.
   
   
2. 
   Why does commenting out one convolution layer make the model train faster, even when torch.backends.cudnn.deterministic = True?
   
   
3. 
   If I still want deterministic results (for reproducibility) without sacrificing training speed, how should I configure my code?
   

I have trained MLP, GRU, and other models with torch.backends.cudnn.deterministic = True before, and I never encountered such slow training. I am very curious about the underlying reason here.

Any insights or suggestions would be greatly appreciated. Thank you very much!

Continue reading...