[Python] RuntimeError: CUDA error: no kernel image is available for execution on the device...

I am working on this model:

class Model(torch.nn.Module):
def __init__(self, sizes, config):
super(Model, self).__init__()

self.lstm = []
for i in range(len(sizes) - 2):
self.lstm.append(LSTM(sizes, sizes[i+1], num_layers=8))
self.lstm.append(torch.nn.Linear(sizes[-2], sizes[-1]).cuda())
self.lstm = torch.nn.ModuleList(self.lstm)

self.config_mel = config.mel_features

def forward(self, x):
# convert to log-domain
x = x.clip(min=1e-6).log10()

for layer in self.lstm[:-1]:
x, _ = layer(x)
x = torch.relu(x)

#x = torch_unpack_seq(x)[0]

x = self.lstm[-1](x)
mask = torch.sigmoid(x)

return mask


and then:

model = Model(model_width, config)
model.cuda()


But I am getting this error:

File "main.py", line 29, in <module>
Model.train(args)
File ".../src/model.py", line 57, in train
model.cuda()
File ".../.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 637, in cuda
return self._apply(lambda t: t.cuda(device))
File ".../.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/.../.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File ".../.local/lib/python3.8/site-packages/torch/nn/modules/rnn.py", line 189, in _apply
self.flatten_parameters()
File ".../.local/lib/python3.8/site-packages/torch/nn/modules/rnn.py", line 175, in flatten_parameters
torch._cudnn_rnn_flatten_weight(
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.


I have no idea why it is happening. I am trying to push model and the inputs in cuda, and I understand if the error was due to some models in CPU and some in GPU. But that is not the case here. I found some pip install solution here: Pytorch CUDA error: no kernel image is available for execution on the device on RTX 3090 with cuda 11.1

but I cannot use it as I am trying to do the work in a remote repo where I don't have access to pip install.

Is there a way I can solve this?

Continue reading...