[Python] F.cross_entropy raised "RuntimeError: CUDA error: device-side assert triggered....

I met this error in loss function. An example below:

import torch
device = "cuda" if torch.cuda.is_available() else "cpu"

a = torch.Tensor([[-10.3353, -28.4371, 2.0768, -4.2789, -8.6644, -6.0815],
[-10.3353, -28.4371, 2.0768, -4.2789, -8.6644, -6.0815],
[-10.3353, -28.4371, 2.0768, -4.2789, -8.6644, -6.0815],
[-10.3353, -28.4371, 2.0768, -4.2789, -8.6644, -6.0815]]).to(device)
b = torch.Tensor([ -100, -1, -100, 2456]).long().to(device)

loss = torch.nn.functional.cross_entropy(a,b,ignore_index=-1)
print(loss)


Then the error is:

---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-9-29fe2d43a573> in <cell line: 1>()
----> 1 loss = torch.nn.functional.cross_entropy(a,b,ignore_index=-1).to(device)
2 loss

/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing)
3027 if size_average is not None or reduce is not None:
3028 reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 3029 return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
3030
3031

RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


I tried to set export TORCH_USE_CUDA_DSA=1 but still the same error raised.

Continue reading...