[Python] MultiScaleDeformableAttention.cpython-310-x86_64-linux-gnu.so: undefined symbol:...

I am currently trying to reproduce DN-DETR on an NVIDIA RTX 5090 GPU. My environment details are:

• GPU: NVIDIA RTX 5090
• PyTorch version: 2.8.0.dev20250322+cu128 (nightly)
• CUDA version: 12.8
• OS: Ubuntu (assuming Linux-based system)

When I try to import MultiScaleDeformableAttention from models/dn_dab_deformable_detr/ops, I get the following error:

(/home/user1/thanhpd) user1@xbk1:~/thanhpd/code/implementation-of-dn-detr/DN-DETR/models/dn_dab_deformable_detr/ops$ python -c "import MultiScaleDeformableAttention"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: /home/user1/thanhpd/code/implementation-of-dn-detr/DN-DETR/models/dn_dab_deformable_detr/ops/MultiScaleDeformableAttention.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c106detail23torchInternalAssertFailEPKcS2_jS2_RKSs


Any suggestions i can try?

What I Have Tried:

• Recompiling the MultiScaleDeformableAttention :

!python setup.py build install

• Try https://github.com/IDEA-Research/Grounded-Segment-Anything/issues/318

export CUDA_HOME=/usr/local/cuda-12.8

• Tried numerous different versions of pytorch and torchvision, but because my GPU is 50x0 cards, so only nighty version of pytorch supports it.

Continue reading...