I was running ncu on a remote A100 server to profile kernel activities. However, no matter what metric sets I used, the program just couldn't exit properly. For example, with the following command, I could get the output.ncu-rep correctly, the program also executed the last line of code I wrote, but it just couldn't exit. The entire command just hung forever unless I hit ctrl-c. P.S. You may change "roofline" to any metric set, e.g. basic, full or whatever. The same hanging would happen. ncu --set roofline -o output -f -- python <my_prog> <my_args...> ==PROF== Connected to process 6603 ==PROF== Profiling <kernel_1> ==PROF== Profiling <kernel_2> ... ==PROF== Disconnected from process 6603 # hanging here I also tried to run this command from my laptop's NVIDIA Nsight Compute, where I could connect to a remote server and start a profile activity remotely from NVIDIA's GUI tool. However, the same issue occurred -- the process log window of Nsight Compute also just hung. Moreover, after I force-killed this hanging program, if I ran another python program that called CUDA API (e.g. torch.cuda.synchronize()), this program wouldn't be able to exit either. The only way I could resolve this hanging issue for subsequent executions was reboot the entire system. It is worth noting that if I profiled a super tiny program with just a single CUDA kernel at the very beginning (when hanging had not occurred since the last reboot), the program would exit properly. But my program would launch several hundreds kernels, and this would cause hanging. Why did this hanging happen? Was that a bug or compatibility issue? Test environment: Python 3.10.18 (managed by Anaconda3) vLLM 0.5.3.post1 CUDA 12.1/12.6/12.9 NCU 2025.2.1.0 (shared across all CUDA versions) Ubuntu 22.04.5 LTS NVIDIA Nsight Compute 2025.2.0 My first program is a simple MLP from PyTorch and vLLM. I ran this function multiple times with different inputs. In total, there could be several hundreds kernel launches. import vllm import torch import torch.nn.functional as F def test_fn(hidden_states, w_gate_up, w_down): placeholder = F.linear(hidden_states, w_gate_up) dim = placeholder.shape[-1] // 2 hidden_shape = (placeholder.shape[:-1] + (dim,)) out = torch.empty(hidden_shape, dtype=placeholder.dtype, device=placeholder.device) vllm._custom_ops.silu_and_mul(out, placeholder) # type: ignore F.linear(out, w_down) My second program is a simple CUDA API call to test whether CUDA API could still be used after the hanging issue. The result was, yes, it could be used but would cause the program to hang at exit. import torch torch.cuda.synchronize() Continue reading...
