I'm trying to train a Tensorflow model on a GPU, but I've been struggling to get it running. Essentially, I'm using a high-performance computer cluster and submit jobs using bash and slurm-files. My environment is the following: # packages in environment at /data/leuven/351/vsc35132/miniconda3/envs/SpecCheck: # # Name Version Build Channel _libgcc_mutex 0.1 main _openmp_mutex 5.1 1_gnu absl-py 2.2.1 pypi_0 pypi astunparse 1.6.3 pypi_0 pypi bzip2 1.0.8 h5eee18b_6 ca-certificates 2025.2.25 h06a4308_0 certifi 2025.1.31 pypi_0 pypi charset-normalizer 3.4.1 pypi_0 pypi expat 2.6.4 h6a678d5_0 flatbuffers 25.2.10 pypi_0 pypi gast 0.6.0 pypi_0 pypi google-pasta 0.2.0 pypi_0 pypi grpcio 1.71.0 pypi_0 pypi h5py 3.13.0 pypi_0 pypi idna 3.10 pypi_0 pypi joblib 1.4.2 pypi_0 pypi keras 3.9.1 pypi_0 pypi ld_impl_linux-64 2.40 h12ee557_0 libclang 18.1.1 pypi_0 pypi libffi 3.4.4 h6a678d5_1 libgcc-ng 11.2.0 h1234567_1 libgomp 11.2.0 h1234567_1 libstdcxx-ng 11.2.0 h1234567_1 libuuid 1.41.5 h5eee18b_0 markdown 3.7 pypi_0 pypi markdown-it-py 3.0.0 pypi_0 pypi markupsafe 3.0.2 pypi_0 pypi mdurl 0.1.2 pypi_0 pypi ml-dtypes 0.5.1 pypi_0 pypi namex 0.0.8 pypi_0 pypi ncurses 6.4 h6a678d5_0 numpy 2.1.3 pypi_0 pypi nvidia-cublas-cu12 12.5.3.2 pypi_0 pypi nvidia-cuda-cupti-cu12 12.5.82 pypi_0 pypi nvidia-cuda-nvcc-cu12 12.5.82 pypi_0 pypi nvidia-cuda-nvrtc-cu12 12.5.82 pypi_0 pypi nvidia-cuda-runtime-cu12 12.5.82 pypi_0 pypi nvidia-cudnn-cu12 9.3.0.75 pypi_0 pypi nvidia-cufft-cu12 11.2.3.61 pypi_0 pypi nvidia-curand-cu12 10.3.6.82 pypi_0 pypi nvidia-cusolver-cu12 11.6.3.83 pypi_0 pypi nvidia-cusparse-cu12 12.5.1.3 pypi_0 pypi nvidia-nccl-cu12 2.23.4 pypi_0 pypi nvidia-nvjitlink-cu12 12.5.82 pypi_0 pypi openssl 3.0.16 h5eee18b_0 opt-einsum 3.4.0 pypi_0 pypi optree 0.14.1 pypi_0 pypi packaging 24.2 pypi_0 pypi pandas 2.2.3 pypi_0 pypi pip 25.0 py312h06a4308_0 protobuf 5.29.4 pypi_0 pypi pyarrow 19.0.1 pypi_0 pypi pygments 2.19.1 pypi_0 pypi python 3.12.9 h5148396_0 python-dateutil 2.9.0.post0 pypi_0 pypi pytz 2025.2 pypi_0 pypi readline 8.2 h5eee18b_0 requests 2.32.3 pypi_0 pypi rich 14.0.0 pypi_0 pypi scikit-learn 1.6.1 pypi_0 pypi scipy 1.15.2 pypi_0 pypi setuptools 75.8.0 py312h06a4308_0 six 1.17.0 pypi_0 pypi sqlite 3.45.3 h5eee18b_0 tensorboard 2.19.0 pypi_0 pypi tensorboard-data-server 0.7.2 pypi_0 pypi tensorflow 2.19.0 pypi_0 pypi termcolor 3.0.0 pypi_0 pypi threadpoolctl 3.6.0 pypi_0 pypi tk 8.6.14 h39e8969_0 typing-extensions 4.13.0 pypi_0 pypi tzdata 2025.2 pypi_0 pypi urllib3 2.3.0 pypi_0 pypi werkzeug 3.1.3 pypi_0 pypi wheel 0.45.1 py312h06a4308_0 wrapt 1.17.2 pypi_0 pypi xz 5.6.4 h5eee18b_1 zlib 1.2.13 h5eee18b_1 And the output from my slurm-file is the following: ======================================================================== /var/spool/slurmd/job64225291/slurm_script: line 14: nvcc: command not found Tue Apr 1 12:45:15 2025 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 525.147.05 Driver Version: 525.147.05 CUDA Version: 12.0 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA H100 80G... On | 00000000:26:00.0 Off | 0 | | N/A 28C P0 73W / 700W | 0MiB / 81559MiB | 0% E. Process | | | | Disabled | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ 2025-04-01 12:45:24.661857: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2025-04-01 12:45:29.017117: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered WARNING: All log messages before absl::InitializeLog() is called are written to STDERR E0000 00:00:1743504330.375728 709539 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered E0000 00:00:1743504330.649505 709539 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered W0000 00:00:1743504332.740332 709539 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once. W0000 00:00:1743504332.740392 709539 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once. W0000 00:00:1743504332.740395 709539 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once. W0000 00:00:1743504332.740396 709539 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once. 2025-04-01 12:45:32.806499: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. I0000 00:00:1743504429.190211 709539 gpu_device.cc:2019] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 78430 MB memory: -> device: 0, name: NVIDIA H100 80GB HBM3, pci bus id: 0000:26:00.0, compute capability: 9.0 2025-04-01 12:47:33.004061: W tensorflow/compiler/mlir/tools/kernel_gen/tf_gpu_runtime_wrappers.cc:40] 'cuModuleLoadData(&module, data)' failed with 'CUDA_ERROR_UNSUPPORTED_PTX_VERSION' 2025-04-01 12:47:33.004102: W tensorflow/compiler/mlir/tools/kernel_gen/tf_gpu_runtime_wrappers.cc:40] 'cuModuleGetFunction(&function, module, kernel_name)' failed with 'CUDA_ERROR_INVALID_HANDLE' 2025-04-01 12:47:33.004111: W tensorflow/core/framework/op_kernel.cc:1844] INTERNAL: 'cuLaunchKernel(function, gridX, gridY, gridZ, blockX, blockY, blockZ, 0, reinterpret_cast<CUstream>(stream), params, nullptr)' failed with 'CUDA_ERROR_INVALID_HANDLE' 2025-04-01 12:47:33.004128: I tensorflow/core/framework/local_rendezvous.cc:407] Local rendezvous is aborting with status: INTERNAL: 'cuLaunchKernel(function, gridX, gridY, gridZ, blockX, blockY, blockZ, 0, reinterpret_cast<CUstream>(stream), params, nullptr)' failed with 'CUDA_ERROR_INVALID_HANDLE' Memory growth set for GPUs Traceback (most recent call last): File "/vsc-hard-mounts/leuven-data/351/vsc35132/SpecCheck/SlurmFiles/../Scripts/SpecCheckVSC.py", line 944, in <module> main() File "/vsc-hard-mounts/leuven-data/351/vsc35132/SpecCheck/SlurmFiles/../Scripts/SpecCheckVSC.py", line 927, in main fit_model(positive_file_path=args.Positive_samples, File "/vsc-hard-mounts/leuven-data/351/vsc35132/SpecCheck/SlurmFiles/../Scripts/SpecCheckVSC.py", line 652, in fit_model model = create_transformer_model(sequence_length, embedding_dim, num_heads, ff_dim, num_transformer_blocks, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/vsc-hard-mounts/leuven-data/351/vsc35132/SpecCheck/SlurmFiles/../Scripts/SpecCheckVSC.py", line 285, in create_transformer_model combined_embeddings = layers.Conv1D(filters=embedding_dim,kernel_size=5,padding='same',activation='relu')(combined_inputs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/leuven/351/vsc35132/miniconda3/envs/h100_tf/lib/python3.11/site-packages/keras/src/utils/traceback_utils.py", line 122, in error_handler raise e.with_traceback(filtered_tb) from None File "/data/leuven/351/vsc35132/miniconda3/envs/h100_tf/lib/python3.11/site-packages/keras/src/backend/tensorflow/core.py", line 139, in convert_to_tensor return tf.cast(x, dtype) ^^^^^^^^^^^^^^^^^ tensorflow.python.framework.errors_impl.InternalError: {{function_node __wrapped__Cast_device_/job:localhost/replica:0/task:0/device:GPU:0}} 'cuLaunchKernel(function, gridX, gridY, gridZ, blockX, blockY, blockZ, 0, reinterpret_cast<CUstream>(stream), params, nullptr)' failed with 'CUDA_ERROR_INVALID_HANDLE' [Op:Cast] name: However, I've been installing tensorflow as instructed on: https://www.tensorflow.org/install/pip Additionally, I've tried various versions and new environments too. Continue reading...
