[Python] Failed to create TensorrtExecutionProvider using onnxruntime-gpu

I am having trouble using TensorRT execution provider for onnxruntime-gpu inferencing. I am initializing the session like this:

import onnxruntime as ort

providers = ["TensorrtExecutionProvider", "CUDAExecutionProvider"]
ort_sess = ort.InferenceSession(model_path, providers=providers)



and getting this error:

[Wnnxruntimeefault, onnxruntime_pybind_state.cc:509 CreateExecutionProviderInstance] Failed to create TensorrtExecutionProvider. Please reference https://onnxruntime.ai/docs/execution-providers/TensorRT-ExecutionProvider.html#requirements to ensure all dependencies are met.
​

ort.get_available_providers() outputs

['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider']
​

and ort_sess.get_providers() outputs

['CUDAExecutionProvider', 'CPUExecutionProvider']
​

I understand that the error message points to version mismatch but I didn't manage to get it right.

According to onnxruntime TensorRT compatibility table I must have one of the following combinations (https://onnxruntime.ai/docs/execution-providers/TensorRT-ExecutionProvider.html):

ONNX Runtime TensorRT CUDA
master 8.4 11.4
1.12 8.4 11.4
1.11 8.2 11.4
1.10 8.0 11.4
1.9 8.0 11.4

And according to CUDA requirements table (https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements) I must have one of these combinations:

ONNX Runtime​

CUDA​

cuDNN​

Notes
1.12, 1.11​

11.4​

8.2.4 (Linux), 8.2.2.26 (Windows)​

libcudart 11.4.43, libcufft 10.5.2.100, libcurand 10.2.5.120, libcublasLt 11.6.5.2, libcublas 11.6.5.2, libcudnn 8.2.4​

I have installed all NVIDIA-related packages via tar installation method from Nvidia docs. Here are my versions:

cuda:
cat /usr/local/cuda/version.json:

"cuda" : {
"name" : "CUDA SDK",
"version" : "11.4.4"
}


cudnn:
cat /usr/local/cuda/version.json:

#define CUDNN_MAJOR 8
#define CUDNN_MINOR 2
#define CUDNN_PATCHLEVEL 4


TensorRT:
pip list | grep tensorrt

tensorrt 8.2.3.0


My onnxruntime-gpu version is 1.11.0 because of the compatibility table above.

I am using TensorRT version 8.2.3.0 because (https://onnxruntime.ai/docs/build/eps.html#tensorrt)

The TensorRT execution provider for ONNX Runtime is built and tested with TensorRT 8.2.3.0.
​

There is, however, another page that states the following (https://onnxruntime.ai/docs/execution-providers/TensorRT-ExecutionProvider.html):

The TensorRT execution provider for ONNX Runtime is built and tested with TensorRT 8.4.
​

So I also tried another combo with TensorRT version TensorRT-8.4.1.5, onnxruntime-gpu==1.12.0, cuda-11.4, cudnn-8.2.4 but got the same error.

I'm using Debian 10.

What am I doing wrong? Has anyone nailed the version matching and could share the version combination? If so, did you also manage to install tensorrt via nvidia pypi index (this would make things more comfortable for me)?

Continue reading...