[Python] Error loading model using Torch TensorRT in Libtorch on Windows

Environment

• Libtorch 2.5.0.dev (latest nightly) (built with CUDA 12.4)
• CUDA 12.4
• TensorRT 10.1.0.27
• PyTorch 2.4.0+cu124
• Torch-TensorRT 2.4.0
• Python 3.12.8
• Windows 10

Compile Torch-TensorRT with Cmake to generate lib and dll



Option : Export


If you want to optimize your model ahead-of-time and/or deploy in a C++ environment, Torch-TensorRT provides an export-style workflow that serializes an optimized module. This module can be deployed in PyTorch or with libtorch (i.e. without a Python dependency).

Step 1: Optimize + serialize


import torch
import torch_tensorrt

model = MyModel().eval().cuda() # define your model here
inputs = [torch.randn((1, 3, 224, 224)).cuda()] # define a list of representative inputs here

trt_gm = torch_tensorrt.compile(model, ir="dynamo", inputs)
torch_tensorrt.save(trt_gm, "trt.ep", inputs=inputs) # PyTorch only supports Python runtime for an ExportedProgram. For C++ deployment, use a TorchScript file
torch_tensorrt.save(trt_gm, "trt.ts", output_format="torchscript", inputs=inputs)

Step 2: Deploy

Deployment in C++:


#include "torch/script.h"
#include "torch_tensorrt/torch_tensorrt.h"

auto trt_mod = torch::jit::load("trt.ts");
auto input_tensor = [...]; // fill this with your inputs
auto results = trt_mod.forward({input_tensor});

ERROR


auto trt_mod = torch::jit::load("trt.ts")




Unknown type name '__torch__.torch.classes.tensorrt.Engine':
File "code/__torch__/torch_tensorrt/dynamo/runtime/_TorchTensorRTModule.py", line 6
training : bool
_is_full_backward_hook : Optional[bool]
engine : __torch__.torch.classes.tensorrt.Engine
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`your text`~~~~~~~~ <--- HERE
def forward(self: __torch__.torch_tensorrt.dynamo.runtime._TorchTensorRTModule.TorchTensorRTModule,
input: Tensor) -> Tensor:

Continue reading...