[Python] Error while inferencing with onnx model on gpu

from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer
from optimum.pipelines import pipeline

model = ORTModelForSequenceClassification.from_pretrained(
"t",
provider="CUDAExecutionProvider"
)

tokenizer = AutoTokenizer.from_pretrained("tooka")

pipe = pipeline("text-classification", model=model, tokenizer=tokenizer, accelerator="ort")

print(pipe("This is great"))


throws:

RuntimeError: Error when binding input: There's no data transfer registered for copying tensors from Device:[DeviceType:1 MemoryType:0 DeviceId:0] to Device:[DeviceType:0 MemoryType:0 DeviceId:0]


It works with cpu fine. And here is available providers: Available providers: ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider'] i have tried different versions of onnx based on the recommendations on the web without success.

Continue reading...