[Python] Errors while converting huggingface dataset to Tensorflow dataset while using cuda

I am trying to do finetuning using a dataset from huggingface. I keep getting InternalError.

My GPU set up is as follows:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 575.64.01 Driver Version: 576.88 CUDA Version: 12.9 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 5070 Ti On | 00000000:01:00.0 On | N/A |
| 0% 55C P3 30W / 300W | 4355MiB / 16303MiB | 1% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 3070 On | 00000000:04:00.0 Off | N/A |
| 0% 41C P8 8W / 240W | 1147MiB / 8192MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+


I have 96 GB ram.

The program works on macbook air using M4 CPU.

Here is the code:

!pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128


!pip3 install tensorflow[and-cuda] transformers datasets ipywidgets tf_keras huggingface_hub

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

# Load the tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-base")
model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-base")



from datasets import load_dataset

# Load the Helsinki-NLP/opus-100 dataset
dataset = load_dataset('Helsinki-NLP/opus-100', 'en-es')
print(dataset['train'][0])
# Access column names for a specific split (e.g., 'train')
column_names = dataset["train"].column_names
print(column_names)

# Preprocess the dataset for input into the model
def preprocess_data(examples):
inputs = ['Translate from English to Spanish: {example["en"]}' for example in examples['translation']]
targets = [example['es'] for example in examples['translation']]

# Tokenize inputs
model_inputs = tokenizer(inputs, max_length=128, truncation=True, padding="max_length")

# Tokenize targets
with tokenizer.as_target_tokenizer():
labels = tokenizer(targets, max_length=128, truncation=True, padding="max_length")

model_inputs["labels"] = labels["input_ids"]

# For decoder inputs
decoder_inputs = tokenizer(targets, max_length=128, truncation=True, padding="max_length")
model_inputs["decoder_input_ids"] = decoder_inputs["input_ids"]

return model_inputs

# Apply preprocessing to the dataset
train_dataset = dataset['train'].select(range(5000)).map(preprocess_data, batched=True)
test_dataset = dataset['test'].select(range(1000)).map(preprocess_data, batched=True)

print(train_dataset[0])

# Convert Hugging Face datasets to TensorFlow datasets
train_dataset = train_dataset.to_tf_dataset(
columns=['input_ids', 'attention_mask', 'decoder_input_ids'],
label_cols=['labels'],
shuffle=True,
batch_size=16,
collate_fn=None
)
test_dataset = test_dataset.to_tf_dataset(
columns=['input_ids', 'attention_mask', 'decoder_input_ids'],
label_cols=['labels'],
shuffle=False,
batch_size=16,
collate_fn=None
)


The error I get is:

2025-07-10 16:03:54.511675: W tensorflow/compiler/mlir/tools/kernel_gen/tf_gpu_runtime_wrappers.cc:40] 'cuModuleLoadData(&module, data)' failed with 'CUDA_ERROR_INVALID_PTX'

2025-07-10 16:03:54.511738: W tensorflow/compiler/mlir/tools/kernel_gen/tf_gpu_runtime_wrappers.cc:40] 'cuModuleGetFunction(&function, module, kernel_name)' failed with 'CUDA_ERROR_INVALID_HANDLE'

2025-07-10 16:03:54.511750: W tensorflow/core/framework/op_kernel.cc:1844] INTERNAL: 'cuLaunchKernel(function, gridX, gridY, gridZ, blockX, blockY, blockZ, 0, reinterpret_cast<CUstream>(stream), params, nullptr)' failed with 'CUDA_ERROR_INVALID_HANDLE'


Is there anyway to fix this issue?

Continue reading...