[Python] ModuleNotFoundError: 'triton.ops' when loading 4-bit quantized model with...

I have this code:

import os
import torch
from datasets import Dataset
from transformers import (
AutoModelForCausalLM,
AutoTokenizer,
BitsAndBytesConfig,
TrainingArguments,
)
from peft import LoraConfig, PeftModel
from trl import SFTTrainer
from kaggle_secrets import UserSecretsClient
# =============================================================================
# 3. MODEL CONFIGURATION AND LOADING (Revised to 8-bit)
# We will load the model in 8-bit to avoid the triton dependency.
# =============================================================================
model_name = "Qwen/Qwen2.5-7B-Instruct"

# Load the base model with 8-bit quantization
print(f"\nLoading base model: {model_name} in 8-bit")
model = AutoModelForCausalLM.from_pretrained(
model_name,
load_in_8bit=True,
device_map="auto", # Automatically maps layers to available devices (GPU/CPU)
)
model.config.use_cache = False # Recommended for training

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

# PEFT (LoRA) configuration remains the same
lora_config = LoraConfig(
r=16,
lora_alpha=32,
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
)


But I get this error when I run (i have imported triton)

Failed to import transformers.integrations.bitsandbytes because of the following error (look up to see its traceback):
No module named 'triton.ops'
​

How to solve this problem?

Continue reading...