[Python] Huge training loss when fine-tuning DeBERTa-v3-small with HuggingFace Trainer + LoRA

I am trying to fine-tune microsoft/deberta-v3-small with HuggingFace Trainer + PEFT LoRA adapters for a binary classification task (truth vs lie transcripts).

My dataset is from the MU3D database. It is small (~320 samples),and labels are 0/1 with 1=truth and 0=lie.

Here is an example of the format of my dataset which I have preprocessed already and have separated into train, validation, and test sets.

Each of the datasets follows the same format:

VideoID SpeakerID Veracity Transcription text_clean
BF003_1PT BF003 1 This is a sample text this is a sample text
BF003_2NL BF003 0 This is Another Test this is another test

I preprocessed the text with a tokenizer (MAX_LEN=512). The training runs on an NVIDIA RTX 2050 (4 GB VRAM).

Here is my pipeline

import os, numpy as np, pandas as pd, torch
from datasets import Dataset, DatasetDict
from sklearn.metrics import accuracy_score, precision_recall_fscore_support, average_precision_score, confusion_matrix
from transformers import (AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments,
Trainer, EarlyStoppingCallback, BitsAndBytesConfig)
from peft import LoraConfig, get_peft_model, TaskType
from pathlib import Path

SEED = 42
MODEL_NAME = "microsoft/deberta-v3-small" # fallback below if OOM
MAX_LEN = 192
OUTPUT_DIR = "artifacts_lora_deberta_small"

torch.manual_seed(SEED); np.random.seed(SEED)

# ---------- IO ----------
train = pd.read_csv("train_subject_disjoint.csv")
val = pd.read_csv("val_subject_disjoint.csv")
test = pd.read_csv("test_subject_disjoint.csv")

def to_hfds(df):
return Dataset.from_pandas(
df[["text_clean","Veracity","VideoID","SpeakerID"]]
.rename(columns={"text_clean":"text","Veracity":"label"}),
preserve_index=False
)

ds = DatasetDict(train=to_hfds(train), validation=to_hfds(val), test=to_hfds(test))



tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
def tok(batch):
return tokenizer(batch["text"], max_length=MAX_LEN, truncation=True, padding="max_length")
ds = ds.map(tok, batched=True)
ds = ds.remove_columns(["text","VideoID","SpeakerID"])
ds.set_format(type="torch")


base = AutoModelForSequenceClassification.from_pretrained(
"microsoft/deberta-v3-small" ,
num_labels=2,
device_map="cuda")


lora_cfg = LoraConfig(
task_type=TaskType.SEQ_CLS,
r=16,
lora_alpha=32,
lora_dropout=0.1,
target_modules=["query_proj","key_proj","value_proj","dense"]
)
model = get_peft_model(base, lora_cfg)
model.print_trainable_parameters()


args = TrainingArguments(
output_dir=OUTPUT_DIR,
eval_strategy="epoch",
save_strategy="epoch",
load_best_model_at_end=True,
metric_for_best_model="f1_macro",
greater_is_better=True,

num_train_epochs=50,
per_device_train_batch_size=4, # small to fit 4GB
per_device_eval_batch_size=16,
gradient_accumulation_steps=4, # effective train batch = 16
fp16= True, # mixed precision
bf16=False, # set True if your GPU supports bf16
learning_rate=2e-5,
weight_decay=0.01,
warmup_ratio=0.1,

logging_steps=50,
save_total_limit=2,
seed=SEED,

gradient_checkpointing=True, # big memory saver on 4GB
)

trainer = Trainer(
model=model,
args=args,
train_dataset=ds["train"],
eval_dataset=ds["validation"],
tokenizer=tokenizer,
compute_metrics=compute_metrics,
callbacks=[EarlyStoppingCallback(early_stopping_patience=10)]
)

trainer.train()
print("\n[VAL]", trainer.evaluate())



However, when my training loop was running, I observed a huge training loss below

Epoch Training Loss Validation Loss Accuracy F1 Macro F1 Weighted Pr Auc Precision Macro Recall Macro
1 No log 0.697312 0.500000 0.333333 0.333333 0.532561 0.250000 0.500000
2 No log 0.697031 0.500000 0.333333 0.333333 0.534184 0.250000 0.500000
3 No log 0.696202 0.500000 0.333333 0.333333 0.532561 0.250000 0.500000
4 0.700200 0.695277 0.500000 0.333333 0.333333 0.532101 0.250000 0.500000
5 0.700200 0.694190 0.500000 0.333333 0.333333 0.534942 0.250000 0.500000
6 0.700200 0.693882 0.500000 0.333333 0.333333 0.534942 0.250000 0.500000
7 0.694800 0.693520 0.500000 0.333333 0.333333 0.533724 0.250000 0.500000
8 0.694800 0.693351 0.500000 0.333333 0.333333 0.532101 0.250000 0.500000
9 0.694800 0.693283 0.500000 0.333333 0.333333 0.532101 0.250000 0.500000
10 0.697500 0.693154 0.500000 0.333333 0.333333 0.533724 0.250000 0.500000
11 0.697500 0.693144 0.500000 0.333333 0.333333 0.529720 0.250000 0.500000

I was just wondering, how can I improve the training loss? I am kind of new to fine-tuning models. Would I swap this model out with a new one?

Thanks

Continue reading...