[Python] How do I compute validation loss for a fine-tuned Qwen model in Hugging Face...

I trained a Qwen model on my own dataset. Now I need to evaluate my trained model using the loss function, but I don’t know how to do it. I saw examples for other metrics such as accuracy and precision, but how can I evaluate the model using the loss function? I need to plot the different loss functions to evaluate which training session was the best. I have prepared my own dataset for it, but I don't know how I should carry one.

training_args = DPOConfig(
output_dir=logging_dir,
logging_steps=10,
per_device_train_batch_size=2,
per_device_eval_batch_size=2,
loss_type=["sft"],
loss_weights=[1.0],
max_prompt_length = 512,
max_completion_length = 512,
num_train_epochs=100,
max_steps=100000,
load_best_model_at_end=True,
metric_for_best_model="eval_loss",
save_strategy="steps",
save_steps=25000,
eval_strategy="steps",
eval_steps=100,

)

trainer = DPOTrainer(
model=model,
processing_class=tokenizer,
args=training_args,
train_dataset=dataset['train'],
eval_dataset=dataset['valid'],
)

trainer.train()

Continue reading...