[Python] RecBole: Very poor results with sequential models (BERT4Rec, SASRec) training

I am currently trying to research speed, accuracy differences between BERT4Rec and SASRec models using RecBole implementation. Referring to this article mostly for result comparison https://arxiv.org/pdf/1904.06690.pdf.

Using provided ml-1m dataset, I receive really poor results from the training of these models, haven't ran them more than 50 times due to lack of computational power, but they always flatline after around 20 epochs with ndcg@10 of 0.0523. While articles claim to have achieved 0.48+ on this measurement, I was trying to change hyperparameters a lot, assuming the implementation of that algorithm as well as their provided data sets to be without issues, but I can not get it over 0.07.

What could be the issue?

config file:

#Enviroment settings
gpu_id: 0
log_wandb: true
train_neg_sample_args: ~

learning_rate: 0.001
weight_decay: 0.005
mask_ratio: 0.2
hidden_size: 64

data_path: C:\user\MyScripts\dataset
load_col:
inter: [user_id, item_id, rating, timestamp]
item: [item_id, movie_title, release_year, class]
user: [user_id, age, gender, occupation, zip_code]

threshold: {'rating': 3}


And i simply call it with:

from recbole.quick_start import run_recbole
run_recbole(model='BERT4Rec', dataset='ml-1m', config_file_list = ['configExample.yaml'])


What i noticed is a really big train_loss:

Train 40: 100%|█████████████████████████| 48/48 [00:07<00:00, 6.19it/s, GPU RAM: 2.11 G/10.00 G]
17 Apr 13:38 INFO epoch 40 training [time: 7.75s, train loss: 280.5661]
Evaluate : 100%|███████████████████████████| 1/1 [00:00<00:00, 15.09it/s, GPU RAM: 2.11 G/10.00 G]
17 Apr 13:38 INFO epoch 40 evaluating [time: 0.07s, valid_score: 0.034600]
17 Apr 13:38 INFO valid result:
recall@10 : 0.1113 mrr@10 : 0.0346 ndcg@10 : 0.0523 hit@10 : 0.1113 precision@10 : 0.0111
17 Apr 13:38 INFO Finished training, best eval result in epoch 29


Could it be causing the issue? How do i fix it? Thanks!

Changed hypersettings from both recbole.io documentations and the hidden parameters I could see from when starting the script. I have even tried to mimic the parameter values of the article I'm referring to, however, the result is not even close to theirs. (I couldn't change some, like beta1, beta2 bits.)

Continue reading...