I have seen this tutorial on how to train a BERT model from scratch on Hugging Face Transformers library. I'm trying to train a GPT-2 model on 1.5 GB data on Google Colab. I load up all the data using this code: dataset = LineByLineTextDataset( tokenizer=tokenizer, file_path="./my-1.5gb-large-file.txt", block_size=128, ) The file gets too large, training fails because of limited memory in the GPU. Is there a way I could train my GPT-2 model gradually by splitting the dataset? Continue reading...
