Given a set of inputs that I want to use to train a GPT-like model from transformers, how do I configure the training objective (next token prediction) properly in the model / tokenizer? This is not a text domain but in principle my inputs are 500-600 tokens long and I want the model to later generate something from prompts of varying length, e.g. just 1 token, 10 tokens etc. To be clear, the models solely purpose is to complete the prompt. I have code that trains a model and loss is decreasing but I dont think it really does what I wish it to do? The results somewhat make sense but its hard to evaluate. I was wondering if / how I should implement somehow this step (manually) that: input_tokens = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] for i in range(len(input_tokens)): prediction = model.predict(input_tokens[0:i] label = input_tokens Some pseudo code to give more context: alphabet_size = 30 # some pseudo data, already tokenized => n=2 here # each entry should be used by the model to do the NTP # 10 = BOS token # 0 = padding # 2 = EOS input_data = [[1, 15, 21, 10, 13, 0, 2], [1, 5, 18, 21, 5, 7, 2]] # we want to use a GPT-like model from transformers import GPT2Model, GPT2Config # custom wrapper class Model(LightningModule): def init(): self.gpt_model = GPT2Model(config) # could be any decoder model self.output = nn.Linear(hidden_size, vocab_size) def forward(): outputs = self.gpt_model(input_ids) hidden_states = outputs.last_hidden_state logits = self.output = self.output(hidden_states) def training_step(self, batch, batch_idx): input_ids, labels = batch logits = self(input_ids) loss = cross_entropy(logits.view(-1, vocab_size), labels.view(-1), ignore_padding) # is this a valid tokenizer? class Loader: def __get_item__(self, idx): # randomly cut data here? input_ids = self.data[idx] labels = self.data[idx] # or something like this input_ids = self.data[idx][0: rnd] labels = self.data[idx][rnd:rnd+1] return (input_ids, labels) # or input_ids = self.data[idx][:-1] labels = self.data[idx][1:] model = Model() trainer = Trainer(model) trainer.fit(model, train_loader) Continue reading...
