[Python] Llama_cookbook: why are labels not shifted for CausalLM?

I'm studying the llama_cookbok repo, in particular their finetuning example. This example uses LlamaForCausalLM model and samsum_dataset (input: dialog, output: summary). Now, looking at how they process the dataset, if we look at the "labels" part:

• For prompt tokens, every label is -100
• For summary tokens, we have labels = input_ids
  

Code:

prompt = tokenizer.encode(tokenizer.bos_token + sample["prompt"], add_special_tokens=False)
summary = tokenizer.encode(sample["summary"] + tokenizer.eos_token, add_special_tokens=False)

sample = {
"input_ids": prompt + summary,
"attention_mask" : [1] * (len(prompt) + len(summary)),
"labels": [-100] * len(prompt) + summary,
}


I also see this when printing the actual samples with the Dataloader they create (Note: if you want to reproduce the behavior, you'll have to change the path to the dataset to knkarthick/samsum in the package internals; or maybe use a different dataset):

batch = next(iter(train_dataloader))
print(batch['input_ids'][0][35:40])
print(batch['labels'][0][35:40])

Output:
tensor([19791, 512, 32, 36645, 41778])
tensor([ -100, -100, 32, 36645, 41778])


Why is the summary part correct? I thought that for CausalLM we must have labels = input_ids[i + 1] for every label we want to predict.

Continue reading...