[Python] Trained Huggingface EncoderDecoderModel.generate() produces only bos-tokens

I am working on a Huggingface transformers EncoderDecoderModel consisting of a frozen BERT-Encoder (answerdotai-ModernBERT-base) and a trainable GPT2-Decoder. Due to the different architectures for Encoder and Decoder, both come with different tokenizers. I use the according BERT tokenizer for the BERT model named above and a self-trained tokenizer for the GPT decoder.
While training of the model, I do some sample decodings from generated sequences during the evaluation phase to get better insights into the progress of the model training. The decoded sequences are produced via the forward() method belonging to the model:

outputs = self.model.forward(
input_ids=x, attention_mask=m, labels=y, decoder_input_ids=y
)
example_logits = outputs.logits[0]
predicted_token_ids = torch.argmax(example_logits, dim=-1)
decoded_prediction = self.decoder_tokenizer.decode(predicted_token_ids)


During the training, the generated sequences start to look better and better.

For Inference, I use the generate() method:

output = model.generate(
inputs=input_ids,
attention_mask=attention_mask,
decoder_start_token_id=decoder_tokenizer.bos_token_id,
streamer=text_streamer,
)
# outputs = model.forward(
# input_ids=input_ids,
# attention_mask=attention_mask,
# labels=decoder_input_ids,
# decoder_input_ids=decoder_input_ids,
# )
# output = outputs.logits.argmax(dim=-1)
print("Output: ", output)
output = decoder_tokenizer.decode(output.squeeze())


The output consists of many bos-tokens. It looks like the model has not been trained but it was. If I use the forward() method during the inference, the results are excellent again so i guess there must be some misconfiguration of the generate() method. Special tokens in the tokenizers are set correctly but do they have to be transferred into the model's config as well (I did that but no effect)? I also tried newly initialized (untrained) models for both forward() and generate() and the results were different, too. Another assumption: Maybe the problem is related to teacher forcing? How can I make the generate() method working?

Continue reading...