[Python] I tried to inherit the __init__ method of class T5ForConditionalGeneration and add a...

I tried to inherit the init method of class T5ForConditionalGeneration and add a linear layer L. L.weight is always abnormal when participating in calculations.I want to resolve this anomaly.I use Seq2SeqTrainetTrainer as a trainer.

class WithBranch(T5ForConditionalGeneration):
def __init__(self, config,):
super().__init__(config)
self.grl = GradientReversal(0.1)
hidden_size = config.d_model #768
self.l = nn.Linear(hidden_size, 2)
nn.init.xavier_uniform_(self.l.weight)
nn.init.zeros_(self.l.bias)
def forward(self, **kwargs):
...
feature = torch.randn(2, 768).to('cuda')
l_logits = self.l(features)
...
enter code here


I monitored L, and its result is

self.l.weight=
tensor([[3.3061e-39, 0.0000e+00, 1.5414e-44, ..., 0.0000e+00, 5.2152e-33,
0.0000e+00],
[6.0819e-33, 0.0000e+00, 1.5414e-44, ..., 0.0000e+00, 1.1576e-26,
0.0000e+00]], device='cuda:0', requires_grad=True)


I conducted some tests externally, and these weights are normal：

import torch.nn as nn
l = nn.Linear(768, 2)
print(l.weight)


output: tensor([[-0.0003, -0.0085, -0.0350, ..., -0.0014, 0.0282, 0.0016],
[ 0.0193, 0.0263, -0.0099, ..., 0.0208, -0.0082, -0.0107]],
requires_grad=True)

Continue reading...