[Python] Does adding a bunch of zeros to word embeddings maintain semantic meaning?

I parsed pre-downloaded 300 dimensional vectors into 768 dimensional vectors by simply hstacking an additional 468 zeros. I did this so that it would be compatible with the Query, Key and Value matrices I downloaded of DistilBert using the transformers library. Below is my code

from transformers import DistilBertModel
import numpy as np

model = DistilBertModel.from_pretrained("distilbert-base-uncased")

layer_0 = model.transformer.layer[0]

W_q = layer_0.attention.q_lin.weight.detach().numpy()
W_k = layer_0.attention.k_lin.weight.detach().numpy()
W_v = layer_0.attention.v_lin.weight.detach().numpy()

zeros = np.zeros((len(input_matrix), 468))
expanded_matrix = np.hstack((input_matrix, zeros))


The input matrix is a matrix (ndarray) of all the words my user inputs. This is part of the self attention mechanism I have written that will produce a modified matrix that I will feed into a feed forward network that will then give me the emotional tone of my user's text. I need to know if adding these extra zeros will lead to my results being skewed due to an incorrect self attention mechanism. It is important to note that the reason I have added these zeros is because the 3 extracted matrices are of shape (768, 768) each and I must multiply of input matrix by these parameters. Considering that without the adding of the zeros, the shape of the input matrix is (num_of_words_of_input, 300), the matrix multiplication (using @) will return an error (300 != 768). I am aware that I can use the DistilBert tokeniser however it is imperative that I don't. Also note that my logic behind this is that when you add zeros to the end of a vector to enter it into a higher dimensional space, it maintains the same magnitude and direction, just the space its in has more dimensions. Like if the 2d vector (3, 2) is put into 3d by making it (3, 2, 0) then its pointing in the same direction and with the same magnitude just in a 3d space.

Furthermore, when I computed all the Query and Key dot products (Query and Key meaning here below) like you are supposed to

Q = expanded_matrix @ W_q
#Q = Query of matrix so that the query of the first word in my input corresponds with the first row of Q
K = expanded_matrix @ W_k
#Same here
V = expanded_matrix @ W_v
#Same her as well


I got results that showed expected relationships between words. Nothing really seems off here. Thing is, for layer 0 (I computed dot products for all 6 layers) all the dot product values were in the negatives. Does this mean what I did as wrong? For the other 5 layers some results were 0 or below (expected) and others were higher (also expected).

I am intrigued to hear any possible insights and any answers are very much appreciated. If you know of another transformer with pre-trained Query, Key and value matrices that are compatible with my input matrix (num_of_words_of_input, 300)(before expansion) then do inform me. Thank you!

Continue reading...