[Python] Training data and labels don't have the same shape after masking due padding

I'm a bit struggling with both the concepts of padding and masking. The task is a POS Tagging in NLP and I have 14987 sentences of training, 46 classes.

Basically I have a tensor of labels which is padded to the same length using -1.

Therefore, applying the mask based on the value -1 one can let ignore the -1 position.

The code is the following:

max_sentence_len = 124
y_train = pad_sequences(my_train_y, maxlen=max_sentence_len, padding='post', value=-1)
y_val = pad_sequences(my_valid_y, maxlen=max_sentence_len, padding='post', value=-1)
y_test = pad_sequences(my_test_y, maxlen=max_sentence_len, padding='post', value=-1)
# Output: y_train, y_val, y_test of shape
# (14987, 124)
# (3465, 124)
# (3683, 124)


So for example:

> y_train
array([[ 0, -1, -1, ..., -1, -1, -1],
[ 1, 2, 3, ..., -1, -1, -1],
[ 1, 1, -1, ..., -1, -1, -1],
...,
[ 4, 8, 1, ..., -1, -1, -1],
[ 4, 8, -1, ..., -1, -1, -1],
[ 4, 8, 1, ..., -1, -1, -1]], dtype=int32)


The code for the mask, which I don't know tbh if is correct:

def mask(tensor, value):
mask = tf.not_equal(tensor, value) # mask
mask = tf.cast(mask, tf.bool)
print("mask", mask.shape)
# Reshape del tensore
tensor_flat = tf.reshape(tensor, [-1]) # Shape: (batch_size * max_len, dim)
# Reshape della maschera a 1D
mask_flat = tf.reshape(mask, [-1]) # Shape: (batch_size * max_len,)
# Applica la maschera per ottenere solo i timesteps validi
masked_labels = tf.boolean_mask(tensor_flat, mask_flat) # Output: (num_valid_timesteps, dim)
return masked_labels


Then, the applying of the mask:

masked_labels = mask(y_train_tensor, -1)
masked_val_labels = mask(y_val_tensor, -1)
masked_test_labels = mask(y_test, -1)

nout = len(np.unique(masked_labels)) # nclasses for the onehot encoding

labels_tr = tf.one_hot(masked_labels, depth=nout)
labels_vl = tf.one_hot(masked_val_labels, depth=nout)
labels_test = tf.one_hot(masked_test_labels, depth=nout)


print("labels tr:", labels_tr.shape)
print("labels vl", labels_vl.shape)
print("labels test:", labels_test.shape)
# Outputs:
mask (14987, 124)
mask (3465, 124)
mask (3683, 124)
labels tr: (204567, 46)
labels vl (51574, 46)
labels test: (46625, 46)


At the same time, I have a train sequences which in this case are padded with 0 to the same length cause, after that, there is also a function which associates each fasttext 300-dim embedding to each word and reserves a 300dim vector of 1e-10 for OOV token and a 300dim vector of zeros for padding position (marked with 0 as I said before)

tokenizer = Tokenizer(oov_token='<UNK>')
tokenizer.fit_on_texts(my_train_input)

# Converti le frasi di train, validation e test in sequenze di indici
X_train_sequences = tokenizer.texts_to_sequences(my_train_input)
X_val_sequences = tokenizer.texts_to_sequences(my_valid_input)
X_test_sequences = tokenizer.texts_to_sequences(my_test_input)

# Applica il padding a tutte le sequenze
X_train_padded = pad_sequences(X_train_sequences, maxlen=max_sentence_len, padding='post', value=0)
X_val_padded = pad_sequences(X_val_sequences, maxlen=max_sentence_len, padding='post', value=0)
X_test_padded = pad_sequences(X_test_sequences, maxlen=max_sentence_len, padding='post', value=0)


The function which associates each word its embedding is the following, as I said before I reserve a 300dim vector of 1e-10 for OOV token and one 300dim of zeros for the padding position.

def getX(input_indices, model, vector_size=300):

x = []
OOV_count = 0
token_count = 0

# Cambiamo il vettore di OOV con piccoli valori costanti, e manteniamo gli zeri per il padding
oov_vector = np.full(vector_size, 1e-10) # Usa un valore non nullo per distinguere dagli zeri

for sentence in input_indices:
v = []
for idx in sentence:
if idx == 0: # Se è padding, usa il vettore di zeri
wv = np.zeros(vector_size)
else:
word = tokenizer.index_word.get(idx, None)
if word and word in model.key_to_index:
wv = model[word]
else:
wv = oov_vector # Assegna il vettore OOV non nullo
OOV_count += 1
token_count += 1
v.append(wv)
x.append(v)

print("Out of vocabulary rate: %f" % (OOV_count / token_count))
print("Vocabulary cover rate: %f" % ((token_count - OOV_count) / token_count))

return np.array(x, dtype=np.float32)


X_train = getX(X_train_padded, model)
X_val = getX(X_val_padded, model)
X_test = getX(X_test_padded, model)


The result is of getX is train tensor (14987, 124), val tensor (3465, 124) and test tensor (3683, 124) so they aren't consistent with the labels (after masking they were (204567, 46) etc as seen before). Now my hypothesis was to give in input to the RNN the train and the labels, and resolve the mismatch applying another mask (but based on 0 and not -1) at the end of the states computation (so on the 3D tensor), I thought that since I had same padded position (I guess(?)) for train and labels sequences, so I'd have obtained same shape.

But, given a mask to apply on the training states such that:

def train_mask(tensor, value):
mask = tf.not_equal(tensor[..., 0], value) # mask
mask = tf.cast(mask, tf.bool)
print("mask", mask.shape)
# Reshape del tensore 3D in una forma 2D per rendere applicabile la maschera
tensor_flat = tf.reshape(tensor, [-1, tensor.shape[-1]]) # Shape: (batch_size * max_len, dim)
# Reshape della maschera a 1D
mask_flat = tf.reshape(mask, [-1]) # Shape: (batch_size * max_len,)
# Applica la maschera per ottenere solo i timesteps validi
masked_states = tf.boolean_mask(tensor_flat, mask_flat) # Output: (num_valid_timesteps, dim)
return masked_states

# States computation
states = RNN.compute_states(X_train)
# Output: X_train: (14987, 124, 1000)

masked_states = train_mask(states, 0) # I thought to obtain (204567, 1000) like before for the labels

#Output: masked_states: (1858388,1000)


The masked labels are (204567, 46) but the masked states are (1858388,1000) so I have the mismatch (204567 != 1858388)

It could be that after states computation I have other 0 and not only those of the padding, but I can't do masking before states computation because tf wants 3D tensor in input...

Does anyone have any suggestion?

Continue reading...