I am trying to write a model to “translate” sign language in real time. I try to do this this way: https://github.com/dgovor/Sign-Language-Translator but with dataset. I am using ASL Citizen as dataset from which I retrieve keypoints using mediapipe, which I then use as a dataset. The dataset is divided originally into Test - 30k videos, Train - 40k, Validation - 12k and has 2300 classes. Unfortunately, every model I have written so far and the data change techniques do not allow the model to be learned more than 70% which is quite strange because I have done things like: originally split data to max 200 frames (data shape - (200, 126)) padding and masking to a value of -999 limiting the data to max 100 frames, and taking every 2nd frame if more than 200 frames (data shape - (100, 126)) no masking and padding value of 0.0 class weighting different batch sizes using SMOTE with changing data to 2d arrays changing the amount of data in the Test, Train and Validation folders to: 10k, 59k, 13k and repeating the above steps using and checking on other different models each of the above ways And regardless of whether the data has a different shape or more training data is the result remains the same and learns to a maximum of about 70%, what is more strange is the training that has, for example, such values: Epoch 1/5000 1850/1850 ━━━━━━━━━━━━━━━━━━━━ 0s 49ms/step - categorical_accuracy: 3.4063e-04 - loss: 7.9569 Epoch 1: val_categorical_accuracy improved from -inf to 0.00300, saving model to Model/model.keras 1850/1850 ━━━━━━━━━━━━━━━━━━━━ 102s 53ms/step - categorical_accuracy: 3.4087e-04 - loss: 7.9568 - val_categorical_accuracy: 0.0030 - val_loss: 7.1603 - learning_rate: 5.0000e-04 Epoch 2/5000 1849/1850 ━━━━━━━━━━━━━━━━━━━━ 0s 48ms/step - categorical_accuracy: 0.0056 - loss: 6.8887 Epoch 2: val_categorical_accuracy improved from 0.00300 to 0.03792, saving model to Model/model.keras 1850/1850 ━━━━━━━━━━━━━━━━━━━━ 95s 51ms/step - categorical_accuracy: 0.0056 - loss: 6.8883 - val_categorical_accuracy: 0.0379 - val_loss: 5.6242 - learning_rate: 5.0000e-04 Epoch 3/5000 1850/1850 ━━━━━━━━━━━━━━━━━━━━ 0s 48ms/step - categorical_accuracy: 0.0525 - loss: 5.3936 Epoch 3: val_categorical_accuracy improved from 0.03792 to 0.11668, saving model to Model/model.keras 1850/1850 ━━━━━━━━━━━━━━━━━━━━ 96s 52ms/step - categorical_accuracy: 0.0526 - loss: 5.3934 - val_categorical_accuracy: 0.1167 - val_loss: 4.4322 - learning_rate: 5.0000e-04 Epoch 4/5000 1850/1850 ━━━━━━━━━━━━━━━━━━━━ 0s 49ms/step - categorical_accuracy: 0.1578 - loss: 4.1421 Epoch 4: val_categorical_accuracy improved from 0.11668 to 0.19252, saving model to Model/model.keras 1850/1850 ━━━━━━━━━━━━━━━━━━━━ 97s 52ms/step - categorical_accuracy: 0.1578 - loss: 4.1420 - val_categorical_accuracy: 0.1925 - val_loss: 3.7442 - learning_rate: 5.0000e-04 Epoch 5/5000 1849/1850 ━━━━━━━━━━━━━━━━━━━━ 0s 50ms/step - categorical_accuracy: 0.2696 - loss: 3.3104 Epoch 5: val_categorical_accuracy improved from 0.19252 to 0.37574, saving model to Model/model.keras 1850/1850 ━━━━━━━━━━━━━━━━━━━━ 100s 54ms/step - categorical_accuracy: 0.2696 - loss: 3.3104 - val_categorical_accuracy: 0.3757 - val_loss: 2.6425 - learning_rate: 5.0000e-04 As you can see Val Accuracy is greater than train in the beginning and I think it's not supposed to happen because my dropout is only 0.1, later in training it's different and train accuracy is higher: Epoch 81/5000 1850/1850 ━━━━━━━━━━━━━━━━━━━━ 0s 79ms/step - categorical_accuracy: 0.8465 - loss: 0.4662 Epoch 81: val_categorical_accuracy did not improve from 0.67887 1850/1850 ━━━━━━━━━━━━━━━━━━━━ 155s 84ms/step - categorical_accuracy: 0.8465 - loss: 0.4663 - val_categorical_accuracy: 0.6658 - val_loss: 1.7038 - learning_rate: 5.0000e-04 Epoch 82/5000 1850/1850 ━━━━━━━━━━━━━━━━━━━━ 0s 78ms/step - categorical_accuracy: 0.8478 - loss: 0.4656 Epoch 82: val_categorical_accuracy did not improve from 0.67887 1850/1850 ━━━━━━━━━━━━━━━━━━━━ 154s 83ms/step - categorical_accuracy: 0.8478 - loss: 0.4656 - val_categorical_accuracy: 0.6645 - val_loss: 1.6863 - learning_rate: 5.0000e-04 Epoch 83/5000 The best model so far is that one: def __init__(self, input_shape, word_labels, mask_value): self.input_shape = input_shape self.word_labels = word_labels self.mask_value = mask_value self.model = self.build_model() def build_model(self, activation="leaky_relu"): data = Input(name='input', shape=self.input_shape) #masked_data = Masking(mask_value=self.mask_value)(data) # LSTM Layer x = LSTM(units=64, return_sequences=True, activation='relu')(data) x = BatchNormalization()(x) x = Dropout(0.1)(x) # GRU Layer x = GRU(units=128, return_sequences=False, activation='relu')(x) x = BatchNormalization()(x) x = Dropout(0.1)(x) # Dense Layer for feature extraction x = Dense(units=256, activation='relu')(x) x = BatchNormalization()(x) x = Dropout(0.1)(x) output = Dense(self.word_labels.shape[0], activation="softmax")(x) model = Model(inputs=data, outputs=output) return model def compile_model(self): optimizer = keras.optimizers.Adam(learning_rate=0.0005) loss = 'categorical_crossentropy' metrics = ['categorical_accuracy'] self.model.compile(loss=loss, optimizer=optimizer, metrics=metrics) And the way I train it is like tkis: print("Loading Data...") integer_train_labels = np.argmax(self.train_num_labels, axis=1) class_counts = Counter(integer_train_labels) total_samples = len(integer_train_labels) num_classes = len(self.word_labels) class_weight = {cls: total_samples / (count * num_classes) for cls, count in class_counts.items()} print("Finished Loading Data...") print("Creating Model...") if os.path.exists("Model/model.keras"): input_shape = (self.max_frames, 126) SLR_Model = self.SLR_Model(input_shape, self.word_labels, self.mask_value) SLR_Model.compile_model() SLR_Model.model.load_weights("Model/model.keras") print(SLR_Model.model.get_weights()) new_model = False else: input_shape = (self.max_frames, 126) SLR_Model = self.SLR_Model(input_shape, self.word_labels, self.mask_value) SLR_Model.compile_model() SLR_Model.summary(line_length=110) print(SLR_Model.model.get_weights()) new_model = True print("Model Created...") print("Training Model...") backup = BackupAndRestore(backup_dir="Model/backup") earlystopper = EarlyStopping(monitor='val_categorical_accuracy', patience=50, verbose=1, mode='max') checkpoint = ModelCheckpoint("Model/model.keras", monitor='val_categorical_accuracy', verbose=1, save_best_only=True, mode='max') trainLogger = TrainLogger("Model") tb_callback = TensorBoard('Model/logs', update_freq=1) reduceLROnPlat = ReduceLROnPlateau(monitor='val_categorical_accuracy', factor=0.9, min_delta=1e-10, patience=25, verbose=1, mode='auto') is_Trained = False if is_Trained is False: if new_model is True: SLR_Model.train(self.train_keypoints, self.train_num_labels, epochs=self.epochs, callbacks=[backup, earlystopper, checkpoint, trainLogger, reduceLROnPlat, tb_callback], x_val=self.val_keypoints, y_val=self.val_num_labels, batch_size=self.batch_size, class_weight=class_weight) else: SLR_Model.train(self.train_keypoints, self.train_num_labels, epochs=self.epochs, callbacks=[backup, earlystopper, checkpoint, trainLogger, reduceLROnPlat, tb_callback], x_val=self.val_keypoints, y_val=self.val_num_labels, batch_size=self.batch_size, class_weight=class_weight) I really don't know what I can do more to achieve at least 80% accuracy, is max accuracy really 70%, data is wrong, model is too simple or maybe there is a wrong technique I am using ? And if this thing with greater val accuracy in the beginning is normal ? I really wish to learn how can I build such models and right know I am kind of stuck with it and I don't know anymore if I can do something about it. Can someone take a look at this problem and advise me ? Many thanks for any ideas that I can try or check out Continue reading...
