[Python] Classification Neural Network Not Converging

I developed a basic classification network for MNIST, but during training, the validation accuracy tr,aims around 10%. I have experimented with various optimizers (SGD, Adam, Nadam), as well as different learning rates (0.1, 1e-3, 1e-4, 1e-5), yet the validation accuracy remains consistent at around 10% in every epoch.

Here is my code:

import tensorflow as tf
from tensorflow import keras

(X_train, y_train), (X_test, y_test) = keras.datasets.mnist.load_data()
X_train, X_val = X_train[5000:]/255.0, X_train[:5000]/255.0
y_train, y_val = y_train[5000:]/255.0, y_train[:5000]/255.0

class_NN = keras.models.Sequential([
keras.layers.Input(shape = [28, 28]),
keras.layers.Flatten(),
keras.layers.Dense(300, activation = "relu"),
keras.layers.Dense(100, activation = "relu"),
keras.layers.Dense(10, activation = "softmax")
])

class_NN.compile(loss = "sparse_categorical_crossentropy",
optimizer = keras.optimizers.SGD(learning_rate = 1e-3),
metrics = ["accuracy"])
class_NN.fit(X_train, y_train, epochs = 15,
validation_data = (X_val, y_val))


I took out the /255.0 from the code before starting the training while writing this question. Currently, the accuracy in validation is increasing as we train. I achieved a validation accuracy of around 96%. What causes the increase in validation accuracy during training when the /255.0 is taken out?

X_train, X_val = X_train[5000:], X_train[:5000]
y_train, y_val = y_train[5000:], y_train[:5000]

Continue reading...