[Python] Validation accuracy and AUC not increasing. Please help me understand where I went...

I am a student currently working on this project. This is my architecture code:

# Load pre-trained InceptionResNetV2
base_model = InceptionResNetV2(weights='imagenet', include_top=False, input_shape=(300, 300, 3))

# Freeze all layers initially
for layer in base_model.layers:
layer.trainable = False

# --- ADD CUSTOM CLASSIFICATION HEAD WITH MULTI-HEAD ATTENTION ---
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = BatchNormalization()(x)
x = Dropout(0.3)(x)

# --- IMPLEMENT MULTI-HEAD ATTENTION MECHANISM ---
# Reshape to prepare features for attention
x_reshaped = Reshape((1, x.shape[-1]))(x)

# Implement multi-head attention (3 heads)
num_heads = 3
head_outputs = []

for i in range(num_heads):
# Each attention head
attention = Dense(64, activation='relu', name=f'attention_projection_{i}')(x_reshaped)
attention = Dense(1, activation='tanh', name=f'attention_score_{i}')(attention)
attention = Flatten()(attention)
attention_weights = Activation('softmax')(attention)
attention_weights = RepeatVector(x.shape[-1])(attention_weights)
attention_weights = Permute([2, 1])(attention_weights)

# Apply attention weights
head_output = Multiply()([x_reshaped, attention_weights])
head_output = Lambda(lambda x: K.sum(x, axis=1))(head_output)
head_outputs.append(head_output)

# Combine attention heads
if num_heads > 1:
context_vector = Concatenate()(head_outputs)
# Project back to original dimension
context_vector = Dense(x.shape[-1], activation='relu')(context_vector)
else:
context_vector = head_outputs[0]

# --- SKIP CONNECTION ---
# Add skip connection from after GlobalAveragePooling to after attention
context_vector = Add()([context_vector, x])

# --- FINAL LAYERS WITH ENHANCED REGULARIZATION ---
x = GaussianNoise(0.02)(context_vector)
x = Dense(256, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(0.005))(x)
x = BatchNormalization()(x)
x = Dropout(0.3)(x)
x = Dense(128, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(0.005))(x)
x = BatchNormalization()(x)
x = Dropout(0.3)(x)

# --- OUTPUT LAYER ---
output = Dense(1, activation='sigmoid')(x)

# --- CREATE MODEL ---
model = Model(inputs=base_model.input, outputs=output)

# --- COMPILE MODEL ---
# Add gradient clipping to prevent exploding gradients

optimizer = Adam(learning_rate=0.0001, clipnorm=1.0)
model.compile(optimizer=optimizer,
loss='binary_crossentropy',
metrics=['accuracy', 'AUC', 'Precision', 'Recall'])

# # Print model summary
# print(model.summary())

# --- ENHANCED CALLBACKS ---
lr_scheduler = ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=5, verbose=1)
early_stop = EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True, verbose=1)
checkpoint = ModelCheckpoint('best_model.h5', monitor='val_auc', mode='max',
save_best_only=True, verbose=1)
callbacks_list = [lr_scheduler,early_stop, checkpoint]

# --- TRAIN THE MODEL ---
history = model.fit(
X_train,Y_train,
epochs=70,
validation_data=(X_test, Y_test),
batch_size=16,
verbose=1,
callbacks=callbacks_list
)


This is my model architecture

Dataset overview: 1260 coloured images (railway track defect dataset), labelled data, clean dataset with equal number of classes (i.e 50% defective and 50% non defective)

Image pixel have already been normalized (i.e [0,1] before feeding into dataset.

What I tried:

1. I tried lr_scheduler cosine decay, not working
2. Data augmentation reduced the performance slightly
3. Tweaking the values of dropout, learning rate and and gaussian noise but still not working
4. Fine tuning the later layers in base model not working

The validation accuracy is fixed between 0.91-0.92 and validation AUC did not improve from 0.9624. I want to know how I can improve its performance

Continue reading...