[Python] Keras multilabel training dataset completeness

I am attempting to train a multilabel classification model with Keras. The application will be in an industrial setting to classify defect images routinely captured during manufacturing. The expected output would be to correctly identify defect "A" or defect "B". However it is possible both defect "A" and "B" can occur together, hence the need for multilabel classification. (In reality there will be over 10 different defect classifications and multiple defects could theoretically be found in one image, if unlikely.)

I am new to ML but have followed tutorials/guides and been able to produce a multi-classification (single label) model that works well identifying whether the defect is "A" or "B". So far, so good. Now I want a model that can also identify "A" and "B".

As a test for the model development, my training dataset consists of "squares" (200 images with white squares with a random rotation and position on a black background) and "triangles" (also 200 images of white triangles of random size/rotation on a black background). I intentionally do not have training images with BOTH triangles and squares since I want to replicate the defect image training where I will not have a sufficient number of images with multiple defects. Intuitively, it seems this should not matter. All the model should do is tell me whether it thinks there is a square in the image, regardless if there is also a triangle.

However I am struggling with getting a multilabel classification model working. I feel like I'm missing something more high-level rather than just needing to tweak my model parameters, for example. The model is not correctly identifying test images with both squares and triangles, though it works well for when there is one or the other. I followed the various online guides to be sure the model has:

• sigmoid final activation
• binary_crossentropy loss
• Adam optimizer
• onehot multilabels produced with MultiLabelBinarizer()

I'm using the following module versions on a linux mint 22.1 system:

• python 3.12.3
• pandas 2.2.3
• keras 3.10.0
• numpy 2.3.1
• tensorflow 2.19.0
• scikit-learn 1.7.1
• pillow 11.1.0
• opencv-python 4.12.0.88

NOTE:

• Images are converted to grayscale (single channel) for the training since this will reflect the application to the defect images which are also grayscale. Bonus that it saves on memory usage.

CODE with extraneous comments removed


from PIL import ImageDraw, Image
import numpy as np

# SmallerVGGNet modules
import pandas as pd
from keras.models import Sequential
from keras.layers import BatchNormalization
from keras.layers import Conv2D
from keras.layers import MaxPooling2D
from keras.layers import Activation
from keras.layers import Flatten
from keras.layers import Dropout
from keras.layers import Dense
from keras import backend as K

# training modules
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from keras.optimizers import Adam
from keras.preprocessing.image import img_to_array
from sklearn.preprocessing import MultiLabelBinarizer
from sklearn.model_selection import train_test_split
#from pyimagesearch.smallervggnet import SmallerVGGNet
import matplotlib.pyplot as plt
from imutils import paths
import tensorflow as tf
import numpy as np
import argparse
import random
import pickle
import cv2
import os

tf.compat.v1.enable_eager_execution() # enable eager execution

class SmallerVGGNet:
@staticmethod
def build(width, height, depth, classes, finalAct="softmax"):
model = Sequential()
inputShape = (height, width, depth)
chanDim = -1
if K.image_data_format() == "channels_first":
inputShape = (depth, height, width)
chanDim = 1
model.add(Conv2D(64, (2, 2), padding="same", input_shape=inputShape))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(64, (2, 2), padding="same"))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(128, (2, 2), padding="same"))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(128, (2, 2), padding="same"))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))

# first (and only) set of FC => RELU layers
model.add(Flatten())
model.add(Dense(256))
model.add(Activation("relu"))
#model.add(BatchNormalization())
model.add(Dropout(0.5))
model.add(Dense(classes))
model.add(Activation(finalAct))
return model

# initialize the number of epochs to train for, initial learning rate, batch size, and image dimensions
EPOCHS = 50
BS = 32
IMAGE_DIMS = (200, 200, 1) # 1 channel for grayscale

# initialize the data and labels

data = []
labels = []

print("[INFO] loading images...")
for item in ['squares','triangles']:
image_path = f"Train/{item}"
imagePaths = sorted(list(paths.list_images(image_path)))
for imagePath in imagePaths:
# load the image, pre-process it, and store it in the data list
image = cv2.imread(imagePath, 0) # 0 reads as grayscale
image = img_to_array(image)
data.append(image)
labels.append([item])

# scale the raw pixel intensities to the range [0, 1]
data = np.array(data, dtype="float") / 255.0
labels = np.array(labels)
print("[INFO] data matrix: {} images ({:.2f}MB)".format(len(data), data.nbytes / (1024 * 1000.0)))

# binarize the labels using scikit-learn's special multi-label binarizer implementation
print("[INFO] class labels:")
mlb = MultiLabelBinarizer()
labels = mlb.fit_transform(labels)
# loop over each of the possible class labels and show them
for (i, label) in enumerate(mlb.classes_):
print("{}. {}".format(i + 1, label))

# partition the data into training and testing splits using 80% of the data for training and the remaining 20% for testing
(trainX, testX, trainY, testY) = train_test_split(data, labels, test_size=0.2, random_state=42)
# construct the image generator for data augmentation
aug = ImageDataGenerator(rotation_range=25, width_shift_range=0.1,
height_shift_range=0.1, shear_range=0.2, zoom_range=0.2,
horizontal_flip=True, fill_mode="nearest")

print("[INFO] compiling model...")
model = SmallerVGGNet.build(width=IMAGE_DIMS[1], height=IMAGE_DIMS[0],
depth=IMAGE_DIMS[2], classes=len(mlb.classes_), finalAct="sigmoid")
model.compile(loss="binary_crossentropy", optimizer=Adam(), metrics=["accuracy"])

# train the network
print("[INFO] training network...")
H = model.fit(
x=aug.flow(trainX, trainY, batch_size=BS),
validation_data=(testX, testY),
steps_per_epoch=len(trainX) // BS,
epochs=EPOCHS, verbose=1)

model.save("model_multilabel_gray.keras")
with open('history_gray.pkl', 'wb') as file:
pickle.dump(H.history, file)
with open('classes.pkl', 'wb') as file:
pickle.dump(mlb.classes_, file)


I have tried changing various parameters in the model, but the resulting model only finds one of the shapes in the image (only the triangle but not the square when both shapes are present.)

Continue reading...