I am dealing with a highly imbalanced multi-class classification problem, and came across the focal loss function on tensorflow, after implementing it, I still see a high initial loss of ~4000-5000. I am using sparse labels in my final layer, so I adjusted the focal loss function to convert my sparse labels to one-hot encoded and verified that everything was working manually, and I am returning the reduced mean of the focal loss vector computed across each batch, here is the code: class SparseFocalLoss(tf.keras.losses.Loss): def __init__(self, gamma = 2.0, alpha = 0.25, from_logits = False, name = "sparse_focal_loss", reduction = tf.keras.losses.Reduction.AUTO): super().__init__(reduction = reduction, name = name) self.gamma = gamma self.alpha = alpha self.from_logits = from_logits def call(self, Y_true, Y_pred): if self.from_logits: Y_pred = tf.nn.softmax(Y_pred, axis = -1) #Gets total classes Classes = tf.shape(Y_pred)[-1] #sets classes to shape m*n = [1, nClasses] Y_true = tf.cast(tf.reshape(Y_true, [-1]), tf.int32) #Converts above to OHE where y (class) = 1, 0 elsewhere #[0, 1, 0, 0, 0, 0, ..., 0] for nClasses Y_true_OHE = tf.one_hot(Y_true, depth=Classes) #Shapes the prediction in the same shape as Y_true Y_pred = tf.reshape(Y_pred, [-1, Classes]) #Clips the values to ensure no 0's or 1's Y_pred = tf.clip_by_value( t=Y_pred, clip_value_min=1e-12, clip_value_max=1. - 1e-12) #FL = -a * (1 - Pt)**gamma * log(Pt) Pt = tf.reduce_sum(Y_true_OHE * Y_pred, axis = -1) FL = -1 * self.alpha * tf.pow(1 - Pt, self.gamma) * tf.math.log(Pt) return(tf.reduce_mean(FL, axis = -1)) Mathematically, everything looks good I validated this on pen & paper but I am confused as to why the initial loss here is about as high as using the base sparse categorical crossentropy loss. I compute an initial bias for my classes as well, these are applied on a per class basis - adapted from initial bias for a binary classification problem. Continue reading...
