[Python] Why is my loss value increasing when I update the biases?

I am currently working on a convolutional neural network in python, and I am currently implementing back propagation. As of now I am just implementing it for the output layer, and I just wrote the code to calculate the derivatives of the bias values.

Here is the code:

def back_prop(self, learning_rate, d_values = None, desired_outputs = None):
self.d_biases = np.zeros((desired_outputs.size, self.biases.size))
self.d_weights = np.zeros((desired_outputs.size, *self.weights.shape))

#checks if the layer is an output to determine the method for finding the derivatives
if self.is_output_layer:

#finds the derivatives of all of the biases in the output layer given respects to the loss
self.d_biases = np.array([[float(softmax(self.output)[x] - 1) if desired_outputs == x else float(softmax(self.output)[x]) for x in range(self.biases.size)] for i in range(desired_outputs.size)])
self.d_biases *= learning_rate
self.d_biases = np.sum(self.d_biases, axis = 0, keepdims = True)

#updates parameters
self.biases += self.d_biases



desired_outputs is an array of the class indices of a training batch and self.output is a 2d array organized by batches x classes. To calculate loss, I use cross entropy and here is the code:

def check_loss(self, target_indices):
target_percents = self.output[range(len(self.output)),target_indices]
target_percents = np.clip(target_percents, 1e-7, 1-1e-7)
self.losses = -np.log(target_percents)
return np.mean(self.losses)


target_indices will be the same variable as desired_outputs.

Also, check_loss() is in a different class than back_prop() but self.output is still the same value for both.

I am using a simple spiral data pattern from the internet with and x and a y input and 3 classes with no hidden layers to test the code but every time I attempt to propagate a batch through the model the loss increases.

I believe it maybe has something to do with how I am calculating the derivatives, but I checked and to my understanding it works.

According to my knowledge, the derivative for a bias within the output layer is softmax activation of that neuron minus the predicted value for that activation but I could be wrong.

Any clue what is wrong?

Continue reading...