[Python] How to Implement Backpropagation Without Auto-Differentiation for a Feedforward...

I am working on a deep learning assignment that requires implementing a feedforward neural network (FNN) from scratch using only NumPy (without TensorFlow, PyTorch, or other auto-differentiation tools). The network has three layers (2048, 512, 10 neurons), uses ReLU and Softmax activation, and is optimized with mini-batch stochastic gradient descent (SGD).

The assignment requires implementing forward propagation, backpropagation, and weight updates manually, ensuring that I correctly compute gradients for each layer using the backpropagation algorithm.

However, I am struggling with correctly computing the gradients for each layer, especially when handling ReLU activation's derivative and cross-entropy loss's gradient with Softmax. I want to confirm whether my gradient computations and weight updates are correct.

import numpy as np

def relu(x):
return np.maximum(0, x)

def relu_derivative(x):
return (x > 0).astype(float) # ReLU derivative (1 if x > 0, else 0)

def softmax(x):
exp_x = np.exp(x - np.max(x, axis=1, keepdims=True)) # Stability trick
return exp_x / np.sum(exp_x, axis=1, keepdims=True)

def cross_entropy_loss(y_pred, y_true):
return -np.mean(np.sum(y_true * np.log(y_pred + 1e-9), axis=1)) # Prevent log(0)

def softmax_cross_entropy_grad(y_pred, y_true):
return y_pred - y_true # Softmax + cross-entropy derivative

def gradient_check():
np.random.seed(42) # Ensure reproducibility

# Fake input (batch_size=3, input_dim=5)
X = np.random.randn(3, 5)
W1 = np.random.randn(5, 4)
b1 = np.zeros((1, 4))

# Fake one-hot encoded labels (batch_size=3, num_classes=4)
Y = np.array([[0, 1, 0, 0],
[1, 0, 0, 0],
[0, 0, 1, 0]])

Z1 = np.dot(X, W1) + b1
A1 = relu(Z1)
Y_pred = softmax(A1)
loss = cross_entropy_loss(Y_pred, Y)

dL_dA1 = softmax_cross_entropy_grad(Y_pred, Y) # Gradient w.r.t. softmax output
dL_dZ1 = dL_dA1 * relu_derivative(Z1) # Chain rule with ReLU

print("Y_pred (Softmax Output):\n", Y_pred)
print("Loss:", loss)
print("Gradient w.r.t. Softmax Output (dL/dA1):\n", dL_dA1)
print("Gradient after ReLU Derivative (dL/dZ1):\n", dL_dZ1)

gradient_check()


I implemented a feedforward neural network (FNN) using NumPy only and manually computed backpropagation. I expected the loss to decrease and accuracy to improve, but instead, the loss fluctuates, accuracy stays low, and sometimes gradients explode or become NaN. I suspect issues with ReLU’s derivative and Softmax + cross-entropy gradient calculations and need to verify if my backpropagation is correct.

Continue reading...