[Python] Scratch implementation of ridge regression

I am trying to implement Ridge regression but I feel like I am missing something with the Python operators. Here is my code:

import numpy as np


x = np.random.rand(10, 2)
y = np.random.rand(10, 1)
lambda_reg = 0.1
alpha = 0.1
num_iterations = 100000


X_train = np.hstack((np.ones((x.shape[0], 1)), x))

def ridge_regression_gradient_descent(X, y, lambda_reg, alpha, num_iterations):
n, p = X.shape
B = np.zeros(p)
# Gradient descent loop
for _ in range(num_iterations):
y_pred = X.dot(B).reshape(-1, 1)
gradient_B0 = - (1/n) * np.sum(y - y_pred)
gradient_B = - (1/n) * (X[:, 1:].T @ (y - y_pred)) +
lambda_reg * B[1:].reshape(-1, 1) # Gradients for B1 to Bp

B[0] -= alpha * gradient_B0
B[1:] -= alpha * gradient_B.reshape(-1)
return B

B = ridge_regression_gradient_descent(X_train, y, lambda_reg, alpha, num_iterations)
print(B)


Is anyone able to see what I am doing wrong?

I tried multiple changes in the code on the matrix multiplications + also reshaping everything in the right format. I get actually 3 Beta's so this is ok but I don't get anything close to what I get with the formula: B = (X.T X + lambda * I)^-1 * X.T Y

Continue reading...