[Python] Multivariate Linear Regression Gradient Descent Always Converges to Zero Weights (w =...

I'm implementing multivariate linear regression using gradient descent on the Iris dataset (sklearn.datasets.load_iris), but my weights (w) always converge to [0,0,0] regardless of hyperparameter tuning.

Problem Description

• 
  Input Data (X): First 3 features of Iris dataset (sepal length, sepal width, petal length).
  
  
• 
  Target (y): The 4th feature (petal width).
  
  
• 
  Initialization: w = [2,2,2], b = 0, learning rate lr = 0.01.
  
  
• 
  Gradient Descent: 200 iterations.
  

Observations

• 
  The loss decreases but eventually stabilizes with w = [0,0,0].
  
  
• 
  Gradient calculations seem correct (see code below).
  
  
• 
  Tried adjusting lr (0.001, 0.1) and iterations (500, 1000), but no improvement.
  

Source code:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris

data = load_iris().data
X = data[:,:3]
y = data[:,3]

m = len(X)

def j(w, b):
sum = 0
for i in range(m):
sum += ((np.dot(w, X) + b - y) ** 2)
sum /= m
return sum

def dj_dw(w,b,j):
sum = 0
for i in range(m):
sum += (X[i,j] * (np.dot(w, X) + b - y))
sum *= 2
sum /= m
return sum

def dj_db(w,b):
sum = 0
for i in range(m):
sum += (np.dot(w, X) + b - y)
sum *= 2
sum /= m
return sum

w = np.array([2,2,2])
b = 0
lr = 0.001
Loss = []

for i in range(200):
tmp_w0 = w[0] - lr * dj_dw(w,b,0)
tmp_w1 = w[1] - lr * dj_dw(w,b,1)
tmp_w2 = w[2] - lr * dj_dw(w,b,2)
tmp_b = b - lr * dj_db(w,b)
w[0] = tmp_w0
w[1] = tmp_w1
w[2] = tmp_w2
b = tmp_b
Loss.append(j(w,b))

print(w,b)
plt.plot(range(200),Loss)
plt.show()

Continue reading...