[Python] The cost function of logistic regression does not converge

I'm trying to build a logistic regression model with python using gradient descent method. The cost function of a logistic regression model I know is

formula 1

where

formula 2

and the gradient of the cost function is defined by

formula 3

Then I use iris dataset to build a binary classification model, here's my code

import numpy as np
from sklearn.datasets import load_iris

iris = load_iris()
X = iris.data
y = iris.target
mask = y < 2
X = X[mask]
y = y[mask]

m = X.shape[0] # the number of samples
n = X.shape[1] # the dimension of each sample

def sigmoid(x):
return 1 / (1 + np.exp(-x))

def cost(w: np.ndarray, b: float) -> float:
y_hat = sigmoid( X.dot(w) + b )
loss = -(y * np.log(y_hat) + (1 - y) * np.log(1 - y_hat))
return np.mean(loss)

def gradient(w: np.ndarray, b: float) -> np.ndarray:
y_hat = sigmoid( X.dot(w) + b )
grad = np.zeros(n+1)
for j in range(n):
grad[j] = sum( (X[i,j] * (y_hat - y)) for i in range(m) ) / m
grad[n] = sum((y_hat - y) for i in range(m)) / m
return grad

vec = np.zeros(n+1)
lr = 0.01
for i in range(100000):
w = vec[:n]
b = vec[n]
print(i, cost(w,b))
vec -= lr * gradient(w,b)


It turns out that no matter how many times I iterate (even 300k times), the cost function will not converge to the minimum (It keeps descending). What's the problem?

The result of my code

Continue reading...