[Python] Why do my OLS and gradient descent results differ significantly for linear...

I'm writing a paper comparing the performance of ordinary least squares (OLS) to conventional gradient descent (CGD) for linear regression. While my OLS implementation using SciPy matches expected results (e.g., statsmodels), my CGD implementation produces vastly different parameter estimates, even after standardizing the data, using a low learning rate, and running for many epochs.

Here's what I've done:

A. OLS using SciPy:

I compute the coefficients using the closed-form normal equation and pseudoinverse. Normal equation I'm using is as per usual:



The results match statsmodels:

beta_encoding_scipy = pinv(X_with_intercept.T @ X_with_intercept) @ X_with_intercept.T @ y

OLS Results:

Intercept -22.507045
Hours Studied 2.852729
Previous Scores 1.018319
Sleep Hours 0.480321
Sample Question Papers Practiced 0.193910
Extracurricular Activities_No -11.561869
Extracurricular Activities_Yes -10.945176
dtype: float64


B. Gradient Descent Implementation:

I standardized the training data and implemented gradient descent with the following setup:

• Learning rate: 0.001
• Epochs: 5000
• Initial values: All coefficients set to 0

My implementation:

def gradient_descent(features, label, learning_rate, epochs, precision):
# Initialization
X_augmented = np.hstack((np.ones((features.shape[0], 1)), features))
beta = np.zeros(X_augmented.shape[1]) # Initialize coefficients

for epoch in range(epochs):
predictions = X_augmented @ beta
residuals = predictions - label
gradient = (2 / len(label)) * X_augmented.T @ residuals
beta = beta - learning_rate * gradient
return beta

CGD Results:

Intercept 55.142748
Hours Studied 7.392875
Previous Scores 17.722890
Sleep Hours 0.819071
Sample Question Papers Practiced 0.531543
Extracurricular Activities_No -0.149439
Extracurricular Activities_Yes 0.149439
dtype: float64


Observed Discrepancy:

The results from CGD are far from those of OLS, and I am unsure why. I suspect it could be due to the data splitting process (train/test split with StandardScaler applied).

Questions:

• Shouldn't CGD produce results similar to OLS if properly trained?
• Could the train-test split or scaling process be affecting CGD performance?
• Is there anything wrong with my CGD implementation or parameter initialization?

Continue reading...