[Python] Reproduce LGBMRegressor predictions by manually aggregate the values

I am trying to reproduce by myself the LGBMRegressor predictions so when I succeed I will switch mean with median. But for now it seems that I am not able to. Here is a simple script that I created for the proposes of checking if I can reproduce the results.

I need reg_y_hat to be the same as self_y_hat. what am I missing? if I know what samples in train fall to each leaf I can aggregate the prediction myself...

import numpy as np
import lightgbm as lgb
from sklearn.model_selection import train_test_split

# Generate some random regression data
np.random.seed(42)
X = np.random.rand(100, 5)
y = 4 * X[:, 0] - 2 * X[:, 1] + np.random.rand(100) * 0.1

# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train the LGBMRegressor
model = lgb.LGBMRegressor(objective='regression', n_estimators=2, learning_rate=0.1, random_state=42)
model.fit(X_train, y_train)

# Regular predict:
reg_y_hat = model.predict(X_test)

# Get the initial prediction (mean of y_train)
init_pred = np.mean(y_train)

# Get the train leaf values
train_leaf_indices = model.predict(X_train, pred_leaf=True)
leaf_samples = {(i, leaf_id): [] for i in range(model.n_estimators) for leaf_id in np.unique(train_leaf_indices[:, i])}

# Store corresponding target values for each leaf
for i, row in enumerate(train_leaf_indices):
for j, leaf_id in enumerate(row):
leaf_samples[(j, leaf_id)].append(y_train)

# Compute avg for each leaf:
leaf_agg = {}
for key, values in leaf_samples.items():
leaf_agg[key] = np.mean(values)

# Predict by aggregating the mean values and adding the initial prediction:
preds = []
test_leaf_indices = model.predict(X_test, pred_leaf=True)
for row_indices in test_leaf_indices:
row_pred = init_pred
for i, leaf_index in enumerate(row_indices):
row_pred += model.learning_rate * (leaf_agg[(i, leaf_index)] - init_pred) # only the residual contribution of the leaf after initial prediction
preds.append(row_pred)
self_y_hat = np.array(preds)

# Verify the results
print('Difference between reg_y_hat and self_y_hat:', np.abs(reg_y_hat - self_y_hat).sum())

Continue reading...