[Python] Implementing sklearn.ensemble.GradientBoostingRegressor with...

I'm trying to create models that support multivariate output. One of the models I'm trying to use is the GradientBoostingRegressor which does not natively support multivariate output. There is a workaround by using MultiOutputRegressor which provides this functionality via a "wrapper" of some sort. Unfortunately, that wrapping does not work when trying to do hyperparameter tuning using RandomizedSearchCV.

The following works:

model_gradboost = MultiOutputRegressor(GradientBoostingRegressor(n_estimators=100, random_state=42))
model_gradboost.fit(X_train, y_train)
y_pred_gradboost = model_gradboost.predict(X_test)


But the following does not

gradboost_n_estimators = [100, 200, 300, 400, 500, 600, 700, 800]
gradboost_max_depth = [3, 4, 5, 6, 7, 8]
gradboost_eta = [0.01, 0.02, 0.04, 0.08, 0.012, 0.16]
gradboost_max_features = ["sqrt", "log2", None]
gradboost_alpha = [0.1, 0.2, 0.4, 0.6, 0.8, 0.9]
gradboost_subsample = [0.5, 0.6, 0.7, 0.8, 0.9, 1]
gradboost_loss = ["squared_error", "absolute_error", "huber", "quantile"]

gradboost_dict_params = {
"n_estimators": gradboost_n_estimators
, "max_depth": gradboost_max_depth
, "learning_rate": gradboost_eta
, "max_features": gradboost_max_features
, "alpha": gradboost_alpha
, "subsample": gradboost_subsample
, "loss": gradboost_loss
}

model_gradboost = MultiOutputRegressor(GradientBoostingRegressor(random_state=42))
random_search = RandomizedSearchCV(model_gradboost, param_distributions=gradboost_dict_params, n_iter=200, cv=5, scoring='r2', return_train_score=True)
random_search.fit(X_train, y_train)


and causes an error, namely:

ValueError: Invalid parameter 'subsample' for estimator MultiOutputRegressor(estimator=GradientBoostingRegressor(random_state=42)). Valid parameters are: ['estimator', 'n_jobs'].


How can I get around this problem?

Thanks in advance.

Continue reading...