[Python] Why is an additional fitting performed after RandomizedSearchCV when refit=True?

I'm working on hyperparameter tuning using RandomizedSearchCV from scikit-learn and noticed that even though the refit parameter is set to True, an additional fitting step is performed after finding the best parameters.

Here's a minimal reproducible example:

from sklearn.datasets import make_classification
from sklearn.model_selection import RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier
from scipy.stats import randint

# Generate a synthetic dataset
X, y = make_classification(n_samples=1000, n_features=20, random_state=42)

# Define a model and parameter distribution
model = RandomForestClassifier(random_state=42)
param_dist = {
'n_estimators': randint(10, 100),
'max_depth': randint(3, 20),
}

# Perform RandomizedSearchCV
search = RandomizedSearchCV(
model, param_dist, n_iter=10, cv=3, random_state=42, refit=True
)
search.fit(X, y)

# Access the best estimator
best_model = search.best_estimator_

print(best_model)


I understand that the refit=True flag indicates that the best model should be refit on the entire dataset after hyperparameter tuning. However, why isn't the "best model" from the cross-validation directly returned as search.best_estimator_? What is the reasoning behind performing another fitting step on the full dataset?

I would appreciate an explanation about the design decision in scikit-learn and how this impacts the interpretation of the results. Thank you!

Continue reading...