[Python] Why does SequentialFeatureSelector return at most "n_features_in_ - 1" predictors?

I have a training dataset with six features and I am using SequentialFeatureSelector to find an "optimal" subset of the features for a linear regression model. The following code returns three features, which I will call X1, X2, X3.

sfs = SequentialFeatureSelector(LinearRegression(), n_features_to_select='auto',
tol=0.05, direction='forward',
scoring='neg_root_mean_squared_error', cv=8)
sfs.fit_transform(X_train, y_train)


To check the results, I decided to run the same code using the subset of features X1, X2, X3 instead of X_train. I was expecting to see the features X1, X2, X3 returned again, but instead it was only the features X1, X2. Similarly, using these two features again in the same code returned only X1. It seems that the behavior of sfs is always to return a proper subset of the input features with at most n_features_in_ - 1 columns, but I cannot seem to find this information in the scikit-learn docs. Is this correct, and if so, what is the reasoning for not allowing sfs to return the full set of features?

I also checked to see if using backward selection would return a full feature set.

sfs = SequentialFeatureSelector(LinearRegression(), n_features_to_select='auto',
tol=1000, direction='backward',
scoring='neg_root_mean_squared_error', cv=8)
sfs.fit_transform(X_train, y_train)


I set the threshold tol to be a large value in the hope that there would be no satisfactory improvement from the full set of features of X_train. But, instead of returning the six original features, it only returned five. The docs simply state

If the score is not incremented by at least tol between two consecutive feature additions or removals, stop adding or removing.
​

So it seems that the full feature set is not being considered during cross-validation, and the behavior of sfs is different at the very end of a forward selection or at the very beginning of a backwards selection. If the full set of features outperforms any proper subset of the features, then don't we want sfs to return that possibility? Is there a standard method to compare a selected proper subset of the features and the full set of features using cross-validation?

Continue reading...