[Python] How to correctly use RFECV for feature selection in a Scikit-Learn pipeline with a...

I am working on the Kaggle House Price Prediction competition and have built a Scikit-Learn pipeline that includes:

• 
  Preprocessing (handling missing values, scaling, encoding)
  
  
• 
  Feature Engineering
  
  
• 
  Encoding (LabelEncoder, Ordinal Encoders, etc.)
  

My full pipeline looks like this:

# Full pipeline
full_pipeline = Pipeline([
('preprocessor', preprocessor), # Preprocessing pipeline
('feature_engineering', feature_engineer), # Feature engineering pipeline
('encoder', encoding_pipeline), # Encoding pipeline
])



I have tested this pipeline using a Decision Tree baseline model with 5-fold cross-validation to prevent data leakage:

# Create DecisionTreeRegressor model
decision_tree = DecisionTreeRegressor(random_state=42)

# Create baseline decision tree pipeline
baseline_decision_tree_pipeline = Pipeline([
('preprocessing', full_pipeline),
('decision_tree', decision_tree)
])

# Define CV strategy
kf = KFold(n_splits=5, shuffle=True, random_state=42)

# Get CV RMSE scores
cv_scores = cross_val_score(baseline_decision_tree_pipeline,
X, # Raw unprocessed data
y, # Target
cv=kf,
scoring='neg_root_mean_squared_error',
n_jobs=-1)

print("CV RMSE scores:", -cv_scores)
print("Average CV RMSE:", (-cv_scores).mean())


I now want to integrate RFECV (Recursive Feature Elimination with Cross-Validation) to select the best features while preventing data leakage from pre-processing steps like imputation. However, I am unsure about the best approach:

Option 1: RFECV inside the cross-validation loop

If I use RFECV inside the pipeline, then for each of the 5 folds, it will select a different subset of "optimal" features.

How do I decide on the final set of selected features across all folds?

Option 2: Fit the pre-processing pipeline on the training set and then perform RFECV

This seems like it would introduce data leakage, since the training folds used for RFECV would have information leaked from their corresponding test folds when the pre-processing was done.

Questions

1. 
   Is it correct to include RFECV inside the pipeline? This ensures feature selection happens within each fold, but how do I extract the final set of optimal features? I have attempted a few different things but cannot get what I am looking for.
   
   
2. 
   Am I well off and there is a better way to do this altogether?
   

This is my current attempt at including RFECV in the pipeline (Option 1) which returns the 5 RMSE scores:

# Perform Recursive Feature Elimination with Cross-Validation
selector = RFECV(estimator=decision_tree, step=1, cv=kf, scoring="neg_root_mean_squared_error")

# Create baseline decision tree pipeline with feature selection
dt_pipeline = Pipeline([
('preprocessing', full_pipeline),
('feature_selection', selector),
('decision_tree', decision_tree)
])

cv_scores_rfecv = cross_val_score(
dt_pipeline,
X, # Raw unprocessed data
y, # Target
cv=kf,
scoring='neg_root_mean_squared_error',
n_jobs=-1
)

print("CV RMSE scores:", -cv_scores_rfecv)
print("Average CV RMSE:", (-cv_scores_rfecv).mean())

Continue reading...