[Python] Overfitting on train data for a customer churn dataset

I'm learning some basic machine learning for classification purposes. I trained a Logistic Regression model on a customer churn dataset from Kaggle (https://www.kaggle.com/datasets/muhammadshahidazeem/customer-churn-). However, I'm experiencing a severe case of overfitting:

Train Accuracy: 0.94

Test Accuracy: 0.57

This indicates that my model generalizes poorly.

To avoid data leakage, I implemented a Pipeline to handle categorical features and ensure proper preprocessing. Additionally, I used hyperparameter tuning with Stratified K-Fold cross-validation to optimize the model parameters.

Here is my code after importing the necessary libraries:

Train_data = 'customer-churn-dataset-training-master.csv'
Test_data = 'customer-churn-dataset-testing-master.csv'


Pipeline Setup

ordinal_encoded_features = ['Subscription Type', 'Contract Length']
label_encoded_features = ['Gender']

ordinal_cat = [
['Basic', 'Standard', 'Premium'],
['Monthly', 'Quarterly', 'Annual']
]

preprocessor = ColumnTransformer([
('onehot', OneHotEncoder(handle_unknown='ignore', drop='first'), label_encoded_features),
('ordinal', OrdinalEncoder(categories=ordinal_cat, handle_unknown='use_encoded_value', unknown_value=-1),
ordinal_encoded_features)
], remainder='passthrough')

Log_pipeline = {
'Logistic Regression': Pipeline([
('preprocessor', preprocessor),
('scaler', MinMaxScaler()),
('classifier', LogisticRegression(random_state=42))
])
}


Hyperparameter Tuning with Stratified K-Fold

param_grid = {
'Logistic Regression': {
'classifier__C': [0.1, 1, 5, 10, 15, 20],
'classifier__penalty': ['l1', 'l2'],
'classifier__solver': ['liblinear', 'saga'],
'classifier__class_weight': [None, 'balanced']
}
}

kf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
log_model = {}
best_score = {}

for name, pipe in Log_pipeline.items():
Random_search = RandomizedSearchCV(pipe, param_grid[name], cv=kf, scoring='f1', n_jobs=-1,
random_state=42, n_iter=20)
Random_search.fit(x_train, y_train)

log_model[name] = Random_search.best_estimator_
best_score[name] = Random_search.best_score_


What I’ve Tried:

Used MinMaxScaler for feature scaling.

Implemented Stratified K-Fold cross-validation.

Tuned hyperparameters using RandomizedSearchCV.

Despite this, the test accuracy remains significantly lower than the training accuracy.

Question:

What could be the reason for this overfitting issue? Are there any improvements I can make to improve generalization performance?

Continue reading...