[Python] Why does my RandomForestClassifier overfit despite using cross-validation?

I'm working on a binary classification problem using RandomForestClassifier from scikit-learn. My dataset has ~10,000 rows and ~20 numerical features. I used train_test_split and cross_val_score, but my model performs very well on training data and poorly on the test set. Here is my code:

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import classification_report

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

clf = RandomForestClassifier(n_estimators=100, random_state=42)
clf.fit(X_train, y_train)

# Cross-validation
cv_scores = cross_val_score(clf, X_train, y_train, cv=5)
print("Cross-validation scores:", cv_scores)

# Evaluation on test set
y_pred = clf.predict(X_test)
print(classification_report(y_test, y_pred))


Observed Output:

CV Accuracy: ~95%

Test Accuracy: ~80%

Even with cross-validation, my test accuracy drops significantly, suggesting overfitting.

Reducing number of trees and depth (max_depth)

Setting max_features to 'sqrt'

Using min_samples_leaf=5

What are common reasons a RandomForestClassifier might still overfit even when cross-validation scores are high? Is there a better way to validate model generalization beyond cross_val_score?

Continue reading...