I am working on the loan default prediction data set available on Kaggle which has a highly skewed class distribution. The best model I have gotten so far is as follows using ExtraTreesClassifier: Best Model: {'bootstrap': True, 'ccp_alpha': 0.0, 'class_weight': {0: 1, 1: 5}, 'criterion': 'gini', 'max_depth': 10, 'max_features': 0.8, 'max_leaf_nodes': None, 'max_samples': None, 'min_impurity_decrease': 0.0, 'min_samples_leaf': 1, 'min_samples_split': 2, 'min_weight_fraction_leaf': 0.0, 'monotonic_cst': None, 'n_estimators': 300, 'n_jobs': None, 'oob_score': False, 'random_state': 42, 'verbose': 0, 'warm_start': False} ROC AUC Score: 0.7530 Classification Report: precision recall f1-score support 0 0.92 0.85 0.89 45170 1 0.30 0.47 0.36 5900 accuracy 0.81 51070 macro avg 0.61 0.66 0.63 51070 weighted avg 0.85 0.81 0.83 51070 I haven't tried XGboost and Catboost so far. I tried resampling but I didn't get much performance improvement. I tried also using monotonic_cst but it boosted the recall on the minority class at the cost of slight lowering of the precision as follows: ROC AUC Score: 0.7430 Classification Report: precision recall f1-score support 0 0.94 0.65 0.77 45170 1 0.21 0.71 0.32 5900 accuracy 0.66 51070 macro avg 0.58 0.68 0.55 51070 weighted avg 0.86 0.66 0.72 51070 I looked at the work of several people posted online in which they show very good performance with more than 95% recall and Precision on both classes but the issue is that they use the resampled data as the testing set which I feel is not a true measure of how well your model performs on the original data. If you actually split the data before resampling then the performance get hit severely. How can I further improve the model especially on minority class? What is the approach that is followed in industry? Continue reading...
