I made the plot below: Please ignore the sort of ugliness as I am not done, but you can just focus on the top row. The Y axis is the conditioned expectation that I calculated (see code at the bottom) which is a probability in [0,1], and the X axis is my independent variable (area or bedroom in the bottom). I am a bit confused by my plot, all of the expectations are significantly above 0.5, which I take to mean that I would get more positive values than not, but these are all supposed to be minority classes. I used SMOTE to get them to be 50/50 balances, but I still think the conditional expectations shouldn't be this high, especially since my model performs very well. Below is my monte-carlo integration for the conditional expectation. def calculate_partial_dependence(model, X, feature_index, grid_resolution=100, sample_num = 20000): """ Calculate the partial dependence of a feature on the model's predictions. Parameters: - model: The model - X: Dataset - feature_index: The index of the feature for which to calculate the partial dependence. - grid_resolution: The number of points in the grid for the feature. - Sample_num: number of uniform draws to do for each point in the grid Returns: - feature_values: The values of the feature at which the partial dependence is calculated. - pd_values: The partial dependence values. """ #Fuck sklearn we ball ourselves out #Min and maxes f = X[:, feature_index] X_deleted = np.delete(X,feature_index, axis = 1) a = np.min(f) b = np.max(f) values = np.linspace(start = a, stop = b, num = grid_resolution) expectations = np.zeros_like(values) KDE = KernelDensity() if X_deleted.shape[1] != 0: for idx, condition in enumerate(values): #Find nearby X, and then compute "conditional ish" density, since we don't nessecarily have enough values for a real conditional density xl = 0 q = 5 while xl <= 10: r = (condition - q*(b-a)/grid_resolution, condition + q*(b-a)/grid_resolution) #Filter indices by r X_temp = X_deleted[(f >= r[0]) & (f <= r[1]),:] xl = X_temp.shape[0] q += 1 #Generate sample distribution generator = KDE.fit(X_temp).sample(sample_num) col = np.ones((sample_num, 1))*condition sample = np.concatenate((col, generator), axis = 1) #Predict probabilities with model output = model.predict_proba(sample)[:, 1] #Average out since monte-carlo expectations[idx] = np.mean(output) else: #The null model only has one variable, so I don't need to ``condition'' expectations = model.predict_proba(values)[:, 1] return values, expectations Am I misunderstanding something here? I tried normal librarys like SKlearn for the conditional expectation as well and they gave me about the same answer. Is something fishy with my model or am I not understanding what I'm looking at. Note: This is crossposted from stats/cross-validated stack exchange, I did not recieve any answers over there but it is still up. Continue reading...
