I'm working on a regression task to predict soil organic matter (%) using Gradient Boosting. In order to validate my model, I'm comparing standard KFold (random splits) vs GroupKFold where groups are based on spatial clusters (means clustering of coordinates) My question: Am I misunderstanding how GroupKFold works when using spatial clusters as groups? Should increasing the number of clusters to match the number of samples result in performance similar to KFold? Or is there a fundamental reason (e.g., data leakage, overlapping clusters, or the way grouping affects variance) why convergence to KFold performance does not happen even when clusters approach sample-level granularity? Any insight on whether this is a misconception about the relationship between group-based CV and random CV would be appreciated. Additional Info: Each group in GroupKFold contains only data points from a single cluster. I used KMeans clustering on spatial coordinates to define these groups. Here’s the context: The leftmost plot in the image shows performance using regular KFold (random splits). The remaining plots show GroupKFold with increasing numbers of clusters (12, 150, 200, 600, 1219), used as spatial groups in GroupKFold. As I increase the number of clusters (with a goal of approaching the number of unique samples), I expected the performance of GroupKFold to eventually converge toward that of KFold. However, even with N Clusters = 1219 (roughly equal to number of samples), the performance (R², RMSE, Bias) does not match that of KFold. Continue reading...
