I am currently using the scikit-learn interface for XGBoost in my project. However, I have an extremely large dataset, and each time I call .fit, the data is converted to a DMatrix, which is very time-consuming, especially when using a GPU that trains relatively fast. I've benchmarked using the native interface by using a single DMatrix for each fit, and the results show a significant difference (14s per fit vs 0.9s per fit). The problem is, I need a scikit-learn model so that it works with the rest of my program. Is there a way to use a DMatrix with the scikit-learn interface in XGBoost, or any workarounds to avoid the repeated conversion to DMatrix while still maintaining compatibility with scikit-learn? See the below code to get a reproducible way to cause this issue. from sklearn.datasets import make_classification from xgboost import XGBClassifier import xgboost as xgb # Large synthetic dataset X, y = make_classification(n_samples=500_0000, n_features=20, n_informative=10, n_redundant=10, random_state=42) # scikit-learn t = time.time() model = XGBClassifier(tree_method="gpu_hist", gpu_id=0, predictor="gpu_predictor", max_bin=256) model.fit(X, y) print("scikit-learn interface: ", time.time() - t) # scikit-learn again t = time.time() model.fit(X, y) print("scikit-learn (2nd) interface: ", time.time() - t) print() # DMatrix dtrain = xgb.DMatrix(data=X, label=y) t = time.time() model = xgb.train({"tree_method": "gpu_hist", "gpu_id": 0, "predictor": "gpu_predictor"}, dtrain) print("native interface: ", time.time() - t) # DMatrix again t = time.time() model = xgb.train({"tree_method": "gpu_hist", "gpu_id": 0, "predictor": "gpu_predictor"}, dtrain) print("native (2nd) interface:: ", time.time() - t) Output: scikit-learn interface: 14.393212795257568 scikit-learn (2nd) interface: 14.048950433731079 native interface: 3.9494242668151855 native (2nd) interface:: 0.9888997077941895 As you can see, there is a big time discrepancy between scikit-learn and native. Continue reading...
