[Python] CUDA error 700: an illegal memory access was encountered"

Hey I encounter error:

Application terminated with error: ??+0 (0x709D9F003D8A)
??+0 (0x709D9E884BA4)
??+0 (0x709D9E9F388C)
??+0 (0x709D9FF2FCF5)
??+0 (0x709D9FF31448)
??+0 (0x709D9EB84E21)
??+0 (0x709D9EB84C8A)
??+0 (0x709E0509CAA4)
??+0 (0x709E05129C3C)

(TCatBoostException) catboost/cuda/cuda_lib/cuda_base.h:243: CUDA error 700: an illegal memory access was encountered
Terminating due to uncaught exception 0x5e68a9b0410 what() -> "catboost/cuda/cuda_lib/cuda_base.h:243: CUDA error 700: an illegal memory access was encountered"
of type TCatBoostException
Aborted (core dumped)


I found that if MAE or Depthwise or GPU removed from this args, it works and no error appears.

CPU: Intel Core i7-10700F GPU: GeForce RTX 3070 OS: Ubuntu 24.04.1 LTS

I was unable to reproduce the issue with minimal code, it only stops working with my huge dataset (30GB approx.).

Here is the snippet i use

data_frame_res["residual"] = y_pred_base - y_test.values
train_data_res, test_data_res = train_test_split(data_frame_res, test_size=0.2, random_state=_SEED_VALUE_)
x_train_res = train_data_res.drop(["residual"], axis=1)
y_train_res = train_data_res["residual"]

model_kwargs={
"iterations": 750,
"learning_rate": 0.05,
"depth": 12,
#"l2_leaf_reg": 6,
#"bootstrap_type": "Bayesian",
"task_type": "GPU",
"bagging_temperature": 0.5,
"grow_policy": "Depthwise",
"loss_function": "MAE"
}

model_res = price_boost.train(
model_kwargs=model_kwargs,
train_kwargs={
"X": x_train_res,
"y": y_train_res,
"cat_features": _CAT_FEATURES_,
"text_features": ["normalized_equipment"],
#"early_stopping_rounds": 100,
"verbose": 10,
#"eval_set": (x_val, y_val)
}
)


Where x_train_res is DataFrame with bunch of categorical + numerical features + one text feature. The y_train_res is just pd.Series with float64 numbers.

The reason why I was not able to reproduce the problem could be that my minimal dataset was either too small or had less features, not sure.

Continue reading...