[Python] How to use CUDA to train LGBM model from SynapseML

I have gone through the entire docs on SynapseML LightGBMRanker module. There was no attribute I could pass to tell the model to use CUDA for training. For e.g. for the plain LightGBM library, there is an attribute called device_type which can be set to cude by device_type="cuda" but there is no such thing for the SynapseML flavor of the LGBM model.

I need to use the LGBM via SynapseML because I'm working in a Databricks PySpark environment and SynapseML is well-integrated with Spark Dataframes.

This is my current initialization of the Model:

lgbm_ranker = LightGBMRanker(
featuresCol=output_feature_vector_col,
labelCol="label",
groupCol="request_id",
objective=objective_param,
labelGain=label_gain_param,
learningRate=learning_rate_param,
numIterations=num_iterations_param,
numLeaves=num_leaves_param,
maxDepth=max_depth_param,
minDataInLeaf=min_data_in_leaf_param,
featureFraction=feature_fraction_param,
baggingFraction=bagging_fraction_param,
baggingFreq=bagging_freq_param,
lambdaL1=lambda_l1_param,
lambdaL2=lambda_l2_param,
evalAt=eval_at_k_values,
metric="ndcg",
useBarrierExecutionMode=True,
verbosity=2
)


Another weird thing happening was that when I run the training loop in my Databricks Python Notebook, there is no training logging happening (no loss, nothing). JFYI, I am using an MLFlow Experiment to log all of this.

Thanks a lot!

Continue reading...