[Python] Issues getting PyCaret/Fugue to work with Spark Backend

I am trying to use PyCaret with Fugue for a spark backend and I'm running into some issues with getting everything to work properly.

1. I can run a PyCaret experiment using spark as a back end if my data is local, but my data comes from spark, so it seems a waste to pull it from spark and then send it back to perform model training. However, if I try to load the data in spark and run a PyCaret experiment, I get: RuntimeError: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.

Is there a way to make this happen so that PyCaret process the data directly from spark?

1. If I execute a PyCaret experiment using spark as a back end by wrapping my PyCaret code and calling it through transform, it runs the wrapper function on each partition, as expected, but then I get 1 distinct model for each partition. How, then, can I combine them into a single model to run on an out-of-sample dataset?

for question 1) I ran a normal experiment I tried this:

df = spark.read.parquet("hdfs:///myData.parquet")
df = df.sampleBy("target", fractions={0: 0.5, 1: 0.25}, seed=0)
train_df, test_df = df.randomSplit([0.6, 0.4])

s = setup(train_df, test_data=test_df, target='target', log_experiment=True, experiment_name='Experiment 1', use_gpu=False, log_plots=True, n_jobs=-1)

models = compare_models()


for question 2) I ran

best = transform(df, wrapper, schema=schema, engine=spark).toPandas()


and got a full model set for each partition, all distinct from each other

Continue reading...