[Python] dask.delayed got error: It appears that you are attempting to reference SparkContext...

I use spark and dask to append into a Databricks delta table. Here a simplify code that I use:

ds = xr.open_zarr(path, consolidated=True, chunks={})


def clean(ii):


ds_bite = ds.isel(time=range(ii[0], ii[1])).to_dataframe()

schema = StructType([StructField(col, TimestampType() if col == 'time' else FloatType(), True)
for col in ds_bite.columns])

spark_df = spark.createDataFrame(ds_bite, schema)
spark_df.write.mode("append").saveAsTable('somewhere.my_tab')


delayed_results = [delayed(clean)(idx) for idx in index_start_end]
x = compute(*delayed_results, scheduler="processes")



the error are at line: x = compute(*delayed_results, scheduler="processes")

error: [CONTEXT_ONLY_VALID_ON_DRIVER] It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.

Continue reading...