[Python] CONTEXT_ONLY_VALID_ON_DRIVER : how to access/pass the spark context pandas_udf in...

I have a python file (main.py) executing in adf, and i have another file (udf.py) for the pandas udfs. I am importing the udf.py functions into main.py like below:

from udf import func1, func2


when this is executing in adf we are getting error saying: CONTEXT_ONLY_VALID_ON_DRIVER

So, how can i pass the spark context or how to solve this issue.

everything works fine if kept in same file as plain script than modularized. I am suspecting it is something to do with worker and driver nodes. but need some light on the same.

Continue reading...