[Python] ERROR: Spark User did not initialize spark context

Im getting this error from spark running with hadoop and pyspark

ApplicationMaster: Waiting for spark context initialization...
25/05/12 23:56:11 INFO ApplicationMaster: Final app status: FAILED, exitCode: 13
25/05/12 23:56:11 ERROR ApplicationMaster: Uncaught exception:
java.lang.IllegalStateException: User did not initialize spark context!


This is my code:

import pyspark
import numpy as np
from pyspark.sql import SparkSession
from utils import YtXSparkJob
import random


def main(args):
# inicializar spark
spark = SparkSession.builder.appName("sPCA").getOrCreate()
sc = spark.sparkContext # así obtienes el SparkContext moderno
random.seed(42)
np.random.seed(42)
....
# some code
YtXSparkJob(..., spark)




if __name__ == '__main__':
"""Run sPCA"""
#... some code
main(args)



I am using a utils.py file and it seems the error is coming from there, looking at the UI of spark cluster nodes. Utils.py has some helper functions I need, these functions need the spark context so i pass it as a parameter

def YtXSparkJob(Y, Ym, Xm, CM, D, d, spark):
sc = spark.sparkContext
# I need to use sc in this function to broadcast some variables


And this is how I run the spark code in my terminal

spark-submit --master yarn --deploy-mode cluster --archives env.tar.gz#environment --py-files accumulators.py utils.py sPCA.py --input "hdfs:///grupoh/challenge/input/datos_1.txt" --dim 2 --maxIters 10 --output "hdfs:///grupoh/challenge/output/datos_1_spark_pca.txt"

Continue reading...