[Python] How to fix '[errno 2] no such file or directory' error in apache-spark

Discussão em 'Python' iniciado por Stack, Abril 14, 2025.

  1. Stack

    Stack Membro Participativo

    I am trying to set up apache-spark with python and visual studio code. I followed a tutorial up to this point and am not getting any errors with my code but when I try to run the code

    spark-submit rdd/testingCode.py


    I get a long list of errors and warnings. It also says

    can't open file '/usr/local/opt/python-spark-tutorial/rdd/testingCode.py': [Errno 2] No such file or directory'


    although the file does exist with this path!

    I believe it may have to do with my python path? Help please!

    Here is the code I am trying to run:

    from pyspark import SparkContext, SparkConf

    conf = SparkConf().setAppName("word count").setMaster("local[3]")
    sc = SparkContext(conf = conf)

    inputIntegers = list(range(1,6))
    integerRdd = sc.parallelize(inputIntegers)
    for x in integerRdd.collect():
    print(x)


    And here is the error:

    WARNING: An illegal reflective access operation has occurred
    WARNING: Illegal reflective access by
    org.apache.hadoop.security.authentication.util.KerberosUtil
    (file:/usr/local/opt/spark/spark-2.3.3-bin-hadoop2.7/jars/hadoop-
    auth-2.7.3.jar) to method sun.security.krb5.Config.getInstance()
    WARNING: Please consider reporting this to the maintainers of
    org.apache.hadoop.security.authentication.util.KerberosUtil
    WARNING: Use --illegal-access=warn to enable warnings of further
    illegal reflective access operations
    WARNING: All illegal access operations will be denied in a future release
    2019-05-22 12:17:13 WARN NativeCodeLoader:62 - Unable to load
    native-hadoop library for your platform... using builtin-java
    classes where applicable


    /usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python:

    can't open file '/usr/local/opt/python-spark-
    tutorial/rdd/testingCode.py': [Errno 2] No such file or directory
    2019-05-22 12:17:13 INFO ShutdownHookManager:54 - Shutdown hook
    called
    2019-05-22 12:17:13 INFO ShutdownHookManager:54 - Deleting
    directory
    /private/var/folders/fn/mf51ynh97v71r4609l_l1j4m0000gn/T/spark-bdd580eb-468c-430e-bc27-8a2ca7d89697

    Continue reading...

Compartilhe esta Página