I have a setup where I use pipenv for my virtual environments. I am using a MacBook. I have done the following: Installed openJDK 11 using brew Set paths in the .zprofile folder: eval "$(/opt/homebrew/bin/brew shellenv)" export JAVA_HOME="/opt/homebrew/opt/openjdk@11" export PYSPARK_PYTHON="/opt/homebrew/opt/python@3.10/bin/python3.10" export PATH="/opt/homebrew/opt/python@3.10/bin:$PATH" Install pipenv using pip. Install all dependencies using pipenv install -e ., here pyspark version 3.3.0 is included. Activate the shell pipenv shell I use VS code, when I run all tests they pass as expected. However, when I then run pyspark in the terminal in the activated shell environment, I get an error: /Users/<user_name>/.local/share/virtualenvs/<repo_name>/bin/pyspark: line 24: /bin/load-spark-env.sh: No such file or directory /Users/<user_name>/.local/share/virtualenvs/<repo_name>/bin/pyspark: line 68: /bin/spark-submit: No such file or directory /Users/<user_name>/.local/share/virtualenvs/<repo_name>/bin/pyspark: line 68: exec: /bin/spark-submit: cannot execute: No such file or directory When I follow the path /Users/<user_name>/.local/share/virtualenvs/<repo_name>/bin/pyspark there is no folder pyspark but in the path /Users/<user_name>/.local/share/virtualenvs/<repo_name>/bin/ the files (load-spark-env.sh and spark-submit) which causes errors are located. What am I doing wrong here? Continue reading...
