[Python] Why am I getting a FileNotFoundError when trying to create a SparkContext?

I am currently on Python: 3.7.9, spark: spark-3.1.2-bin-hadoop3.2

In this project venv, I have the settings as:

kafka-python 2.0.2
pip 21.2.4
py4j 0.10.9
pyspark 2.4.6
setuptools 57.4.0


System environment variables:

SPARK_HOME C:\spark-3.1.2-bin-hadoop3.2
PYTHONPATH %SPARK_HOME%\python;%SPARK_HOME%\python\lib\py4j-0.10.9-src.zip;%PYTHONPATH%
PATH %SPARK_HOME%\bin;%SPARK_HOME%\python;%PATH%


I am trying to execute following code in Python:

import sys
from pyspark import SparkContext, SparkConf
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils

if __name__ == "__main__":
sc = SparkContext(appName="PythonStreamingReceiverKafkaWordCount")


I get this in Console:

C:\Users\MANU\PycharmProjects\pythonProject1\venv\Scripts\python.exe C:\Users\MANU\PycharmProjects\pythonProject1\kafka_consumer.py

Traceback (most recent call last):
File "C:\Users\MANU\PycharmProjects\pythonProject1\kafka_consumer.py", line 7, in <module>
sc = SparkContext(appName="PythonStreamingReceiverKafkaWordCount")
File "C:\Users\MANU\PycharmProjects\pythonProject1\venv\lib\site-packages\pyspark\context.py", line 133, in __init__
SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
File "C:\Users\MANU\PycharmProjects\pythonProject1\venv\lib\site-packages\pyspark\context.py", line 316, in _ensure_initialized
SparkContext._gateway = gateway or launch_gateway(conf)
File "C:\Users\MANU\PycharmProjects\pythonProject1\venv\lib\site-packages\pyspark\java_gateway.py", line 46, in launch_gateway
return _launch_gateway(conf)
File "C:\Users\MANU\PycharmProjects\pythonProject1\venv\lib\site-packages\pyspark\java_gateway.py", line 101, in _launch_gateway
proc = Popen(command, stdin=PIPE, env=env)
File "C:\Users\MANU\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 800, in __init__
restore_signals, start_new_session)
File "C:\Users\MANU\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 1207, in _execute_child
startupinfo)
**FileNotFoundError: [WinError 2] The system cannot find the file specified**

Process finished with exit code 1


In case it's required: my dir structure is: C:\spark-3.1.2-bin-hadoop3.2\spark\python...

Continue reading...