[Python] ModuleNotFoundError: No module named 'select' while using PySpark

I am testing SparkSession using following example code:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("TestApp").getOrCreate()
data = [("Alice", 29), ("Bob", 35), ("Cathy", 45)]
df = spark.createDataFrame(data, ["Name", "Age"])
df.show()


but the code fails on the line df.show() with a following traceback:

Traceback (most recent call last):
File "<frozen runpy>", line 189, in _run_module_as_main
File "<frozen runpy>", line 112, in _get_module_details
File "C:\spark\spark-3.5.3-bin-hadoop3\python\lib\pyspark.zip\pyspark\__init__.py", line 58, in <module>
File "C:\spark\spark-3.5.3-bin-hadoop3\python\lib\pyspark.zip\pyspark\conf.py", line 23, in <module>
File "C:\spark\spark-3.5.3-bin-hadoop3\python\lib\py4j-0.10.9.7-src.zip\py4j\java_gateway.py", line 19, in <module>
ModuleNotFoundError: No module named 'select'
24/12/20 12:01:08 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
org.apache.spark.SparkException: Python worker failed to connect back.


I have Python 3.11.1, Java (openjdk 11.0.16.1) and Spark version 3.5.3 installed. Environment variables HADOOP_HOME, SPARK_HOME, PYSPARK_HOME, JAVA_HOME, PYSPARK_PYTHON, PYSPARK_DRIVER_PYTHON are configured. Also, I installed pyspark to project's venv using pip install pyspark.

Can anyone please suggest how to fix the error or tell why does it happen? Thanks in advance for any ideas.

Continue reading...