[Python] PySpark SQL Query Fails: Py4JError Method sql(...) Does Not Exist

I'm having an issue running SQL queries in PySpark. The Spark session starts successfully, and I can create and show a DataFrame, but executing spark.sql("SELECT 1").show() throws a py4j.protocol.Py4JError: py4j.Py4JException: Method sql([class java.lang.String, class java.util.HashMap]) does not exist

My environment:

• VS code
• Python 3.10.11
• PySpark 3.5.3
• Java 17.0.10 (LTS)
• Py4J 0.10.9.7

What could be causing this issue? Any insights would be appreciated.

My complete code:

from pyspark.sql import SparkSession

# Initialize Spark session
spark = SparkSession.builder \
.master("local") \
.appName("test code") \
.getOrCreate()

# test df creation
df = spark.range(1, 3)

# test sql query
spark.sql("SELECT 1").show()


Output:

Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
25/02/15 10:29:28 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
25/02/15 10:29:39 WARN SizeEstimator: Failed to check whether UseCompressedOops is set; assuming yes
+---+
| id|
+---+
| 1|
| 2|
+---+

Traceback (most recent call last):
File "C:\Users\test\Semantic Analysis.py", line 23, in <module>
spark.sql("SELECT 1").show()
File "C:\Users\test\venv\lib\site-packages\pyspark\sql\session.py", line 1440, in sql
return DataFrame(self._jsparkSession.sql(sqlQuery, litArgs), self)
File "C:\Users\test\venv\lib\site-packages\py4j\java_gateway.py", line 1322, in __call__
return_value = get_return_value(
File "C:\Users\test\venv\lib\site-packages\pyspark\errors\exceptions\captured.py", line 169, in deco
return f(*a, **kw)
File "C:\Users\test\venv\lib\site-packages\py4j\protocol.py", line 330, in get_return_value
raise Py4JError(
py4j.protocol.Py4JError: An error occurred while calling o25.sql. Trace:
py4j.Py4JException: Method sql([class java.lang.String, class java.util.HashMap]) does not exist
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
at py4j.Gateway.invoke(Gateway.java:274)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.lang.Thread.run(Unknown Source)

Continue reading...