[Python] Dependency error when using cloud storage connector hadoop3

I'm trying to set up a simple pyspark project which writes a DataFrame to a cloud storage bucket but I keep getting errors related to incorrect dependency management. I have tried multiple versions of the cloud storage connector with no success. I've also tried adding additional configuration options I've seen from documentation online and again, no success, just different errors. I do have a different fix but I don't understand why it works, any help would be appreciated.

Code:

from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType, IntegerType

def main():
# Initialize Spark session with GCS support
spark = SparkSession.builder \
.appName("Basic PySpark Example") \
.config("spark.jars.packages", "com.google.cloud.bigdataoss:gcs-connector:hadoop3-2.2.26") \
.config("spark.hadoop.google.cloud.auth.service.account.enable", "true") \
.config("spark.hadoop.fs.gs.auth.service.account.json.keyfile", "<key-file>.json") \
.getOrCreate()

schema = StructType([
StructField("name", StringType(), True),
StructField("age", IntegerType(), True),
StructField("city", StringType(), True)
])

data = [
("John", 30, "New York"),
("Alice", 25, "London"),
("Bob", 35, "Paris")
]
df = spark.createDataFrame(data, schema=schema)
df.show()
gcs_bucket = "name"

df.write.parquet(f"gs://{gcs_bucket}/data/people.parquet")
df.write.csv(f"gs://{gcs_bucket}/data/people.csv")

print(f"Data written to GCS bucket: {gcs_bucket}")
spark.stop()

if __name__ == "__main__":
main()


requirements.txt:

pyspark==3.5.5
google-cloud-storage==3.1.0


Error:

py4j.protocol.Py4JJavaError: An error occurred while calling o49.parquet.
: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "gs"
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3443)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
at org.apache.spark.sql.execution.datasources.DataSource.planForWritingFileFormat(DataSource.scala:454)
at org.apache.spark.sql.execution.datasources.DataSource.planForWriting(DataSource.scala:530)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:391)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:364)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:243)
at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:802)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:75)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:52)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.base/java.lang.Thread.run(Thread.java:1583)


Fix: Substitute

.config("spark.jars.packages", "com.google.cloud.bigdataoss:gcs-connector:hadoop3-2.2.26")


with

.config("spark.jars", "https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-hadoop3-latest.jar")

Continue reading...