[Python] Pyspark, Hadoop, and S3: java.lang.NoSuchMethodError:...

I've been facing compatibility issues related to getting delta-spark to work straight out of the box with S3 and wanted to get some advice. I've tried dozens of combinations of versions between Spark, delta-spark, hadoop-aws, and aws-java-sdk-bundle, but I keep getting some variation of a NoSuchMethodError which strongly indicates a compatibility error.

I've tried using the most recent versions of Spark (3.5.3) and delta-spark (3.2.1) but now I've been trying the explicit example listed on their website.

According to: https://docs.delta.io/latest/delta-storage.html#-requirements-s3-single-cluster They say:

Use the following command to launch a Spark shell with Delta Lake and S3 support (assuming you use Spark 3.5.0 which is pre-built for Hadoop 3.3.4):
​

and uses --packages io.delta:delta-spark_2.12:3.1.0,org.apache.hadoop:hadoop-aws:3.3.4

And according to their compatibility matrix (https://docs.delta.io/latest/releases.html), 3.1.0 works with 3.5.0. Therefore, I re-installed the packages with pyspark 3.5.0 and delta-spark 3.1.0.

My code is as follows:

builder = pyspark.sql.SparkSession.builder.appName(app_name) \
.config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
.config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
.config("spark.hadoop.fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider") \
.config("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem") \
.config("spark.hadoop.fs.s3a.access.key", aws_access_key_id) \
.config("spark.hadoop.fs.s3a.secret.key", secret_access_key) \
.config("spark.hadoop.fs.s3a.endpoint", f"s3.{region_name}.amazonaws.com") \
.config("spark.hadoop.delta.enableFastS3AListFrom", True)

extra_packages = ["org.apache.hadoop:hadoop-aws:3.3.4,com.amazonaws:aws-java-sdk-bundle:1.12.262"]
spark = delta.configure_spark_with_delta_pip(
builder,
extra_packages=extra_packages,
).getOrCreate()
data = {
'id': [1, 2, 3],
'name': ['Alice', 'Bob', 'Charlie'],
}
df = spark.createDataFrame(pd.DataFrame(data))
deltalake_table_spark = f"s3a://{bucket}/test_folder/spark_deltalake_table"
df.write.format("delta").save(deltalake_table_spark)


Where a chunk of the error output is:

Traceback (most recent call last):
File "...", line 183, in test_delta_spark
df.write.format("delta").save(deltalake_table_spark)
File ".../lib/python3.10/site-packages/pyspark/sql/readwriter.py", line 1463, in save
self._jwrite.save(path)
File ".../lib/python3.10/site-packages/py4j/java_gateway.py", line 1322, in __call__
return_value = get_return_value(
File ".../lib/python3.10/site-packages/pyspark/errors/exceptions/captured.py", line 179, in deco
return f(*a, **kw)
File ".../lib/python3.10/site-packages/py4j/protocol.py", line 326, in get_return_value
raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o69.save.
: com.google.common.util.concurrent.ExecutionError: java.lang.NoSuchMethodError: 'org.apache.hadoop.fs.s3a.Listing$FileStatusListingIterator org.apache.hadoop.fs.s3a.Listing.createFileStatusListingIterator(org.apache.hadoop.fs.Path, org.apache.hadoop.fs.s3a.S3ListRequest, org.apache.hadoop.fs.PathFilter, org.apache.hadoop.fs.s3a.Listing$FileStatusAcceptor)'
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2261)


And to verify,

spark._sc._jvm.org.apache.hadoop.util.VersionInfo.getVersion()
Out[18]: '3.3.4'
spark.version
Out[19]: '3.5.0'


As for my java version:

java --version
openjdk 11.0.24 2024-07-16
OpenJDK Runtime Environment Homebrew (build 11.0.24+0)


Checking the output of the error logs, one can see the correct versions are installed:

:: modules in use:
com.amazonaws#aws-java-sdk-bundle;1.12.262 from central in [default]
io.delta#delta-spark_2.12;3.1.0 from central in [default]
io.delta#delta-storage;3.1.0 from central in [default]
org.antlr#antlr4-runtime;4.9.3 from central in [default]
org.apache.hadoop#hadoop-aws;3.3.4 from central in [default]
org.wildfly.openssl#wildfly-openssl;1.0.7.Final from central in [default]


Everything seems to be in sync. 3.3.4 hadoop and hadoop-aws, 3.1.0 delta-spark, 3.5.0 pyspark, and I've tried leaving the aws-java-sdk-bundle out or changing according to the version compatibility.

Other things that I've tried:

1. Instead of defining the hadoop configuration within the builder, I tried doing it after the getOrCreate with spark._jsc.hadoopConfiguration().set(...)
2. Instead of configure_spark_with_delta_pip, I manually fed in the delta-spark jar package in the config with .config("spark.jars.packages", "io.delta:delta-spark_2.12:3.1.0,org.apache.hadoop:hadoop-aws:3.3.4,com.amazonaws:aws-java-sdk-bundle:1.12.262") in the builder definition.

I'm out of options and have searched many other threads but I've had no success with getting this to run. Can someone confirm whether this truly is a compatibility issue or if there's something else I'm missing? Thank you.

Continue reading...