[Python] Pyspark Py4JJavaError while creating the delta-table

Here is the pyspark code which is running on jupyter notebook.

import pyspark
from delta import *
builder = pyspark.sql.SparkSession.builder.appName("MyApp") \
.config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \

.config("spark.sql.catalog.spark_catalog",
"org.apache.spark.sql.delta.catalog.DeltaCatalog")
spark = configure_spark_with_delta_pip(builder).getOrCreate()

Error :


Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext. : java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x30cb5b99) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x30cb5b99

configuration:

• delta-spark=1.1.0
• pyspark=3.2.0
• Java version: openjdk 17.0.1 2021-10-19 OpenJDK Runtime Environment Homebrew (build 17.0.1+1) OpenJDK 64-Bit Server VM Homebrew (build 17.0.1+1, mixed mode, sharing)

.bash_profile :


export HADOOP_HOME=/opt/hadoop-2.8.0
export SPARK_HOME=/opt/spark-3.2.0-bin-hadoop3.2
export PATH=$SPARK_HOME/bin:$PATH
export PYSPARK_PYTHON=python3
export PYSPARK_DRIVER_PYTHON="jupyter"
export PYSPARK_DRIVER_PYTHON_OPTS="notebook"


please help me to resolve the error. Thanks in advance .

Continue reading...