[Python] failed spark script : Failed to find data source: kafka. Please deploy the application

I have a pyspark script that has the following setup

KAFKA_BOOTSTRAP = os.getenv("KAFKA_BOOTSTRAP", "localhost:9092")


BASE = "/home/ouyassine/spark-jars" # no spaces!

jar_list=",".join([
f"{BASE}/spark-sql-kafka-0-10_2.12-4.0.0.jar",
f"{BASE}/spark-token-provider-kafka-0-10_2.12-4.0.0.jar",
f"{BASE}/kafka-clients-3.7.0.jar",
f"{BASE}/commons-pool2-2.12.0.jar",
f"{BASE}/hadoop-aws-3.3.6.jar",
f"{BASE}/aws-java-sdk-bundle-1.12.367.jar",
f"{BASE}/lz4-java-1.8.0.jar",
f"{BASE}/snappy-java-1.1.10.5.jar",
])

spark = (
SparkSession.builder
.appName("clicks-to-s3")
# put jars everywhere
.config("spark.jars", jar_list)
.config("spark.driver.extraClassPath", jar_list.replace(",", ":"))
.config("spark.executor.extraClassPath", jar_list.replace(",", ":"))
.config("spark.sql.shuffle.partitions", "4")
.config("spark.hadoop.fs.s3a.aws.credentials.provider",
"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider")
.config("spark.hadoop.fs.s3a.endpoint", f"s3.{AWS_REGION}.amazonaws.com")
.getOrCreate()
)


in the same script i want to access kafka

raw = (spark.readStream.format("kafka")
.option("kafka.bootstrap.servers", KAFKA_BOOTSTRAP)
.option("subscribe", "clicks")
.option("startingOffsets","earliest")
.load())


when i run the script using command python or spark-submit i get the following error:

Traceback (most recent call last):
File "/home/ouyassine/Documents/projects/Retail Clickstream Mini-Lake/src/stream_to_s3.py", line 75, in <module>
.load())
^^^^^^
File "/home/ouyassine/Documents/projects/Retail Clickstream Mini-Lake/venv/lib/python3.12/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/streaming/readwriter.py", line 307, in load
File "/home/ouyassine/Documents/projects/Retail Clickstream Mini-Lake/venv/lib/python3.12/site-packages/pyspark/python/lib/py4j-0.10.9.9-src.zip/py4j/java_gateway.py", line 1362, in __call__
File "/home/ouyassine/Documents/projects/Retail Clickstream Mini-Lake/venv/lib/python3.12/site-packages/pyspark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 288, in deco
pyspark.errors.exceptions.captured.AnalysisException: Failed to find data source: kafka. Please deploy the application as per the deployment section of Structured Streaming + Kafka Integration Guide.


i have made sure that th jars are loaded and that the kafka container is running. I still can't figure out the problem.

Continue reading...