[Python] Why am I getting [DATA_SOURCE_NOT_FOUND] for Iceberg on Azure Databricks despite...

Trying to get azure databricks running with iceberg. I understand that in shared computes, we need admin privilege to "whitelist" the iceberg jar files.

So running this in personal computes with a runtime of

• 16.3 (includes Apache Spark 3.5.2, Scala 2.12)

jar file installed in the library:

• iceberg-spark-runtime-3.5_2.13-1.9.0.jar. Also tried through Maven coordinates.

Additionally, in the Spark inits:


spark.sql.catalog.spark_catalog.warehouse = abfss://container@storageaccount.dfs.core.windows.net/
spark.sql.catalog.spark_catalog = org.apache.iceberg.spark.SparkCatalog
spark.master local[*, 4]
spark.sql.catalog.spark_catalog.type = hadoop
spark.databricks.cluster.profile singleNode


Though, regardless of using DBFS as the warehouse path or ABFSS, when I run:

df = spark.createDataFrame([(1, "Alice"), (2, "Bob")], ["id", "name"])

df.writeTo("catalogname.schema.tablename") \
.using("iceberg") \
.createOrReplace()


[DATA_SOURCE_NOT_FOUND] Failed to find the data source: iceberg. Make sure the provider name is correct and the package is properly registered and compatible with your Spark version. SQLSTATE: 42K02

Seems like my runtime is matching with the jar file I installed. And not even using Unity Catalog, not sure why it isn't running. Any help would be great!

Continue reading...