[Python] Error connecting to Cosmos DB from Databricks: AnalysisException: Catalog...

I've followed the documentation from start to end.

I'm trying to connect to a CosmosDb so I can write data to it.

My Databricks cluster runtime version is: 11.3 LTS

I have installed the cosmos DB Spark connector: com.azure.cosmos.spark:azure-cosmos-spark_3-3_2-12:4.15.0 per the documentation.

I have the following code:

cosmosEndpoint = "MyEndpoint"
cosmosMasterKey = "MyMasterKey"
cosmosDatabaseName = "SampleDB"
cosmosContainerName = "testContainer"

#Configure Catalog Api to be used

spark.conf.set("spark.sql.catalog.cosmosCatalog", "com.azure.cosmos.spark.CosmosCatalog") spark.conf.set("spark.sql.catalog.cosmosCatalog.spark.cosmos.accountEndpoint", cosmosEndpoint) spark.conf.set("spark.sql.catalog.cosmosCatalog.spark.cosmos.accountKey", cosmosMasterKey)

#create an Azure Cosmos DB database using catalog api

spark.sql("CREATE DATABASE IF NOT EXISTS cosmosCatalog.{};".format(cosmosDatabaseName))

#create an Azure Cosmos DB container using catalog api

spark.sql("CREATE TABLE IF NOT EXISTS cosmosCatalog.{}.{} using cosmos.oltp TBLPROPERTIES(partitionKeyPath = '/id', manualThroughput = '1100')".format(cosmosDatabaseName, cosmosContainerName)) `


I get the error AnalysisException: Catalog 'cosmoscatalog' not found. I have followed the documentation from start to end. Does anyone know why this error occurs?

Continue reading...