[Python] Issues Connecting to Infor Data Lake Using Spark JDBC in Azure Synapse Spark Notebook

I'm encountering difficulties while attempting to connect to Infor Data Lake using Spark JDBC within Azure Synapse. Despite configuring the necessary JDBC driver and ionapi file path, I consistently encounter the following error:

Py4JJavaError: An error occurred while calling o3997.jdbc.

: java.lang.RuntimeException: JDBC Error 100

Unable to connect. The Infor Compass JDBC Driver.ionapi file is not found. The file must be located in the folder containing the Compass JDBC driver JAR file.
​

Setup details:

I've initialized my Spark session in Azure Synapse with the required JDBC driver and specified the ionapi file path using spark.jars.

The ionapi file is stored in Azure Data Lake Storage (ADLS), and I have ensured the path is correctly specified in the JDBC connection properties.

Code snippet:

%%configure -f

{

"conf": {

"spark.jars": "abfss://container@datalake_name.dfs.core.windows.net/infor-compass-jdbc-2023.10.jar",

}

}

storage_account_name = "**"

container_name = "**"

jdbc_jar_path = f"abfss://{container_name}@{storage_account_name}.dfs.core.windows.net/path_to_jdbc_driver/infor-compass-jdbc-2023.10.jar"

ionapi_file_path = f"abfss://{container_name}@{storage_account_name}.dfs.core.windows.net/path_to_ionapi_file/Infor Compass JDBC Driver.ionapi"

spark = SparkSession.builder \

.appName("InforDataLakeConnection") \

.config("spark.jars", jdbc_jar_path) \

.getOrCreate()

jdbc_url = "jdbc:infordatalake://your_infordatalake_endpoint"

jdbc_properties = {

"user": "**",

"password": "**",

"driver": "com.infor.idl.jdbc.Driver",

"ionapi": ionapi_file_path
}

table_name = "table_name"

df = spark.read.jdbc(url=jdbc_url, table=table_name, properties=jdbc_properties)

df.show()


Error analysis: I have verified that ionapi_file_path correctly points to the location of the ionapi file in ADLS. The storage account key used in spark.conf.set matches the correct ADLS storage account.

How to correctly configure the ionapi file path or any additional settings necessary to successfully connect to Infor Data Lake using Spark JDBC in Azure Synapse?

Trying to connect infor datalake using pyspark notebook in Synapse analytics. Encountering issues, as code is unable to read ionapi file for credentials.

Continue reading...