[Python] Fabric Notebook Unauthorized error when reading parquet file

I am relatively new to writing python and using fabric in our company. In doing so I have created a fairly simple script that writes to a parquet file and then I want to take that parquet file and write it to a delta table. However, that is where I am running into trouble.

below is my script. This writes the parquet file just fine. It errors out when I try to take it to the delta table.

My initial try was to do everything together. But I am getting errors when this gets to writing the parquet to delta table. I have tried several different configurations but none seem to work. Even one that works in another script doesn't work here.

import subprocess
import json
import requests
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
from pyspark.sql import SparkSession

access_token = "TOKEN"

# Define your API endpoint and headers
url = "https://api.monday.com/v2"
headers = {
"Authorization": access_token,
"Content-Type": "application/json"
}

# Define your GraphQL query
query = """
query {
users {
created_at
email
account {
name
id
}
}
}
"""

# Send the request
response = requests.post(url, json={'query': query}, headers=headers)

# Check for errors
if response.status_code == 200:
data = response.json()
# print(data)
users = data['data']['users']

# Convert the data to a DataFrame
df = pd.json_normalize(users)

# Write the DataFrame to a Parquet file
# df.to_parquet('users_data.parquet', engine='pyarrow')
# print("Data written to users_data.parquet")
parquet_table_name = "MondayUsers.parquet"
parquet_file_path = "abfss://Files/Monday/"

df.to_parquet(parquet_file_path + parquet_table_name, engine='pyarrow', index=False)
print("Data written to Parquet file")

# Initialize Spark session with Delta Lake configurations
spark = SparkSession.builder \
.appName("FabricNotebook") \
.config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
.config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
.getOrCreate()

# Define the path to your Parquet file
parquet_file_path = parquet_file_path + parquet_table_name

# Read the Parquet file into a Spark DataFrame
try:
#load parquet file to data lake table
from pyspark.sql import SparkSession

# Initialize Spark session
spark = SparkSession.builder.appName("FabricNotebook").getOrCreate()

# Define the path to your Parquet file


# Read the Parquet file into a Spark DataFrame
df = spark.read.parquet(parquet_file_path)

# Define the Delta table name
delta_table_name = "monday.Users"

# Write the Spark DataFrame to the Delta table
df.write.mode("overwrite").option("overwriteSchema", "true").format("delta").saveAsTable(delta_table_name)



except Exception as e:
print(f"Error reading Parquet file or writing to Delta table: {e}")




else:
print(f"Query failed with status code {response.status_code}")





The error this receives is:
Error reading Parquet file or writing to Delta table: An error occurred while calling o6842.toString. Trace: java.lang.IllegalArgumentException: object is not an instance of declaring class at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.base/java.lang.Thread.run(Thread.java:829)

I then thought I'd take it and put it into another section and just read from the parquet and write to a delta table. But then I'm getting "unauthorized" with this

import pandas as pd

parquet_table_name = "MondayUsers.parquet"
parquet_file_path = "abfss://Files/Monday/"

parquet_file = parquet_file_path + parquet_table_name

parquet_df = pd.read_parquet(parquet_file)

spark_df = spark.createDataFrame(parquet_df)

print(spark_df)

# Define the Delta table name
delta_table_name = "monday.Users"

spark_df.write.format("delta").mode("overwrite").saveAsTable(delta_table_name)

Continue reading...