[Python] Error while executing SQL query in Azure App Service - Time data mismatch

I am working with a FastAPI application that connects to Azure Databricks to execute SQL queries using the databricks-sdk. The relevant code snippet for establishing the connection and executing the query is as follows:

def get_databricks_connection():
# Connection logic...
# Returns a Databricks connection object

def exec_query_to_df(connection, query):
# Executes the SQL query and returns the result as a pandas DataFrame
if hasattr(connection, "cursor"):
with connection.cursor() as cursor:
try:
logger.info(f"Executing query: {query}")
cursor.execute(query)
logger.info("Query execution completed successfully")

columns = [column[0] for column in cursor.description]
rows = cursor.fetchall()
result = [tuple(int(x.item()) if isinstance(x, np.int64) else x for x in row) for row in rows]
df = pd.DataFrame(result, columns=columns)
df.replace({np.nan: None}, inplace=True)
return df
except Exception as e:
logger.error(f"Error executing query '{query}': {e}")
raise
# Additional logic for SQLAlchemy connection...

def get_all_voyage_pnl(user_initials):
# Logic to build and execute the query to get voyage PNL data


The issue arises only when I deploy the application to Azure App Service. The application runs correctly without any problems on my local machine. However, when deployed, I encounter an error during query execution. The error logs indicate the following:

ERROR: Error executing query 'SELECT * FROM your_database.your_table WHERE condition': time data '2025-02-15 14:00:00' does not match format '%Y-%m-%d %H:%M:%S.%f'


This error occurs during the execution of the SQL query, and it prevents any rows from being returned. Consequently, the DataFrame processing never takes place.

The query I am attempting to run is designed to fetch voyage PNL data, including the CommenceGmt and CompleteGmt columns. The original data is in the format 2025-02-15 14:00:00, but in Databricks, this is converted to a timestamp in the format:

2025-02-15T14:00:00.000+00:00


However, the error suggests a mismatch with the expected format when executed in the Azure environment, leading to failures without any data being returned.

Additional Information Here are the details about my environment:

Python Version: 3.11

Relevant Libraries:

from databricks.sdk import WorkspaceClient
from databricks import sql
import pandas as pd
import numpy as np


Relevant Requirements.txt:

databricks-sdk>=0.44.1
databricks-sql-connector>=4.0.0
pandas==2.1.3
numpy>=1.26.2


I've confirmed that the connection works correctly on my local machine. Could the issue be related to how Azure App Service handles timezone or timestamp formats?

If there’s any information I might be missing that could help diagnose this issue, I would appreciate your insights!

Questions:

How can I resolve the time data format issue that occurs during execution? Are there any configurations needed in Azure App Service to handle timezones or data formats appropriately?

Continue reading...