[Python] Spark SQL is interpreting a datetime.date object as a mathematical formula or integer...

I've encountered a problem in Spark SQL. It is interpreting a datetime.date object as a mathematical formula, or integer, in a SQL statement I am writing.

currentDateAndTime = datetime.now()
current_month = currentDateAndTime.strftime("%m")
current_year = currentDateAndTime.strftime("%Y")

first_day_of_month = date(int(current_year), int(current_month), 1)
print(first_day_of_month)
type(first_day_of_month)


and you get:

2022-10-01 datetime.date

Then when I do

df = spark.sql("""
SELECT * FROM table_A
WHERE IncidentCreatedDate < {}
""".format(first_day_of_month))


I get an error that says

AnalysisException: cannot resolve '(table_A.`IncidentCreatedDate` < ((2022 - 10) - 1))' due to data type mismatch: differing types in '(tableA.`IncidentCreatedDate` < ((2022 - 10) - 1))' (date and int).; ...


There might be a typo in everything above because I had to type everything out on another laptop since the other one is my work laptop and they don't like me sending anything from that laptop to anywhere else.)

Continue reading...