[Python] Convert dataframe column if its datetime to longType, otherwise keep the same column

I have a dataframe with columns entry_transaction, created_date, updated_date, transaction_date.

Created_date, Updated_date are strings with YYYY-MM-dd HH:mm:ss.SSS, and transaction_day is in long milliseconds. I need to loop through the columns and if a column is in datetime format, convert to long, else keep the default value, and also apply datatype from a schema file.

Following is the code, while am able to convert the datetime to long format, am unable to get the default value for transaction_date.

There are 50 plus tables, and each table has more than 50 columns. so I need to use for loop and apply this change dynamically by checking if the column is a datetime column.

Here is the sample code for reproducing:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("example").getOrCreate()

data = [("2025-03-12 18:47:33.943", "1735862400000", "2025-03-12 18:47:33.943", "2025-03-12 18:47:33.943"), ("2025-03-12 10:47:33.943", "1735862400000", "2025-03-12 12:47:33.943", "2025-03-12 16:47:33.943"), ("2025-03-01 18:47:33.943", "1735862400000", "2025-03-04 18:47:33.943", "2025-03-12 18:47:33.943")]
columns = ["entry_transaction", "transaction_date", "creation_date", "updated_date"]

df = spark.createDataFrame(data, columns)
df.show()
df2 = df.select("entry_transaction","transaction_date", "creation_date", "updated_date")

schema = StructType([StructField('entry_transaction', LongType(), True), StructField('transaction_date', StringType(), True), StructField('creation_date', LongType(), True), StructField('updated_date', LongType(), True)])

for field in schema.fields:
print(field.name, field.dataType)
df2 = df2.withColumn(
field.name,
unix_timestamp(col(field.name).cast(TimestampType())) * 1000000 if to_date(col(field.name), "yyyy-MM-dd").isNotNull else df2[field.name])
sorted_columns = sorted(df2.columns)
df_reordered = df2.select(sorted_columns)
display(df_reordered)


Expected output:

But am getting NULL for transaction_date

Please help how to get the proper dataframe. Thanks,

Continue reading...