[Python] Databricks delta live tables metadata comments for created columns file_path and...

I have been getting my schema passed in with

custom_schema = create_StructType_schema (access_key,secret_access_key,schema_bucket_name,schema_folder,schema_file_name)
metadata_fp = {"comment": "Path of the file from metadata (_metadata.file_path)"}
metadata_lmd = {"comment": "Ingestion timestamp for the current record"}

# Append fields with metadata to schema
custom_schema.add(StructField("file_path", StringType(), True, metadata_fp))
custom_schema.add(StructField("last_modified_date", TimestampType(), True, metadata_lmd))


This works well for all of the columns that are not file_path and last_modifed_date. But when I try to add custom_schema.add either of the rows I get a duplicate column warning for file_path. However, dropping either of the rows and uisng withColumn loses the comments metadata. I wanted to know if there was any way to get F.current_timestamp() and _metadata.file_path with comments on their columns for a delta live table in Databricks

dfRaw=spark.readStream.format("cloudFiles").option("cloudFiles.format",file_format).option("recursiveFileLookup","true").option("cloudFiles.allowOverwrites", True).option("delimiter",file_delimiter).option("multiline","true").option("header",file_header).schema(custom_schema).load(location).select("*", "_metadata.file_path").withColumn("last_modified_date",F.current_timestamp())```




I have tried to add structfields to the schema and dropping the columns and readding them but nothing keeps the metadata comments.

Continue reading...