[Python] How do I set debugging info in function in databricks that is used in applyInPandas

I'm using databricks version 13.3. I have a function which I'm calling by using the applyInPandas function. I need to see the attributes of my pdf dataset which I'm using inside my function. My sample code looks like

import pandas as pd
import os

def my_function(pdf: pd.DataFrame) -> pd.DataFrame:
train = pdf
debug_path = "/dbfs/tmp/debug_log.txt"

# Generate debug string
debug_string = f"Group with {len(pdf)} rows, columns: {list(pdf.columns)}\n"

# Append to log file
with open(debug_path, "a") as f:
f.write(debug_string)

return pdf

result_df = pdf.groupBy("some_column").applyInPandas(my_function, schema=df.schema)

with open("/dbfs/tmp/debug_log.txt", "r") as f:
print(f.read())


But my code is not showing any output. How can I resolve this issue?

Continue reading...