In databricks, I created a spark dataframe, and need to convert it to a pandas dataframe, sdf = spark.sql('select * from my_tbl') pdf = sdf.toPandas() but got error: ArrowInvalid: Casting from timestamp[us, tz=America/New_York] to timestamp[ns] would result in out of bounds timestamp: 253402214400000000 File <command-1629564213930490>, line 1 ----> 1 pdf=sdf.toPandas() I am just wondering how to fix this, or is there anyway to generate a panda dataframe by directly invoking SQL in databricks? Continue reading...
