I have a Spark data frame (df1) with a particular schema, and I have another dataframe with the same columns, but different schema. I know how to do it column by column, but since I have a large set of columns, it would be quite lengthy. To keep the schema consistent across dataframes, I was wondering if I could be able to apply one schema to another data frame or creating a function that do the job. Here is an example: df1 # root # |-- A: date (nullable = true) # |-- B: integer (nullable = true) # |-- C: string (nullable = true) df2 # root # |-- A: string (nullable = true) # |-- B: string (nullable = true) # |-- C: string (nullable = true)` I want to copy apply the schema of df1 to df2. I tried this approach for one column. Given that I have a large number of columns, it would be quite a lengthy way to do it. df2 = df2.withColumn("B", df2["B"].cast('int')) Continue reading...
