[Python] Combining Multiple PySpark DataFrames Dynamically: Incorporating Static Columns with...

I possess multiple PySpark DataFrames that need to be concatenated or unionized to produce a final DataFrame with the following structure: Input:

df1 :[colA, colB, colC, avg_salary_y2020]
df2 :[colA, colB, colC, avg_salary_y2021]
df3 :[colA, colB, colC, avg_salary_y2022]


Output:

df_final: [colA, colB, colC, avg_salary_y2020, avg_salary_y2021, avg_salary_y2022]


Is there an elegant solution to accomplish this task?

Continue reading...