[Python] Pyspark find columns with mismatched data

I have a Pyspark dataframe df1. It has columns like col1, col2, col3, col4, col5.

+----+----+----+----+----+
|col1|col2|col3|col4|col5|
+----+----+----+----+----+
|A |A |X |Y |Y |
|B |C |Y |Y |Z |
|D |D |Z |X |Z |
|E |E |M |M |A |
+----+----+----+----+----+


Have created a list of tuple storing column pairs to compare

t1 = [('col1', 'col2'), ('col3', 'col5')]


Now I want to create new column mismatched_data which should maintain the col name and its original and new value based on the tuple -

+----+----+----+----+----+------------------------------------------------------+
|col1|col2|col3|col4|col5|mismatched_data |
+----+----+----+----+----+------------------------------------------------------+
|A |A |X |Y |Y |[{"col3": {"original_value": "X", "new_value": "Y"}}] |
|B |C |Y |Y |Z |[{"col1": {"original_value": "B", "new_value": "C"}}, {"col3":
{"original_value": "Y", "new_value": "Z"}}] |
|D |D |Z |X |Z |NULL |
|E |E |M |M |A |[{"col3": {"original_value": "M", "new_value": "A"}}] |
+----+----+----+----+----+------------------------------------------------------+


Please let me know how I can achieve this in an efficient manner.

Continue reading...