[Python] determine which columns are equal after pandas merge betwen two dataframes

I performed a merge with pandas using the suffix option like below:

df3 = df1.merge(df2, how='inner',on='key',suffixes=('_first', '_second'))


I now need to:

1. pairwise check i.e. (x_first == x_second ) for the approx 60 pairs of columns
2. If the columns are equal rename x_first to just x and drop x_second
3. If they are not equal keep both columns

How can this be done for a moderately large pandas dataframe (~6M rows by 200 columns )?

Continue reading...