[Python] How to compare every 2 rows(rows 1 and 2, rows 3 and 4, etc..) against eachother and...

I am working on a project that requires me to compare 2 rows (1 and 2, 3 and 4, etc...) and output the differences to a table. Now I have been able to compare the columns and create the table with differences. What I need to clean it up now is a way to show which row identifiers are being compared. Each set of rows has the same identifier. Please see the code and output below.

Source data example:






import pandas as pd



def compare_rows(df):

"""

Compares each pair of consecutive rows in a Pandas DataFrame and outputs the differences.



Args:

df: The input Pandas DataFrame.



Returns:

A new Pandas DataFrame containing the differences between consecutive rows.

"""

diff_list = []

for i in range(0,len(df) - 1,2):





row1 = df.iloc

row2 = df.iloc[i + 1]

print(i)

diff = {}

for col in df.columns:

if row1[col] != row2[col]:

diff[col] = (row1[col], row2[col])



diff_list.append(diff)



return (diff_list)



# Convert spark dataframe to pandas

strSQL = """select * table

where batch_id = (select max(batch_id)

from table)"""



df_spark = spark.sql(strSQL)

df = df_spark.toPandas()



diff_df = compare_rows(df)

df = pd.DataFrame.from_dict(diff_df)



df.to_excel('your_excel_file.xlsx', sheet_name='Sheet1', index=False)



What it outputs now:



What I want:



Continue reading...