[Python] While comparing two DataFrames read from delta, I receive output they are different...

I have a situation where a DataFrame is first saved to Parquet and then to Delta. I needed to explore another format, as saving to one file takes considerable time.

Initially, I compared the original Parquet (coalesced into one file) with the partitioned version using subtract. I found differences, so I removed all columns with empty values. After subtracting the DataFrames, I received empty results.

Next, I worked with Deltas. Despite removing additional columns with empty values, I still did not obtain empty DataFrames after subtraction. However, when I selected one record from each of the subtracted DataFrames and copied the outputs to Excel, they returned as identical after running an IF command.

I also compared the lengths of the values in the mentioned rows, and they were the same, yet they were returned as different by subtract.

How can I identify what differs between these DataFrames? There are other representations of empty values, such as "-" or null, but Copilot indicated that if two DataFrames have identical values row by row, they should not be considered different by subtract.

Unfortunately, I cannot share the data due to business sensitivity.

Continue reading...