[Python] RuntimeWarning: '<' not supported between instances of 'int' and 'tuple', sort order...

I have two dataframes which I am trying to join along the columns. Both the dataframes have sorted index of type pandas.DatetimeIndex. Both the dataframes have columns have columns of type pandas.MultiIndex, i.e. they both have multilevel columns. The indices of the dataframes have no NaN/NaT values but they are not equal, i.e. there are some elements which are present in only one of the two indices. I am trying to achieve my join as follows:

df = pd.concat([left_df, right_df], axis=1, sort=False)


On doing so, I get the error:

RuntimeWarning: '<' not supported between instances of 'int' and 'tuple', sort order is undefined for incomparable objects.


On further inspection, I notice that the columns of the first dataframe (observed using left_df.columns) are tuples where the first element is a string and the second element is a string for some columns and a tuple of two strings for some columns. The columns of the second dataframe are tuples where the first element is a string and the second element is an integer.

As an alternative to pd.concat, I tried using pd.merge as follows:

df = pd.merge(left=left_df, right=right_df, left_index=True, right_index=True) #sort is False and how is 'inner' by default


I get the same error on running the above line as well. It seems like I get this error because Pandas is trying to compare columns to arrange them where one of the elements is a tuple and the other is an integer. But this should not be happening because we have explicitly mentioned sort=False.

As an alternative, I tried doing the same for two dataframes where the first has columns of type Tuple[str, str] and the second has columns of type Tuple[str, Tuple[str, str]] and I do not get the warning in this case.

How do I elegantly without explicitly creating a dataframe and populating it with the columns of each dataframe one by one in a for loop? Is this an unresolved bug in Pandas?

Continue reading...