Applying set() to a list containing multiple NaN values usually removes duplicate NaN entries. Example: set([np.nan, 5, np.nan, 17, 5, np.nan, 23]) yields: {5, 17, nan, 23} However, I now have a list originating from summing (concatenating) different lists contained in a column of a dataframe; some of these lists contain NaNs. When I apply set() to the concatenated list retrieved from , it does not remove duplicate NaNs. See the below screenshot: The object a1 from the screenshot is definitely of list type. I'm not sure, but maybe it depends on if the NaNs were at some point within a numpy array (or pandas dataframe, presumably): set([np.nan, np.nan, np.nan]) yields {nan} set(np.array([np.nan, np.nan])) yields {nan, nan, nan} set(list(np.array([np.nan, np.nan]))) also yields {nan, nan, nan} Any ways to avoid this other than np.unique() or pd.unique()? Any reason why set() would behave like this (I'd assume the expected default would be to remove duplicate NaNs, even if NaN != NaN!)? Continue reading...
