When I do an operation in STATA, for example removing duplicated rows, it will tell me the number of rows removed. For the tidyverse, the package tidylog implements a similar feature, providing feedback on the operation (e.g. for a join, number of joined and unjoined rows, for a filter, nubmer of removed rows, etc.), with the little disadvantage that you will lose the autocompletion of your editor, as it wraps tidyverse functions with definitions like filter(...), to accomodate for the fact that the tidyverse upstream definition could change over time. Is there something similar for pandas? I found pandas-log but seems abandoned. EDIT: For the time being, I'm monkeypatching with: def my_drop_duplicates(df, *args, **kwargs): nrow0 = df.shape[0] df.drop_duplicates(*args, **kwargs) nrow1 = df.shape[0] logging.info(f"Dropped {nrow0-nrow1} duplicates") pd.DataFrame.my_drop_duplicates = my_drop_duplicates I guess you could also do something like def my_drop_duplicates(df, *args, **kwargs): nrow0 = df.shape[0] df.internal_drop_duplicates(*args, **kwargs) nrow1 = df.shape[0] logging.info(f"Dropped {nrow0-nrow1} duplicates") pd.DataFrame.internal_drop_duplicates = drop_duplicates pd.DataFrame.drop_duplicates = my_drop_duplicates Continue reading...
