[Python] How do I design a python algorithm for a dataset of 100k+ rows that checks whether...

I have a dataset of 100k+ rows that contains information about our bidding pipeline, such as whether an opportunity is active or not, and why. Many of these opportunities are marked as inactive because they are 'duplicates' of other opportunities. I have been designing an algorithm that evaluates whether it is likely each opportunity flagged as a duplicate is likely to actually be a duplicate or if they are just covering up another reason why the opportunity isn't continuing, such as they lost it.

The problem I am facing is that it seems only a genius could produce an algorithm that doesn't use either .iterrows() or .apply() to compare each opportunity to every other opportunity in the pipeline to assess if criteria overlap, such as the projects names, the project income, the client, etc, and I am not that genius.

I am not necessarily asking anyone to write code but what other approaches could I use for this that are more efficient?

Please let me know if more information is needed.

Continue reading...