[Python] What's the most efficient way to calculate word counts for 4m records and a word list...

Here is my problem. I have a very large file with about 4m rows, where each row has something like a paragraph of text. Then I have a word list with about 150 terms, some are 1-grams and some are 2-grams. I need to make some very fast Python function to get counts of words as dicts for each row of data.

Here's an MRE to get started.

import pandas as pd

# Sample data

# word_list has 1-grams and 2-grams
word_list = ['apple', 'banana', 'dog', 'machine learning', 'data science', 'big data']

# the real data has 4m rows
texts = 5000 * ['I love apple and banana', 'The dog runs fast', 'machine learning is great', 'apple banana dog', 'data science and big data are related']

# we have it in a df for now with a col called `text`
df = pd.DataFrame({'idx': range(len(texts)), 'text': texts})

# need some fast function to return a dict for each `text` that has word count
def count_words_in_texts(df, word_list):
"""
Return list of dicts with word counts for each row
df: DataFrame with 'text' column
word_list: list of words/phrases (1-grams and 2-grams)
Returns: list of dicts, one per row
"""
# TODO: Implement efficient word counting
pass

# Expected output format:
# [{'apple': 1, 'banana': 1, 'dog':0...}...]

results = count_words_in_texts(df, word_list)
print(results)

Continue reading...