[Python] How to normalize ingredient names in a recipe dataset and handle NOUN + NOUN cases...

I'm working on normalizing ingredient names from a recipe dataset using Python and spaCy. My goal is to extract only the relevant ingredients and ignore measurement units, fractions, and other unnecessary details. For instance, if I have a string like: "5 tablespoons butter, divided". I want to extract "butter" as the normalized ingredient. However, i struggle to parse the string "8 cups broccoli florets" - it loses "broccoli" and outputs only "floret". Other strings, like "3 cups chicken broth", work fine and output "chicken broth".

My final version of the function is:

`def normalize_ingredient(ingredient_text):
doc = nlp(ingredient_text)

# Set of measurement units to exclude
measurement_units = {
"cup", "teaspoon", "tablespoon", "tablespoons", "gram", "ounce", "pound", "can",
"clove", "pinch", "dash", "quart", "liter", "milliliter", "gallon",
"stick", "rib", "head", "package", "inch", "piece", "fluid", "container",
"jar", "loaf", "bottle", "pack", "pint", "cube", "stalk", "slice", "bulb",
"strip", "packet", "envelope", "box", "bag", "carton", "sprig", "leaf",
"half", "purpose", "pound", "ounce", "gram", "milliliter", "liter", "gallon",
"quart", "pint", "dash", "pinch", "clove", "can", "package", "container",
"jar", "loaf", "bottle", "pack", "cube", "stalk", "bulb", "strip", "packet",
"envelope", "box", "bag", "carton", "sprig", "leaf", "fluid", "inch", "piece", "cup",
"bite", "size", "bunch", "cups","all", "sized", "chunks"
}

# Set of fractions to exclude
fractions = {'½', '¼', '¾', '⅓', '⅔', '⅛', '⅜', '⅝', '⅞', '⅙', '⅚', '®'}

# List to store relevant terms
relevant_terms = []

for token in doc:
# Skip numbers, fractions, and measurement units
if (token.like_num or
token.text in fractions or
token.lemma_.lower() in measurement_units):
continue

# Focus on nouns, proper nouns, and adjectives that modify nouns
if token.pos_ in {"NOUN", "PROPN", "ADJ"}:
if token.pos_ == "ADJ" and token.head.pos_ in {"NOUN", "PROPN"}:
relevant_terms.append(token.lemma_.lower())
elif token.pos_ in {"NOUN", "PROPN"}:
relevant_terms.append(token.lemma_.lower())

return " ".join(relevant_terms)`


This skips "broccoli" for unknown for me reasons (this is the one and only incorrectly parsed string for now). I also tried this approach with compounds:

`if token.pos_ in {"NOUN", "PROPN", "ADJ"}:

if token.pos_ == "ADJ" and token.head.pos_ in {"NOUN", "PROPN"}:
relevant_terms.append(f"{token.lemma_.lower()} {token.head.lemma_.lower()}")
elif token.pos_ in {"NOUN", "PROPN"}:
compound = [t for t in token.children if t.dep_ == "compound"]
if compound:
combined = " ".join([t.lemma_.lower() for t in compound] + [token.lemma_.lower()])
relevant_terms.append(combined)
else:
relevant_terms.append(token.lemma_.lower())`


This messed up all the logic and didn't exclude some measurement units.

Finally, i would like to avoid adding special case with broccoli like this:

if token.pos_ in {"NOUN", "PROPN", "ADJ"} or token.lemma_.lower() == "broccoli":
if token.pos_ == "ADJ" and token.head.pos_ in {"NOUN", "PROPN"}:
relevant_terms.append(token.lemma_.lower())
elif token.pos_ in {"NOUN", "PROPN"} or token.lemma_.lower() == "broccoli":
relevant_terms.append(token.lemma_.lower())

Continue reading...