Given a generic text sentence (in a specific context) how can I extract word/entities of interest belonging to a specific "category" using python and any NLP library? For example given a step for a culinary recipe Add an onion to a bowl of carrots as input text, I'd like to retrive onion and carrots while given Sprinkle with paprika. should return paprika. But this should also work with sentences like stir well, and cook an additional minute. that do not contain any food entity in them. So far, what I was able to achieve is using the spacy library for training a NER module to parse sentences. The problem with the NER pipeline is that it is a rule-based parsing, it is trained providing a set of sentences and entities/matches/labels to learn, which works fine as expeted on sentences similar to the one used during train, but performs bad on new and different sentences: nlp = spacy.load('trained_model') document = nlp('Add flour, mustard, and salt') [(ent.text, ent.label_) for ent in document.ents] # >> [('Add flour', 'FOOD'), ('mustard', 'FOOD'), ('salt', 'FOOD')] # (quite) correct output document = nlp('I took a building, car and squirrel on the weekend') [(ent.text, ent.label_) for ent in document.ents] # >> [('building', 'FOOD'), ('car', 'FOOD'), ('squirrel', 'FOOD')] # wrong output document = nlp('stir well, and cook an additional minute.') [(ent.text, ent.label_) for ent in document.ents] # >> [('stir well', 'FOOD'), ('cook', 'FOOD'), ('additional minute.', 'FOOD')] # wrong output I am aware that there are several similar questions and posts, but I have found only solutions working for "semi-structured" text, i.e. list of ingredients as 1 tsp. of sugar, 1 cup of milk, ... which can be easily solved using the previous rule-based approach. Also nltk and part-of-speech (POS) are an option, but I'd prefer an alternative solution rather than having to compare each noun with an exhaustive list of foods. What instead I am looking for is a way of to extract specific entities or at least to classify words in generic text with additional categories beyond those of the basic parsing. Which methods should I use/look at to achieve this? Continue reading...
