[Python] Is it possible to to add a part of speech tag to a spaCy tokenizer special case?

I have a list of compounds and other special cases that I want spaCy to recognise as tokens, such as 'according to' and 'washing machine'. I used 'add_special_case' to do this:

import spacy
from spacy.attrs import ORTH

nlp = spacy.load("en_core_web_sm")

text = "according to reports the washing machine industry is declining"

special_case_1 = [{ORTH: 'according to'}]
nlp.tokenizer.add_special_case('according to', special_case_1)

special_case_2 = [{ORTH: 'washing machine'}]
nlp.tokenizer.add_special_case('washing machine', special_case_2)

doc = nlp(text)

for token in doc:
print(token, token.pos_)


This seems to work, but the POS tag is now proper noun, when it should be preposition (ADP) for 'according to' and NOUN for 'washing machine':

according to PROPN
reports VERB
the DET
washing machine PROPN
industry NOUN
is AUX
declining VERB


Does anybody know if there is a way to assign a POS tag to a special case? (I couldn't find anything in the documentation). Or maybe there is a better way to tokenize compounds?

Solution

The solution was to use the AttributeRuler in spaCy https://spacy.io/usage/linguistic-features#mappings-exceptions

import spacy
from spacy.attrs import ORTH

nlp = spacy.load("en_core_web_sm")

text = "according to reports the washing machine is more
popular than air conditioning because of an unknown reason"

compounds = ['according to', 'because of', 'washing machine',
'air conditioning']

for comp in compounds:
special_case = [{ORTH: comp}]
nlp.tokenizer.add_special_case(comp, special_case)


This gave the following tags to the 4 compounds:

according to PROPN NNP
washing machine PROPN NNP
air conditioning VERB VBG
because of VERB VBZ


Then I used the AttributeRuler:

ruler = nlp.get_pipe("attribute_ruler")

prepositions = ['according to', 'because of']
nouns = ['washing machine', 'air conditioning']

for prep in prepositions:
preposition_patterns = [[{"LOWER": prep}]]
preposition_attrs = {"TAG": "IN", "POS": "ADP"}
ruler.add(patterns=preposition_patterns,
attrs=preposition_attrs)

for noun in nouns:
noun_patterns = [[{"LOWER": noun}]]
noun_attrs = {"TAG": "NN", "POS": "NOUN"}
ruler.add(patterns=noun_patterns, attrs=noun_attrs)


This time the output was what I wanted:

doc = nlp(text)

for token in doc:
print(token, token.tag_, token.pos_)


Full output:

according to IN ADP
reports VBZ VERB
the DT DET
washing machine NN NOUN
is VBZ AUX
more RBR ADV
popular JJ ADJ
than IN ADP
air conditioning NN NOUN
because of IN ADP
an DT DET
unknown JJ ADJ
reason NN NOUN


I still need to work on tokenizing the lemma of a compound noun (including the plural form). For my case, the list of compounds is small enough to manually add the plural form of each compound noun to the compound list, but doing that is probably not the best solution.

Continue reading...