Is there a model or library that has a prebuilt multiword expression repo? I am trying to perform word counts in a series of paragraphs but would like to include ngrams. However, I have run into a snag as some of my sentences contain bigrams and trigrams which hold more meaning than just the monograms. I thought of just doing a series of ngrams and performing the counts but this results in over counting. example using ngrams import nltk sentences = ["the dog jumped over the green car and into the market place", "the cat was sleeping in the market place", "the man in a green car was waiting near the market place"] word_counts = {} for sentence in sentences: word_list = nltk.word_tokenize(sentence) ngrams = nltk.everygrams(word_list, 1, 3) for word in ngrams: if word not in word_counts.keys(): word_counts[word] = 0 word_counts[word] += 1 result {('the',): 7, ('the', 'dog'): 1, ('the', 'dog', 'jumped'): 1, ('dog',): 1, ('dog', 'jumped'): 1, ('dog', 'jumped', 'over'): 1 ... I also considered using mwetokenizer and while this does what I want, I run into the issue where I have to define the MWE. example using mwetokenizer from nltk.tokenize import MWETokenizer mwe_tokenizer = MWETokenizer() mwe_tokenizer.add_mwe(('green','car')) mwe_tokenizer.add_mwe(('market', 'place')) word_counts = {} for sentence in sentences: word_list = nltk.word_tokenize(sentence) new_word_list = mwe_tokenizer.tokenize(word_list) for word in new_word_list: if word not in word_counts.keys(): word_counts[word] = 0 word_counts[word] += 1 result {'the': 7, 'dog': 1, 'jumped': 1, 'over': 1, 'green_car': 2, 'and': 1, 'into': 1, 'market_place': 3, ... Continue reading...
