[Python] Latent Dirichlet Allocation (LDA) performance by limiting word size for Corpus...

I have been generating topics with yelp data set of customer reviews by using Latent Dirichlet Allocation (LDA) in Python (gensim package). While generating tokens, I am selecting only the words having length >= 3 from the reviews (by using RegexpTokenizer):

from nltk.tokenize import RegexpTokenizer
tokenizer = RegexpTokenizer(r'\w{3,}')
tokens = tokenizer.tokenize(review)


This will allow us to filter out the noisy words of length less than 3, while creating the corpus document.

How will filtering out these words affect performance with the LDA algorithm?

Continue reading...