[Python] Which library is more reliable for LDA and perplexity: gensim or scikit-learn?...

When calculating the perplexity of an LDA model for N topics using train-test split with KFold, I noticed that in Gensim, the perplexity consistently increases as the number of topics grows—resulting in the odd situation where even a 1-topic model yields a lower perplexity than more complex ones (N>1 topics). However, when performing the same modeling using scikit-learn, the perplexity decreases as the number of topics increases, reaches a minimum, and then starts rising again (which is the expected behavior). What could be causing this discrepancy? Which library provides more reliable perplexity and LDA measurements?

Continue reading...