[Python] LDA perplexity with train-test split leads to absurd results (best model = 1 topic)

I'm working with LDA on a Portuguese news corpus (~800k documents with an average of 28 words each after cleaning the data), and I’m trying to evaluate topic quality using perplexity.

When I compute perplexity on the same corpus used to train the model, I get the expected U-shaped curve: perplexity decreases, reaches a minimum, then increases as I increase the number of topics.

However, when I split the data using KFold cross-validation, train the LDA model only on the training set, and compute perplexity on the held-out test set, the curve becomes strictly increasing with the number of topics — i.e., the lowest perplexity is always with just 1 topic, which obviously defeats the purpose of topic modeling.

I'm aware that simply using log_perplexity(corpus_test) can be misleading because it doesn’t properly infer document-topic distributions (θ) for unseen documents. So I switched to:

bound = lda_model.bound(corpus_test)
token_total = sum(cnt for doc in corpus_test for _, cnt in doc)
perplexity = np.exp(-bound / token_total)


But I still get the same weird behavior: models with more topics consistently have higher perplexity on test data, even though their training perplexity is lower and their coherence scores are better.

Is there a way to properly evaluate LDA on a test set using perplexity in a way that reflects actual model quality (i.e., not always favoring a degenerate 1-topic solution)?

Any help, suggestions, or references would be greatly appreciated — I’ve been stuck on this for a while. Thanks!

The code:

df = dataframe.iloc[:100_000].copy()

train_and_test = []
for number_of_topics in [5, 25, 45, 65, 85]:

print(f'\033[1m{number_of_topics} topics.\033[0m')

KF = KFold(n_splits=5, shuffle=True, random_state=42)

iteration = 1
for train_indices, test_indices in KF.split(df):
print(f'K{iteration}...')
print('Preparing the corpora.')

train_df = df.iloc[train_indices].copy()
train_texts = train_df.corpus.apply(str.split).tolist()
train_dictionary = corpora.Dictionary(train_texts)
train_corpus = [train_dictionary.doc2bow(text) for text in train_texts]

test_df = df.iloc[test_indices].copy()
test_texts = test_df.corpus.apply(str.split).tolist()
test_corpus = [train_dictionary.doc2bow(text) for text in test_texts]

print('Running the LDA model!')
lda_model = LdaMulticore(corpus=train_corpus, id2word=train_dictionary,
num_topics=number_of_topics,
workers=mp.cpu_count(), passes=10)

bound = lda_model.bound(test_corpus)
tokens = sum(cnt for doc in test_corpus for _, cnt in doc)
perplexity = np.exp(-bound / tokens)
print(perplexity, '\n')

train_and_test.append([number_of_topics, iteration, perplexity])
iteration += 1

Continue reading...