I am trying to cluster words using Kmeans. I have one large document and I use an NLTK RegexpTokenizer first and then filter based on word count, length and remove stopwords. Next, I construct a co-occurrence matrix and use this to train the Kmeans model. Finally, I test the performance of this using silhouette scores. I would like to visualise these clusters. This typically seems to be done using a scatter plot and the labels. How do I reduce this co-occurrence matrix to x and y, when it is currently an N x N matrix (N is the number of unique words)? I have tried using PCA: plt.figure() pca_2d = PCA(n_components=2) reduced = pca_2d.fit_transform(mat) newKm = KMeans(n_clusters=3) labels = newKm.fit_predict(reduced) plt.scatter(reduced[:, 0], reduced[:, 1], c=labels) # selects column 0 (all rows) as x coords and column 1 (all rows) as y. and then cluster labels. plt.title("K-means Clustering") plt.show() But am not sure if this is the best or correct approach? Continue reading...
