[Python] The fit method of sklearn.cluster.KMeans does not work in the map method of...

I have been trying to do some clustering simultaneously using the following code:

processes = 10
with Pool(processes) as pool:
cc = pool.map(model_fit_pred, range(processes))


where model_fit_pred is the function below:

def model_fit_pred(i: int):
model = KMeans(n_clusters=len(features), random_state=i)
model.fit(X)
return model.predict(X)


But the computation kept on and on and my CPU usage did not even surpass 20%. So, in order to find out which method is not doing its task I printed logs after KMeans and fit. There was something wrong with fit.

What is preventing fit from being used in a multi-process computation?


I ran model_fit_pred(0) to test the function itself; it was executed in 11 ms. On the other hand,

processes = 1
with Pool(processes) as pool:
cc = pool.map(model_fit_pred, range(processes))


kept running and I stopped it after 10 min with no return.

Continue reading...