I'm running a GridSearchCV optimization into a parallelized function. The pseudocode looks like this from tqdm.contrib.concurrent import process_map from sklearn.model_selection import GridSearchCV def main(): results = process_map(func, it, max_workers=5) # We never reach here with n_jobs > 1 in GridSearch def func(it): ... grid_search = GridSearchCV(..., n_jobs=5) ... return result if __name__ == "__main__": main() If n_jobs > 1 the script indefinitely hangs when returning results and never proceeds further (although all the func tasks have completed). If I set n_jobs=1 then everything works fine. I think (but I'm not sure) this is related to the fact that process_map uses a different spawn mechansim from GridSearchCV (which internally uses joblib if I understand it correctly). As the heaviest part of this algorithm is the grid search, isn't there any way of maintaining that parallelism together with the outer parallelism layer? Continue reading...
