I read a lot of posts about parallelization using the multiprocessing module but none of them quite answered my question. I have a very long generator giving me parameter values and for each I want to compute some function value. However, I only want to save the best n many, since I am only interested in the best ones and saving all of the results would blow up the RAM. There way I see it, there are two ways to do this: 1) use a common shared memory between the processes where the best values are saved or 2) keep separate lists of the best results for each core/process and later manually merge these lists together. I think the second method would be better, however I am not sure how to implement this. This is what I got so far: import numpy as np import multiprocessing from functools import partial def get_generator(length: int): for i in range(length): yield [i, i + 1] def some_func(x, other_stuff): y = np.sum(x) return y def task(other_stuff, x: np.ndarray): val = some_func(x, other_stuff) if val > task.some_dict['min']: task.l.append(val) task.some_dict['min'] = val return def task_init(l, some_dict): task.l = l task.some_dict = some_dict task.some_dict['min'] = np.NINF n = 20 generator = get_generator(n) other_stuff = np.nan func = partial(task, other_stuff) l = multiprocessing.Manager().list() some_dict = multiprocessing.Manager().dict() p = multiprocessing.Pool(None, task_init, [l, some_dict]) p.imap(func, generator, chunksize=10000) p.close() p.join() This would be somewhat similar to what I want to do. But I really care about performance and in the actual code the comparison/saving of the best values will be more complex so I think that the shared memory approach would be really slow. My question boils down to: If I have e.g. 8 cores, how could I have 8 lists of the best results each for one core that will be returned, so that the cores work completely independent and rather quick? Thank you very much! Continue reading...
