The existing examples of the multiprocessing-module assume, the tasks are independent, but my situation is "hairier". I need to process several hundred thousands of widgets. These are packaged into boxes of 50-100 widgets per box. The processing times may vary widely... All boxes are completely independent from each other. All widgets in one box are independent too, except, each one depends on the very first widget from the same box. So, I thought, I'd create a queue for tasks and a queue for results -- making both queues arguments of multiprocessing.Process target. Now I need to first loop through all boxes, sending first widget from each one to processing. Then, as soon as a box' first widget is done, I'd submit all of the remaining widgets from the box. The question is, how to do that second part properly? Below is the simplified code sending the first widgets from every box for processing. def oneWidget(taskQueue, resultQueue): task = taskQueue.get() start = resource.getrusage(resource.RUSAGE_SELF)[0] result = process(*task) finish = resource.getrusage(resource.RUSAGE_SELF)[0] logger.info("Processed %s in %g", task, finish - start) resultQueue.put(result) tasks = multiprocessing.Queue() results = multiprocessing.Queue() p = Process(target = oneWidget, args = (tasks, results)) p.start() Pending = {} with open(sys.argv[1]) as boxes: for box in boxes: (first, *rest) = box tasks.put(*first) Pending[first] = *rest How do I arrange for the processing of the results -- in the main process -- so that a box' remaining widgets are added to the tasks-queue as soon as the box' first widget is processed? How would the processing finish, on both ends? Is there some nice pythonic way of doing all this? Continue reading...
