I am trying to implement a stop flag that gracefully prevents new parallel jobs from being started. Specifically, I am running a large number of simulations that each takes a few to many hours; in order not to lose intermediate results, I would like the ability to tell my code to finish the ones that are currently running but not start any new ones. After lots of trial and error, including with imap, apply_async, etc, I came back to imap_unordered and a stop-aware generator. CM_STOP_FLAG = 'tmp/cm_stop_flag' ... def stop_aware_generator(params): for param in params: if os.path.exists(CM_STOP_FLAG): print("Stop flag detected. Stopping generator.") break yield param def run_sims(n_procs): ... with Pool( processes=n_procs, initializer=_init_worker, initargs=(lock_a, lock_b) ) as pool: for _ in pool.imap_unordered( cmcompute, stop_aware_generator(param_list), chunksize=1 ): pass My understanding is that this should at most pre-load n_procs * chunksize jobs, but while the generator clearly detects the flag being set (as evidenced by on-screen output), it keeps submitting jobs regardless. What is the correct way of implementing a graceful exit/stop flag? Thanks. Continue reading...
