The following code start 5 multiprocessing tasks. Those tasks print a message then go to sleep a random period of time then do this again. That is repeated 10 time so the entire set of tasks should complete in about 50 seconds at most. When the pool is first created, the addresses of the 5 pool objects are printed. This code works fine, but it was terminated before it completed all its work. When it is run again, you can see new addresses are assigned for the 5 pool objects. After this process of stopping the operation before it is done (keyboard interrupt), several times, the code stops working. The tasks are never created and the code falls right through to the end. And the allocation of pool objects show that they are allocating the same objects over and over whereas before new object addresses where assigned. So it appears that OS resources are being allocated and never released. And when those resources are all used up, the code stops working. But even when this command prompt it is running under is terminated (its window closed down), those resources are not being returned. And when a new command prompts is open and the app run again, you can see the same object IDs being assigned, the tasks are never started, and the code falls though. Normally the code stops at the pool.join() waiting for the workers (task) to complete. There does not seem to be a way to get the OS to release these pool resources. The only way is to restart the OS. During debugging of code, it is unlikely that one would ever have it run to completion and to have the pool.close() and poll.join() executed. SO after a few execution of the code being tested, no more tasks are allocated. Thus the question - how do we ensure the spawned off tasks are terminated and their resources returned to the pool? import multiprocessing import time import random # Simultaneous tasks running def task(id): for i in range(10): stime = random.randint(1,5) print(f"Task {id}: Woke up. Now sleep {stime}") time.sleep(stime) exit(0) if __name__ == "__main__": print ('create pool of tasks') pool = multiprocessing.Pool(processes=5) print (pool) # Start 5 asynchronous tasks for i in range(1,6): result = pool.apply_async(task, i) print (result) pool.close() # Wait for all tasks to complete pool.join() print ('end program') Continue reading...
