[Python] How to propagate KeyboardInterrupt to multiprocessing Pool workers and gracefully...

I am using the multiprocessing.Pool in Python to parallelize a task that requires entering and exiting a context manager for each item in the workload. I want to ensure that when a KeyboardInterrupt occurs, the program gracefully exits by allowing each worker process to complete the exit method of its context manager by propagating the interruption (or something like that).

Here is a simple code snippet to illustrate my thoughts:

class MyContext:
def __init__(self, name):
self.name = name

def __enter__(self):
print(f'Entering context {self.name}')
return self

def __exit__(self, exc_type, exc_value, traceback):
print(f'Exiting context {self.name}')
time.sleep(5)
print(f'Exited context {self.name}')


def process_name(name):
with MyContext(name):
print(f'Processing {name}')
time.sleep(100)
print(f'Finished {name}')


names = ['Alice', 'Bob']

for name in names:
process_name(name)


Here is the result I expect:

Entering context Alice
Processing Alice
Entering context Bob
Processing Bob
^CExiting context Alice
Exiting context Bob
Exited context Alice
Exited context Bob
KeyboardInterrupt (maybe multiple times)


Here is the actual output:

Entering context Alice
Processing Alice
Entering context Bob
Processing Bob
^CExiting context Alice
Exiting context Bob
KeyboardInterrupt


The context manager's exit method is called, but it doesn't finish before the program crashes. What should I change to make it work as intended?

Continue reading...