[Python] Cancelling All Tasks on Failure with `concurrent.futures` in Python

I am using Python's concurrent.futures library with ThreadPoolExecutor and ProcessPoolExecutor. I want to implement a mechanism to cancel all running or unexecuted tasks if any one of the tasks fails. Specifically, I want to:

1. Cancel all futures (both running and unexecuted) when a task fails.
2. Raise the error that caused the first task to fail if that error is silently ignored; otherwise, let Python raise it naturally.

Here is the approach I have tried:

from concurrent.futures import ProcessPoolExecutor, as_completed
from functools import partial

copy_func = partial(copy_from, table_name=table_name, column_string=column_string)
with ProcessPoolExecutor(max_workers=cores_to_use) as executor:
futures = {executor.submit(copy_func, file_path): file_path for file_path in file_path_list}
for f in as_completed(futures):
try:
f.result()
except Exception as e:
executor.shutdown(wait=False) # Attempt to stop the executor
for future in futures:
future.cancel() # Cancel all futures
raise e # Raise the exception

Questions:

• Is this the correct way to handle task cancellation in ThreadPoolExecutor and ProcessPoolExecutor?
• Are there any better approaches to achieve this functionality?
• How can I ensure that the raised exception is not silently ignored?
• How can I free up all resources used by concurrent.futures after an exception?

Thank you!

Continue reading...