What I want to do: I have CPU intensive computation that I want to offload to a worker process so as not to block my main server. Without ProcessPoolExecutor my code looks like this: my_model = model_init() def handler(args): return my_model.process(args) How do I utilize ProcessPoolExecutor? consider the function that creates a ProcessPoolExecutor: concurrent.futures.ProcessPoolExecutor(max_workers=1, initializer=init_function, initargs=(model_name)) say I have: def init_function(): model = model_init() The core of the problem is how to "persist" the model created in the init_function? Later on I want to submit tasks to the ProcessPoolExecutor and be able to use the model created in the init_function to process the task. Its difficult to explain - the issue is that ProcessPoolExecutor only knows about two functions the initializer and the function I will provide in submit. I don't know how to pass state between these functions. How can the function that is argument to submit access a variable that was initialized when initializer was called? The only example I found is here and that does not go over this issue. Continue reading...
