[Python] How to dynamically add a thread or pick a thread from the pool of thread in python

I'm looking to schedule tasks to be executed by threads, with the ability to dynamically add or remove tasks from the process. My goal is to pre-create a pool of threads, for example, [th1, th2, th3], and then assign tasks (e.g., t1, t2, t3) to these threads via an API call to my system.

When I receive a task like t1, I want to select a random thread from the pool and assign this task to it. So far, I've experimented with ThreadPoolExecutor in Python, but it doesn't seem to provide a way to dynamically add or remove tasks from an already running process.

Note: The tasks I'm dealing with are event listeners that run continuously until I decide to stop listening to the event bus. Therefore, I thought having a pool of threads from which I can pick and assign event listener tasks would be beneficial.


def run(task_arg):
# core logic of the listener which continuously listens for the
# events

def using_executor():
argument_dicts = [
{"topic": "/captureSaveEvent"},
]
logging.basicConfig()

with concurrent.futures.ThreadPoolExecutor(max_workers=20) as executor:
futures = {executor.submit(run, arg): arg for arg in argument_dicts}
for future in concurrent.futures.as_completed(futures):
try:
future.result()
except Exception as e:
print(f"Thread for {futures[future]} failed with exception: {e}")



In the above I have defined the task to be submitted in argument_dicts but I want to add a task to this same process by adding a new thread or picking a thread of the thread pool and assign the task to it, I don't want to create a new process and create new threads for every new task I want to add, since the task are listener events which won't exit, so the process which created the threads will continue to be active as long as I quit the listener or some issue in the listener logic.

What I expect is, I need to know the process id, and using that pid, I should be able to pick a thread from this process which is free to take new tasks and submit the job to it.

Continue reading...