Challenge Suppose I have a list L=30 tasks. Each task takes a variable time to complete and has a 10% chance of failing. If a task T fails, add the task back to a queue and prioritize it's completion. Also I can only process at most N=8 number of tasks. I want to use Python's asyncio.as_completed() function to yield the result of a task as soon as it's completed. Once done, add one more task from the task_list L (to make sure we are constantly processing 8 tasks at any given instance). How can I do this in Python? Current Scenario In my current scenario, I have a queue of L=1000 tasks and at a time I can solve N=20 tasks like this: # This approach creates a task of all 1000 items in the queue L and then processes them. while not download_complete(): task_list = [asyncio.create_task(work()) for work in L] for task in asyncio.as_completed(task_list): result = await task yield result Problem The problem however with my current approach is when a task fails, I add it back to the task queue L. Now asyncio.as_completed() will first finish processing all the tasks which are currently already in the task_list before creating a task for all items which have failed during the previous iteration. It's because of this, the failed tasks don't get a priority over the tasks which are already in the task_list. Question So how can I make it so that I can constantly keep processing N number of tasks at a time and yeild the result of the next earliest completion. Once a task is done, fetch one more task from a queue of 1000 tasks and if a task fails, add it back to asyncio.as_completed and prioritize it completion? Continue reading...
