[Python] When using asyncio, how can I throttle the rate of a dependent sequence of web...

I am attempting to send out a grid, or lattice, or "matrix" of API requests using asyncio: For a "category 1," I may wish to request the datasets 1a, 1b, 1c. For category 2, I may wish to request the data sets 2a, 2b, 2c. And for category 3, I may wish to request the datasets 3a, 3b, 3c. Whether I request 1b depends on the nature of 1a, and whether I request 1c depends on the nature of 1a and 1b, so I have to request 1a, 1b, and 1c in the listed order. An analogous statement holds for categories 2 and 3.

The server receiving these requests takes about 1 sec to respond to each request, but can respond to up to 10 requests/sec. So I may be able to reduce my runtime by 90% by parallelizing the requests using asyncio. But to do so I need to determine how to throttle my rate of sending to 10 requests/sec.

If I wanted to collect all 1a, 1b, ..., 3c (i.e., if the requests were independent), then I believe I could just use a sequence of create_task calls in a for loop like:

secs_per_req = 0.1
async with aiohttp.ClientSession() as session:
async with asyncio.TaskGroup() as tg:
for k in [1, 2, 3]:
tg.create_task(collect_a_data(k, session))
await asyncio.sleep(secs_per_req)
tg.create_task(collect_b_data(k, session))
await asyncio.sleep(secs_per_req)
tg.create_task(collect_c_data(k, session))
await asyncio.sleep(secs_per_req)


However, the fact that 1b and 1c depend on 1a, and 1c depends on 1b, perhaps suggests that I should try a for loop like:

secs_per_req = 0.1
async with aiohttp.ClientSession() as session:
async with asyncio.TaskGroup() as tg:
for k in [1, 2, 3]:
tg.create_task(collect_abc_data(k, session))
await asyncio.sleep(secs_per_req)


where collect_abc_data(k, session) collects ka and, if needed, kb and kc, and contains all of the logic for deciding whether to request kb and kc. But then I need a way to limit the rate of the requests within collect_abc_data. A natural way to do this would be to sleep (in some manner) immediately before sending each request. However:

• 
  Including an await asyncio.sleep(secs_per_req) before each request in collect_abc_data does not limit the rate at which requests are sent. The parent function just moves on to the next task and continues sending out requests without pause.
  
  
• 
  Including a time.sleep(secs_per_req) before each request achieves closer to the desired result; it blocks any other task from running during the sleep interval, which limits the rate at which requests are sent. However, it also blocks in a second manner: it forces the requests for 1a, 2a, and 3a to be sent before it processes the response to the request for 1a. For my use, this is nonideal because it delays the collection of ka, kb, and kc, which then delays downstream processing of this data. This becomes especially problematic when there are, say, tens of thousands of k values to request, as there are in my case. (The delay may then amount to hours.)
  

So, if the above approaches fail, is there standard way that this problem is solved within the asyncio framework?

Continue reading...