[Python] When is `async for` faster than `for` (python)?

In python, one can replace for [...] in [generator] by async for [...] in [awaitable generator]. Clearly, this lowers the overall execution time of a program if the [generator] contains some slow I/O operation that can be turned into a coroutine. I give a detailed example of this case below, along with analytic expressions for the execution times in various cases.

My question

Is this the only case where using async for instead of for lowers the overall execution time of a program?

Or can async for be used for other purposes as well? If yes, please give practical examples that illustrate how async for makes smth run faster than with for.

Example code

• Make sure to call TEST_ASYNC_FOR() in the correct way; see comments at end of code.
• Feel free to change BLK, NBLK, GEN and N to play around.

async def TEST_ASYNC_FOR():

import asyncio
import time

BLK = 0.15 # appears as `time.sleep(BLK)` inside both for-loops
NBLK = 0.16 # appears as `await asyncio.sleep(NBLK)` inside both for-loops
GEN = 0.17 # appears as `time.sleep(GEN)` (`await asyncio.sleep(GEN)`) inside the standard (async) generator
N = 3 # number of iterations in each for-loop

def generator():
for i in range(N):
time.sleep(GEN) # slow I/O operation blocks thread
yield i

async def async_generator():
for i in range(N):
await asyncio.sleep(GEN) # slow I/O operation yields control to loop
yield i

async def standard_for():
for i in generator():
time.sleep(BLK) # CPU-intensive operation blocks thread
await asyncio.sleep(NBLK) # slow I/O operation yields control to loop

async def async_for():
async for i in async_generator():
time.sleep(BLK) # CPU-intensive operation blocks thread
await asyncio.sleep(NBLK) # slow I/O operation yields control to loop

t0 = time.perf_counter()
await standard_for()
print(f"1x for-loop:\t\t\t {time.perf_counter()-t0:.2f} | expected: {N*BLK + N*NBLK + N*GEN:.2f} = N*BLK + N*NBLK + N*GEN")

t0 = time.perf_counter()
await async_for()
print(f"1x async for-loop:\t\t {time.perf_counter()-t0:.2f} | expected: {N*BLK + N*NBLK + N*GEN:.2f} = N*BLK + N*NBLK + N*GEN")

def standard_for_duration(BLK, NBLK, GEN, N):
t = 2*N*BLK + 2*N*GEN + NBLK + (N-1)*max(NBLK-(BLK+GEN), 0)
return t, "2*N*BLK + 2*N*GEN + NBLK + (N-1)*max(NBLK-(BLK+GEN), 0)"

t0 = time.perf_counter()
await asyncio.gather(standard_for(), standard_for())
t_st, expr_st = standard_for_duration(BLK, NBLK, GEN, N)
print(f"2x for-loop (separate tasks):\t {time.perf_counter()-t0:.2f} | expected: {t_st:.2f} = {expr_st}")

def async_for_duration(BLK, NBLK, GEN, N):
t = 2*N*BLK + N*GEN + NBLK + (N-1)*max(NBLK-BLK, 0) + (N-1)*GEN*(BLK > NBLK)*(NBLK > GEN) + GEN*(NBLK > BLK)*(BLK > GEN)
return t, "2*N*BLK + N*GEN + NBLK + (N-1)*max(NBLK-BLK, 0) + (N-1)*GEN*(BLK > NBLK)*(NBLK > GEN) + GEN*(NBLK > BLK)*(BLK > GEN)"

t0 = time.perf_counter()
await asyncio.gather(async_for(), async_for())
t_async, expr_async = async_for_duration(BLK, NBLK, GEN, N)
print(f"2x async for-loop (separate tasks): {time.perf_counter()-t0:.2f} | expected: {t_async:.2f} = {expr_async}")

await TEST_ASYNC_FOR() # use inside jupyter lab, which already has an event loop running

# import asyncio
# asyncio.run(TEST_ASYNC_FOR()) # use in a standalone python script


The code outputs:

1x for-loop: 1.44 | expected: 1.44 = N*BLK + N*NBLK + N*GEN
1x async for-loop: 1.44 | expected: 1.44 = N*BLK + N*NBLK + N*GEN
2x for-loop (separate tasks): 2.08 | expected: 2.08 = 2*N*BLK + 2*N*GEN + NBLK + (N-1)*max(NBLK-(BLK+GEN), 0)
2x async for-loop (separate tasks): 1.59 | expected: 1.59 = 2*N*BLK + N*GEN + NBLK + (N-1)*max(NBLK-BLK, 0) + (N-1)*GEN*(BLK > NBLK)*(NBLK > GEN) + GEN*(NBLK > BLK)*(BLK > GEN)

Continue reading...