[Python] Why does Python's `await` recursion cause the recursion depth to exceed the limit?

I know we should never write pathological code like below in production, the following are are just some examples for illustration.

My question is, why can the following code lead to an exception of exceeding the maximum recursion depth limit?

import asyncio
import logging
import sys

sys.setrecursionlimit(19) # to make it occurr quickly

async def main(num: int = 0):
print(num)
await asyncio.sleep(0.0001)
await main(num + 1)

if __name__ == "__main__":
logging.basicConfig(level=logging.DEBUG)
asyncio.run(main())


# $ python tmp.py
DEBUG:asyncio:Using selector: EpollSelector
0
1
2
3
4
5
6
7
Traceback (most recent call last):
File "/data/projects/data1/xxx/yyy/tmp.py", line 16, in <module>
asyncio.run(main())
File "/data/users/xxx/miniconda3/lib/python3.12/asyncio/runners.py", line 194, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/data/users/xxx/miniconda3/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/users/xxx/miniconda3/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/data/projects/data1/xxx/yyy/tmp.py", line 11, in main
await main(num + 1)
File "/data/projects/data1/xxx/yyy/tmp.py", line 11, in main
await main(num + 1)
File "/data/projects/data1/xxx/yyy/tmp.py", line 11, in main
await main(num + 1)
[Previous line repeated 4 more times]
File "/data/projects/data1/xxx/yyy/tmp.py", line 10, in main
await asyncio.sleep(0.0001)
File "/data/users/xxx/miniconda3/lib/python3.12/asyncio/tasks.py", line 661, in sleep
h = loop.call_later(delay,
^^^^^^^^^^^^^^^^^^^^^^
File "/data/users/xxx/miniconda3/lib/python3.12/asyncio/base_events.py", line 761, in call_later
timer = self.call_at(self.time() + delay, callback, *args,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/users/xxx/miniconda3/lib/python3.12/asyncio/base_events.py", line 778, in call_at
timer = events.TimerHandle(when, callback, args, self, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/users/xxx/miniconda3/lib/python3.12/asyncio/events.py", line 112, in __init__
super().__init__(callback, args, loop, context)
File "/data/users/xxx/miniconda3/lib/python3.12/asyncio/events.py", line 45, in __init__
if self._loop.get_debug():
^^^^^^^^^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded


Yes, this is a pathological recursive function with no exit conditions. If we remove all async and await then call it synchronously, it will of course report an error of exceeding the recursion limit, because the stack frame of each function call is immediately newly created in situ (within the stack frame of the caller function), and the nesting level quickly gets out of control.

But since the asynchronous mechanism is used, shouldn't the call stack frame of each function be very shallow, and then rely on the callbacks arranged by the event loop to organize the dependencies between these stack frames? In other words, I understand that each stack frame should be independent, it only needs to wait for the completion of the awaited callee, without considering how and where it is completed. The event loop will be responsible for providing a location for the actual call (rather than directly expanding it in situ like synchronization). Therefore, even if there are some memory issues, it should be memory exhaustion, not recursion limit exceeding.

However, it turns out that my speculation is incorrect, it seems that await really maintains the dependency with callees directly in the stack frame of the outermost async func just like synchronization:

File "/data/projects/data1/xxx/yyy/tmp.py", line 11, in main
await main(num + 1)
File "/data/projects/data1/xxx/yyy/tmp.py", line 11, in main
await main(num + 1)
File "/data/projects/data1/xxx/yyy/tmp.py", line 11, in main
await main(num + 1)
[Previous line repeated 4 more times]


Can anyone explain what is going on?

By the way, I know that changing await main(num+1) to asyncio.create_task(main(num+1)) will eliminate the dependency between coroutines, so the recursion limit exception will not be triggered. This is in line with expectations even in my wrong theory.

Continue reading...