[Python] How to handle an asyncio Python library and may be called both from CLI and a Jupyter...

So I have a custom library, packaged with setuptools. In essence, it is a library that sends a lot of tasks to an API in parallel, thus concurrency is used with asyncio. So far, everyone has been using it from the command line like so:

python -m my_library.main

Or even from a Docker container like docker run $IMAGE (the app must be containerized because it has to run in cloud environments that demand a docker container).

All of this without a problem. But now, a new requirement has arisen that some users want to execute this library's code from a Jupyter Notebook. This is being very troublesome, because the library uses asyncio.run under the hood to start running its coroutines, and that raises an error when called from a notebook because there is another event loop running.

I have tried to rewrite all entry point functions and everything in the middle as async functions, but then python -m my_library.main breaks because "main wasn't awaited".

The library's structure is something like this:

main.py

def handle()
if args["async_mode"]:
asyncio.run(async_handle)
else:
regular_handle()

def main()
handle()


And ideally I would want to call it from a notebook like this:

from my_library.main import main

main()


How should I proceed?

Continue reading...