I've found myself in a tricky situation which I can't seem to find my way out of: I've written an application which uses asyncio (and, in particular, aiohttp). In part of this application, I need to use a third-party library (in my case, weasyprint) which is not async. I want to pass a callback (here, a url fetcher) into this third-party library. This callback must obviously be sync, but I want to use my existing async code as part of this callback. (I already have an aiohttp client session I want to use, and I've implemented some client middlewares in this session which do some useful things. I don't want to have to write a second fully sync version of this code.) I've tried: Creating a second event loop (as a new thread) inside the callback, and running my async url fetcher code in this second loop with loop.run_until_complete. I then used a queue.Queue to block execution on the main thread until the secondary thread with the second event loop returned a result. Unfortunately, this threw errors due to trying to run tasks in the wrong event loop. It seems I can't use the client session outside the event loop where it was created. Running weasyprint itself in a thread (using asyncio.to_thread) and then trying to use loop.run_until_complete in the callback (with loop here referencing the single main event loop). Unfortunately, this failed because "this event loop is already running." It didn't seem to matter that it was only awaiting the weasyprint thread, it still wouldn't let me run another coroutine in the loop. I do understand that, obviously, weasyprint won't be able to take advantage of the performance benefits of using async code, but simply for the sake of not having to write the same code twice (a sync version and an async version) I'd like to find a way to force it to use an async callback, even though that callback in reality just runs synchronously. I really only have a surface-level understanding of asyncio; I don't know a lot of the lower-level details about how event loops work and all that, so I'm sure the above approaches are a bit naive, but it seems to me like there ought to be some way to accomplish this. Continue reading...
