I'm building a Python project in which I need to create instances of many different HTTP clients with diferent cookies, headers and proxies. For that, I decided to use HTTPX AsyncClient. However, when testing a few things, I noticed that it takes so long for a client to be created (both AsyncClient and Client). I wrote a little code to validate this, and here it is: import httpx import time if __name__ == '__main__': total_clients = 10 start_time = time.time() clients = [httpx.AsyncClient() for i in range(0, total_clients)] end_time = time.time() print(f'{total_clients} httpx clients were created in {(end_time - start_time):.2f} seconds.') When running it, I got the following results: 1 httpx clients were created in 0.33 seconds. 5 httpx clients were created in 1.35 seconds. 10 httpx clients were created in 2.62 seconds. 100 httpx clients were created in 25.11 seconds. In my project scenario, I'm gonna need to create thousands of AsyncClient objects, and the time it would take to create all of it isn't viable. Does anyone know a solution for this problem? I considered using aiohttp but there's a few features that HTTPX has that AioHTTP doesn't. Continue reading...
