[Python] Performance results differ between run_in_threadpool() and run_in_executor() in...

Here's a minimal reproducible example of my FastAPI app. I have a strange behavior and I'm not sure I understand the reason.

I'm using ApacheBench (ab) to send multiple requests as follows:

ab -n 1000 -c 50 -H 'accept: application/json' -H 'x-data-origin: source' 'http://localhost:8001/test/async'


FastAPI app

import time
import asyncio
import enum
from typing import Any

from fastapi import FastAPI, Path, Body
from starlette.concurrency import run_in_threadpool

app = FastAPI()
loop = asyncio.get_running_loop()
def sync_func() -> None:
time.sleep(3)
print("sync func")

async def sync_async_with_fastapi_thread() -> None:
await run_in_threadpool( time.sleep, 3)
print("sync async with fastapi thread")

async def sync_async_func() -> None:
await loop.run_in_executor(None, time.sleep, 3)

async def async_func() -> Any:
await asyncio.sleep(3)
print("async func")

@app.get("/test/sync")
def test_sync() -> None:
sync_func()
print("sync")

@app.get("/test/async")
async def test_async() -> None:
await async_func()
print("async")

@app.get("/test/sync_async")
async def test_sync_async() -> None:
await sync_async_func()
print("sync async")

@app.get("/test/sync_async_fastapi")
async def test_sync_async_with_fastapi_thread() -> None:
await sync_async_with_fastapi_thread()
print("sync async with fastapi thread")


Here's the ApacheBench results:

async with (asyncio.sleep) : *Concurrency Level: 50

• Time taken for tests: 63.528 seconds
• Complete requests: 1000
• Failed requests: 0
• Total transferred: 128000 bytes
• HTML transferred: 4000 bytes
• Requests per second: 15.74 [#/sec] (mean)
• Time per request: 3176.407 [ms] (mean)
• Time per request: 63.528 [ms] (mean, across all concurrent requests) Transfer rate: 1.97 [Kbytes/sec] received*

sync (with time.sleep): Concurrency Level: 50

• *Time taken for tests: 78.615 seconds
• Complete requests: 1000
• Failed requests: 0
• Total transferred: 128000 bytes
• HTML transferred: 4000 bytes
• Requests per second: 12.72 [#/sec] (mean)
• Time per request: 3930.751 [ms] (mean)
• Time per request: 78.615 [ms] (mean, across all concurrent requests) Transfer rate: 1.59 [Kbytes/sec] received*

sync_async (time sleep with run_in_executor) : *Concurrency Level: 50

• Time taken for tests: 256.201 seconds
• Complete requests: 1000
• Failed requests: 0
• Total transferred: 128000 bytes
• HTML transferred: 4000 bytes
• Requests per second: 3.90 [#/sec] (mean)
• Time per request: 12810.038 [ms] (mean)
• Time per request: 256.201 [ms] (mean, across all concurrent requests) Transfer rate: 0.49 [Kbytes/sec] received*

sync_async_fastapi (time sleep with run_in threadpool): *Concurrency Level: 50

• Time taken for tests: 78.877 seconds
• Complete requests: 1000
• Failed requests: 0
• Total transferred: 128000 bytes
• HTML transferred: 4000 bytes
• Requests per second: 12.68 [#/sec] (mean)
• Time per request: 3943.841 [ms] (mean)
• Time per request: 78.877 [ms] (mean, across all concurrent requests) Transfer rate: 1.58 [Kbytes/sec] received*

In conclusion, I'm experiencing a surprising disparity in results; especially, when using run_in_executor, where I'm encountering significantly higher average times (12 seconds). I don't understand this outcome.

--- EDIT --- After AKX answer.

Here the code working as expected:
import time
import asyncio
from anyio import to_thread

to_thread.current_default_thread_limiter().total_tokens = 200
loop = asyncio.get_running_loop()
executor = ThreadPoolExecutor(max_workers=100)
def sync_func() -> None:
time.sleep(3)
print("sync func")

async def sync_async_with_fastapi_thread() -> None:
await run_in_threadpool( time.sleep, 3)
print("sync async with fastapi thread")

async def sync_async_func() -> None:
await loop.run_in_executor(executor, time.sleep, 3)

async def async_func() -> Any:
await asyncio.sleep(3)
print("async func")

@app.get("/test/sync")
def test_sync() -> None:
sync_func()
print("sync")

@app.get("/test/async")
async def test_async() -> None:
await async_func()
print("async")

@app.get("/test/sync_async")
async def test_sync_async() -> None:
await sync_async_func()
print("sync async")

@app.get("/test/sync_async_fastapi")
async def test_sync_async_with_fastapi_thread() -> None:
await sync_async_with_fastapi_thread()
print("sync async with fastapi thread")

Continue reading...