[Python] Processing requests in FastAPI sequentially while staying responsive

My server exposes an API for a resource-intensive rendering work. The job it does involves a GPU and as such the server can handle only a single request at a time. Client should submit a job and receive 201 - ACCEPTED - as a response immediately after. The processing can take up to a minute and there can be a few dozens of requests scheduled.

Here's what I came up with, boiled to a minimal reproducible example:

import time
import asyncio
from fastapi import FastAPI, status


app = FastAPI()
fifo_queue = asyncio.Queue()


async def process_requests():
while True:
name = await fifo_queue.get() # Wait for a request from the queue
print(name)
time.sleep(10) # A RESOURCE INTENSIVE JOB THAT BLOCKS THE THREAD
fifo_queue.task_done() # Indicate that the request has been processed


@app.on_event("startup")
async def startup_event():
asyncio.create_task(process_requests()) # Start the request processing task


@app.get("/render")
async def render(name):
fifo_queue.put_nowait(name) # Add the request parameter to the queue
return status.HTTP_201_CREATED # Return a 201 status code


The problem with this approach is that the server does not stay responsive. After sending the first request it gets busy full time with it and does not respond as I have hoped.

curl http://127.0.0.1:8000/render\?name\=001


In this example simply replacing time.sleep(10) with await asyncio.sleep(10) solves the problem, but not in the real use case (though possibly offers a clue as for what I am doing incorrectly).

Any ideas?

Continue reading...