[Python] Gunicorn / Uvicorn extremely rare 502 Errors

We observed an issue with our containerized Python FastAPI application, which we deploy using Gunicorn and Uvicorn on Kubernetes. In extremely rare cases (~1 in 100.000) a single request seem to be silently dropped / not correctly processed by the the gunicorn master process or the uvicorn workers. My nginx reverse proxy outputs the following error message:

[error] 26#26: *504921 connect() failed (111: Connection refused) while connecting to upstream, client: 10.95.59.1, server: , request: "POST /v1/info HTTP/1.1", upstream: "http://10.96.229.82:8000/v1/info", host: "our-internal-host.com:443"
"POST /v1/info HTTP/1.1" 502


The problem is this specific request is not visible in the application logs. With debug logs activated neither gunicorn nor uvicorn logs the request at all. We can not observe any pattern when this happens (e.g. it happens during the day with high load ~ 300 req/min as well as during the night were we have only ~20 req/min). It happens on every endpoint we have, so I'd assume it has also nothing to do with the business logic inside the endpoint. Furthermore, I am not able to artificially reproduce this error in our DEV stages. If I execute a loadtest and throw 500.000 requests at the service everything seems to be fine (either from locally or from inside our Kubernetes cluster).

This is my gunicorn config (redacted the unimportant stuff):

from apscheduler.schedulers.background import BackgroundScheduler

loglevel = "info" if stage == "PROD" else "debug"
bind = "0.0.0.0:8000"

worker_class = "uvicorn.workers.UvicornWorker"
workers = 4
timeout = 180
graceful_timeout = 30

accesslog = "-"
errorlog = "-"
access_log_format = '%(p)s %(t)s "%(r)s" %(s)s %(D)s'

pidfile = "/tmp/app.pid"

def on_starting(server):
scheduler = BackgroundScheduler()
scheduler.add_job(call_liveness_probe, "cron", minute="*/2", start_date=datetime.now() + timedelta(minutes=5))
scheduler.start()



We used to also have the max_requests-parameter activated to prevent memory leaks and restart workers every 5000 requests. This resulted in similar undetected and unlogged errors. The default which is now used is set to disabled.

Dockerfile using a RHEL9 Base Image (only important stuff):

ARG BASE_IMAGE
FROM ${BASE_IMAGE}

USER 1001

ENV PYTHONUNBUFFERED=1

CMD gunicorn -c /opt/app-root/config/gunicorn_config.py app.main:app


We use Python 3.11 (also happens with 3.12) and the following packages are installed

asyncpg==0.30.0
aiohttp[speedups]==3.11.*
apscheduler==3.11.*
autodynatrace==2.1.0
fastapi==0.115.*
gunicorn==23.0.*
orjson==3.10.*
numpy==2.2.*
packaging==23.2
pandas==2.2.*
pydantic==2.10.*
pyopenssl==25.0.*
requests==2.32.*
uvicorn[standard]==0.34.*


Our endpoints are mostly async (I/O on DB operations), our average response times are ~20ms and always lay in between 3ms - 250ms. We use fastapi BackgroundTasks to write data to the DB, so requests are already returned and do not have to wait for some DB writes to finish. I also tried to increase the uvicorn workers to 12 or decrease to 1. Neither solution mitigated or increased/decreased the error rate.

Does anybody have an idea what the cause could be or how to better debug it? (I don't want to do a TCP Dump in production and can't reproduce it consistently (or for that matter at all) myself and have to wait potentially weeks for it to happen on our DEV stages. This makes it really hard in addition to the fact that gunicorn and uvicorn do not log that request at all)

Continue reading...