[Python] What could cause strange delays while sending delays from a python aiohttp server?

We have a service with this architecture:

• HTTPS requests come into an A10 load balancer that does L4 load balancing
• Behind it are 2 backend servers with Apache running that terminate the TLS connection
• In Apache there is a ProxyPass rule that talks to a http service on localhost
• this local service uses gunicorn and is implemented in Python using aiohttp (Python 3.11.2 running on Debian Linux ("bookworm") running as a VMWare VM)

Now we have some cases where users reported timeouts. Both the python application and Apache write logs, and so we've traced some of the timeouts to a weird delay. The python application logs a message right before it passes it response to gunicorn (and thus to Apache), and sometimes the time between this log and the Apache access log is really long, like 300s.

This is a somewhat rare occurrence, in a day where we process about 880k requests there are just 17 requests where the delay is 30s or more, which makes this kinda hard to debug.

Capturing all the network traffic in a huge .pcap file and then sifting through it is kinda hopeless, far too much data, and the responses from Apache are encrypted, so it makes it really hard to trace request IDs in the pcap files.

Most of the requests with delays have a response size of a few kilobytes to a few megabytes, though very seldomly we also see slow requests with response body <1kb, which should fit into a single TCP package.

Does anybody have a good hypothesis where this delay could come from, and how I could debug it?

Continue reading...