I'm running a custom FastAPI-Docker Image (more specifically the python3.7 one) on an Azure App Service (2 Cores, 3.5 GB Memory, OS: Linux). The image is hosted using an Azure Container Registry. The container serves requests which have a PDF-File attached, processes the PDF-file and returns the result to a website. A user can then verify that the result is either correct or not correct. In case that the result is correct, the website then sends a second POST-request to the FastAPI-Service to store the results in an AtlasDB. Initially, when the Azure App Service has just been started, everything works fine. However, after a while the documents could not be uploaded using the website anymore. The service crashes with the following error: 2020-11-02T14:21:40.225130726Z [2020-11-02 14:21:40 +0000] 1 [CRITICAL] WORKER TIMEOUT (pid:9) This, however, only happens, after some documents have been uploaded. The amount of time it takes until the error is raised is not deterministic. Sometimes it is about 90 seconds and sometimes it takes up to 3 minutes until the timeout is reached. Other documents that can be processed in under 90 seconds are still processed just fine. At first, I thought that it has to do with the gunicorn workers. I created a custom gunicorn_conf.py (using the standard config) and increased the WORKERS_PER_CORE to 3 as well as the GRACEFUL_TIMEOUT and TIMEOUT to 180 seconds. But the error still remains. Additionally, I upscaled the App-Service plan to see, whether more computing power would solve the problem. Unfortunately, it does not. Whenever I start the docker-image locally everything works flawlessly. My guess is, that the error still is gunicorn-worker related, but I couldn't find any solution to the problem. I also downloaded the App Service Logs and Container Logs but there isn't any further information or hint on why the worker crashes. Does anyone have some advice on how to find the issue? Thank you in advance and have a nice day! Continue reading...
