I have three FastAPI services (collection, sync, auth) and each one of them has a separate celery instance, queues and workers. Each time we add a new microservice, we need to have a separate instance of celery and it's getting difficult to manage. I want to have a shared celery app lets call it shared-celery-app service, which has three queues (high_volume, resource_intensive, long_running) and it'll have three different workers one for each queue. I want to be able to execute tasks defined in the three services above but using the shared-celery-app workers. I've done my research are there are two options: Install the three FastAPI apps as python packages in my shared-celery-app python app. I'll import the tasks and register them in my shared-celery-app. This issue here is that i'll have to install from a git repo e.g. the main branch, but when it deploys to dev I want it to use the dev branch of the above three repos, which is not possible I can copy the tasks from the three services into shared-celery-app. The issue with this approach is that i'll end up with a lot of duplicate code e.g. models, schema, utils etc. I'd appreciate any help, thank you in advance. Continue reading...
