[Python] Using Celery with RabbitMQ as broker vs using just RabbitMQ + Pika for async tasks,...

The debate I am in currently is whether we should stick with RabbitMQ implementation using Pika or move to celery, what all advantages are there if we go with Celery. From what I have understood is Celery is a distributed job queue that simplifies the management of task distribution. It uses broker (RabbitMQ, Redis and so on) for the sending and receiving the message between client and worker, it also can optionally use backend such as Redis to store the results. Where as RabbitMQ is a message Queue which can be used to perform the Jobs in async manner. Eventually if we use either RabbitMQ and implement it using Pika in python it will do the same Job which is to execute long running processes in background.

The few advantages that I see in using Celery are:

• Can store result of each task, using backend (such as redis).
• Easier to implement.
• Also allows to add retries.
• Is a distributed job queue, can be run on multiple nodes/clusters.
• Packages like flower can be used to monitor each task, their states, results, time taken and some other metadata too.
• Task chaining

But on the other side it seems it does restrict us to use some of the features of RabbitMQ and also it has some limitations like it will connect with broker synchronously (issue on github https://github.com/celery/celery/issues/3884 ) I am familiar with this Question already asked here Why use Celery instead of RabbitMQ? but it does not seem to be clear.

Any help would be highly appreciated.

Continue reading...