[Python] Celery + SQS: Worker Process Delay - Task Received but Execution Starts After 30+...

Celery Worker Delay: Task Received but Execution Starts After 30+ Seconds

Environment

• Python 3.9.7
• Celery 5.5.3
• kombu 5.5.4
• billiard 4.2.1
• AWS SQS as message broker
• Docker container running on m1 chip, (Rosetta VMM)

Issue


I'm experiencing a significant delay between task reception and execution in my Celery worker. The logs show that while the task is received immediately, there's a ~30-second gap before the actual execution begins.

From the logs:

-------------- celery@b85d00492b1e v5.5.3 (immunity) --- ***** ----- -- ******* ---- Linux-6.10.14-linuxkit-x86_64-with-glibc2.28 2025-06-15 16:24:19

• *** --- * ---
• ** ---------- [config]
• ** ---------- .> app: app:0x7ffffd337b80
• ** ---------- .> transport: sqs://sqs.eu-west-1.amazonaws.com/account_id/queue_name
• ** ---------- .> results:
• *** --- * --- .> concurrency: 5 (prefork) -- ******* ---- .> task events: ON --- ***** ----- -------------- [queues] .> celery exchange=celery(direct) key=celery

[tasks] . simple_task

[2025-06-15 16:24:20,684: INFO/MainProcess] Connected to sqs://sqs.eu-west-1.amazonaws.com/account_id/queue_name [2025-06-15 16:24:20,722: INFO/MainProcess] Found credentials in environment variables. [2025-06-15 16:24:21,055: INFO/MainProcess] Found credentials in environment variables. [2025-06-15 16:24:21,298: INFO/MainProcess] Found credentials in environment variables. [2025-06-15 16:24:21,560: INFO/MainProcess] Found credentials in environment variables. [2025-06-15 16:24:21,802: INFO/MainProcess] Found credentials in environment variables. [2025-06-15 16:24:22,062: INFO/MainProcess] celery@b85d00492b1e ready.

[2025-06-15 16:24:40,332: INFO/MainProcess] Task simple_task[message_id] received

[2025-06-15 16:26:01,441: WARNING/ForkPoolWorker-3] simple_task

[2025-06-15 16:26:01,448: WARNING/ForkPoolWorker-3] Start time:2025-06-15 16:26:01.438941

[2025-06-15 16:26:01,448: WARNING/ForkPoolWorker-3] End time:2025-06-15 16:26:01.448292

[2025-06-15 16:26:01,448: WARNING/ForkPoolWorker-3] Duration:0:00:00.009351

[2025-06-15 16:26:01,458: INFO/ForkPoolWorker-3] Task simple_task[message_id] succeeded in 0.11523487499653129s: None

Notice the ~50-second gap between reception and execution.

Configuration


# Django settings
CELERY_BROKER_URL = os.environ.get("CELERY_BROKER_URL", "sqs://sqs.eu-west-1.amazonaws.com/account_id/queue_name")
CELERY_RESULT_BACKEND = "django-db"
CELERY_BROKER_TRANSPORT_OPTIONS = {
"visibility_timeout": 3600,
"region": os.getenv("AWS_REGION", "eu-west-1")
}
CELERY_BROKER_TRANSPORT_OPTIONS["predefined_queues"] = {
'celery': {
'url': 'https://sqs.eu-west-1.amazonaws.com/account_id/queue_name'
}
}

Task Definition


@shared_task(name="simple_task")
def simple_task():
start_time = datetime.datetime.now()
print("simple_task", flush=True)
end_time = datetime.datetime.now()
print(start_time, end_time, end_time - start_time, flush=True)

Worker Command


celery -A hotdesk_backend.celery worker -l INFO -E

What I've Tried

1. Verified AWS credentials are properly configured (logs show "Found credentials in environment variables")
2. Confirmed SQS connection is successful
3. Checked that the task is being received by the worker
4. Verified the task execution itself is fast (only 0.009351 seconds)
5. celery cli tools/commands does not work with sqs, kombu.transport.SQS.UndefinedQueueException: Queue with name '83d87fa7-b21f-32fd-84e7-80a738fd87c8-reply-celery-pidbox' must be defined in 'predefined_queues'.

Questions

1. What could be causing this delay between task reception and execution?
2. Are there any SQS-specific configurations I should check?
3. Could this be related to the worker's concurrency settings (currently set to 5)?
4. Are there any known issues with Celery 5.5.3 and SQS that might cause this behavior?

Any insights or suggestions would be greatly appreciated!

Continue reading...