[Python] Celery SQS and S3 Missing bucket name exception

I have a celery, SQS config set up with FastAPI as follows

import os

from dotenv import load_dotenv

load_dotenv(".env")

result_backend = "db+sqlite:///results.sqlite"

result_persistent = True
task_serializer = "json"
result_serializer = "json"
accept_content = ["json"]
timezone = "Europe/London"
enable_utc = True
broker_connection_retry_on_startup = True
result_expires = 3600
task_track_started = True


SQS_QUEUE_URL = os.environ.get("SQS_QUEUE_URL")
SQS_QUEUE_NAME = os.environ.get("SQS_QUEUE_NAME")

broker_url = "sqs://"
broker_transport_options = {
"predefined_queues": {
SQS_QUEUE_NAME: {"url": SQS_QUEUE_URL},
},
}
task_default_queue = SQS_QUEUE_NAME
task_queues = {
SQS_QUEUE_NAME: {
"exchange": "celery",
"routing_key": "celery",
},
}

imports = ("internals.task_scheduler",)


The celery app is setup as

from celery import Celery

celery_app = Celery()

celery_app.config_from_object("config.celeryconfig")


if __name__ == "__main__":
celery_app.start()


everything works fine when I was using sqlite as result_backend.

The problem started when I changed the result_backend to S3 Bucket.

S3_BUCKET_NAME = os.environ.get("S3_BUCKET_NAME")

result_backend = "s3://{}/".format(S3_BUCKET_NAME)

result_backend_transport_options = {
"bucket_name": S3_BUCKET_NAME,
"key_prefix": "tmp/",
}


I get the exception

...raise ImproperlyConfigured('Missing bucket name')
celery.exceptions.ImproperlyConfigured: Missing bucket name


I am not sure what I was doing wrong. I have tried using one of result_backend or result_backend_transport_options. I confirmed that the bucket_name is loaded correctly from the enviroment using print statement and there is no mistake in the spelling. I have even hardcoded the bucket name still no headway. I can seems to find any help in the documentation or git repo for celery.

Continue reading...