[Python] structlog with Celery

I have a working celery app. I want to add structured logging to it.

A complete working example would be hard to provide, so let me demostrate:

import structlog
import logging

logging.config.dictConfig(...)
structlog.configure(...)

app = Celery()


Then, in some task:

logger = structlog.getLogger()

logger.info("Foo")


In this app, when there are logs outside Celery tasks, they print out in a JSON format - like I want. However, each log from a Celery task has this:

[2024-04-25 20:37:14,305: INFO/ForkPoolWorker-2] {...}


[2024-04-25 20:37:14,305: INFO/MainProcess] {...}


prefix. And those logs don't have keys that should be here (due to my logging configuration), so that suggests that the logging isn't "configured" inside Celery tasks.

My question is: how do I make Celery tasks log how I want them to? Could I maybe move those two logging.config.dictConfig and structlog.configure calls to a separate function (let's say initialize_logging) and add it as a parameter or something to Celery so it executes it at the beggining of every task or something like this?

Continue reading...