[Python] How to aggregate results from parallel tasks and store only the final result (ignore...

I'm using Celery to execute a workflow comprised of multiple parallel tasks. Here’s the pattern I want:

1. Dispatch multiple tasks in parallel.
2. Aggregate all their results into a final result.
3. Store only the aggregated result in the result backend—not the intermediate outputs from each individual task.

What I’ve tried so far


Using chord(group)(callback) seems ideal for aggregating results, but it obviously stores the intermediate results. So I've tried setting ignore_result=True, which causes infinite retries (which is pretty much expected).

What I need help with

• Is using a chord still the right approach here?
• Can I prevent storing intermediate results when using a chord, and only keep the final aggregated result?
• If a chord isn't suitable, what alternative workflow patterns (chains, groups, callbacks, etc.) achieve this cleanly?

Example sketch of what I want


from celery import group, chord

@app.task
def worker_task(arg):
return compute_something(arg)

@app.task
def aggregate(results):
return sum(results)

# Desired invocation:
chord(group(worker_task.s(arg) for arg in inputs))(aggregate.s())

Continue reading...