[Python] How to correctly pair elements from two XCom lists in Airflow...

I am using Apache Airflow and trying to trigger multiple DAGs from within another DAG using TriggerDagRunOperator.expand().

I have two lists being returned from an upstream task via XCom:

confs → A list of dictionaries containing conf parameters.

dags_to_trigger → A list of DAG IDs to be triggered.

Each list contains 5 elements, and I want to pair them one-to-one (i.e., the first element of confs should be used with the first element of dags_to_trigger, the second with the second, and so on).

Problem:

When I use expand() like this:

trigger_my_dags = TriggerDagRunOperator.partial(
task_id="trigger_my_dags",
wait_for_completion=False,
).expand(conf=confs, trigger_dag_id=dags_to_trigger)


Airflow cross-pairs the elements, triggering 25 DAG runs instead of 5 (it takes each conf from description_sources and pairs it with every DAG ID from dags_to_trigger, rather than pairing them by index).

Question:

How can I ensure that Airflow correctly pairs the elements from both lists so that each DAG run gets the corresponding conf from description_sources?

Would restructuring the data into a single list of dictionaries like this help?

[{"conf": { ... }, "trigger_dag_id": "dag_1"}, {"conf": { ... }, "trigger_dag_id": "dag_2"}, ...]


If so, how do I correctly use expand() with this structure?

Thanks in advance!

Continue reading...