[Python] Airflow BranchPythonOperator works incorrectly when called in the following ways in...

I implemented the following pipeline:

DAG Graph

As seen from the graph, "tracker" and "tracker_second" are BranchPythonOperators. I am passing them a list of tasks that need to be executed via Xcom (this is done through the same Xcom!):

BranchPythonOperator Tasks

I need to run specific tasks, the names of which are passed through the Xcom variable manual_run_task_list. Moreover, tasks from the first and second groups should not overlap.

During testing, I discovered strange behavior from the BranchPythonOperator. I passed the following parameters:

XCom

Three tasks from the first group and three tasks from the second group should run, but in practice, the outcome is different:

Graph-Result

While investigating Xcom, I found that "tracker_second" had an unclear result for the parameter skipmixin_key:

Xcom variables

It ran all tasks, not just the specific ones like "tracker" did.

Can you please help me understand the reason for this behavior?

Apache-Airflow Version: 2.10.4

Solution

The solution to this problem is to use a separate manual_run_task_list for each group of tasks instead of one for all tasks. That is, for "tracker," you would have one list of tasks from the first group, and for "tracker_second," you would have a second list.

However, this seems to complicate the structure of the pipeline, so I would like to avoid that.

Continue reading...