[Python] How to run an Airflow DAG with different parameters across different schedules?

I have a DAG that runs a python file (like -- python /path/to/file.py -a -b -c -d) and there are multiple parameters that are to be passed in the command that runs this file. Now, I want to run this same file at different times of the day (the schedule interval looks like this - '0 3,5,7,9,11,13,15,17,19,21 * * *') and each of the schedules has different parameters. What is/are the best way(s) to maintain & implement this kind of a setup?

The first and most basic solution is to create 10 different DAGs with the respective params in the command itself. However, this method would be similar to cronjobs and will limit the reusability of the DAG also.

The second solution which I found was to use the TriggerDagRunOperator. There will be one main DAG to run the python file that I want which will be triggered by other DAGs using the TriggerDagRunOperator to pass the parameters for different schedules. This improves the reusability and makes DAG maintenance easier but I would still have to create as many number of DAGs as the number of different parameters.

Is there any other way which is more efficient?

Continue reading...