[Python] Airflow EMRServerlessCreateApplicationOpertor can't detect application name from...

I am trying to create Airflow DAG for EMR Serverless application creation.

EMRServerlessCreateApplicationOperator(
task_id = "create-emrs-app",
job_type = "SPARK",
release_label = "emr-7.1.0",
config= {
"name" : "{{dag_run.conf['application_name']}}",
...other config
},
dag = main_dag
)


However DAG is failing with validation exception saying 'name' is not matching the regex pattern.

On troubleshooting I have found that 'application_name' is not getting rendered. It is trying to create an EMRS application using name "{{dag_run.conf['application_name']}}"

To further debug if jinja template rendering is working or not, I have create one more task in the same DAG using PythonOperator and print the application_name in python callable. This task is working fine and printing the name.

PythonOperator(
task_id = "log_application_name",
python_callable = print_app_name,
op_kwards = {application_name : "{{dag_run.conf['application_name']}}" },
dag = main_dag
)


I have also checked the Airflow UI 'Rendered Template' for both tasks. For PythonOperator task, template rendering is showing in UI however for EMR application creation task, 'Rendered Template' section is empty.

So any idea why jinja template rendering is not working for EMRServerlessCreateApplicationOperator.

I am using airflow 2 and amazon provider package version 8.5

Continue reading...