[Python] When status of Airflow Neptune task is `LOAD_IN_QUEUE` Airflow tries to retry

I am work in multi-tenant data pipeline platform. So, we have about 5 tenants for now. We use AWS MWAA (Apache Airflow) as orchestration tool. Each tenant has separate DAG and run serially in order to get rid of overload in MWAA. Now we need to enable parallel execution of the DAGs in Airflow. In order to complete the Data pipeline run in bounded time.

In each data pipeline we are using more than 10 tasks, In that we have AWS NeptuneDB loader module. As we already enabled queue_request = True, I ran Airflow DAG's parallel manner. But, which didn't worked as we expected.

When 1st DAG’s Neptune process is running. 2nd DAG’s Neptune process’s status will be LOAD_IN_QUEUE. While getting this status the Airflow DAG consider this as failure and waits for retry. When the 1st DAG’s Neptune process is completed. Then, the 2nd DAG’s Neptune process starts when the retry comes.

Now, I'm expecting the DAG to wait until the 1st DAG's Neptune process completed without retrying. When the STATUS=LOAD_IN_QUEUE Airflow treats this as error so it tries for retry.

So, kindly add your inputs on this

Continue reading...