I am currently using Airflow to run some SQL tasks on Clickhouse. Airflow is run within Docker containers on a Compute instance in GCP, and the tasks are generally provisioned using the Airflow ClickhouseOperator. At different instances, the Airflow task states for tasks which may, or may not, have run would be cleared and the webserver UI would look as in the image below. As a result, no further task gets run. This always happens with more recent tasks (the most recent dag run till 2 - 10 runs before it). It never happens with older dag runs, while ignoring the more recent ones, an in all occurrences, all task run states on the dag run get cleared. I have often just deleted the dag runs with cleared states, and have them all run again, but this generally has produced some data inconsistencies, since some of those runs already finished successfully at their scheduled time, and for some, the runs may have stopped somewhere along the way, which will not be known. Any ideas on how to actually get the root cause and prevent future occurrences? Airflow Version: 2.6.3 Python Version: 3.7 Continue reading...
