I'm encountering an issue with my Airflow DAGs in Google Cloud Composer after trying to fix an import error. Context: I have created several DAGs for my Data warehouse (DW) implemented in Google cloud Platform (GCP). Two of them are customized and they use the class BigQueryCreateEmptyTableOperator. Everytime I want to commit changes from my Github to my DW in GCP (g.e. for SQL queries), actions from Github Actions do work, but everytime I go to Composer in GCP I can see that DAGs are not being updated due to this error: ImportError: cannot import name 'BigQueryCreateEmptyTableOperator' from 'airflow.providers.google.cloud.operators.bigquery' What I've tried? Based on the error and some research, I understood that BigQueryCreateEmptyTableOperator has been replaced by BigQueryCreateTableOperator. I updated my DAG code by changing the import statement and the operator itself. After deploying this change, the ImportError disappeared, but now all my DAGs have vanished from the Airflow UI. However, I can confirm that the DAG python files still exist in the correct GCS bucket folder (/dags). I've checked the Airflow Scheduler logs, but I'm not seeing any obvious parsing errors that would explain why the DAGs are not being registered. My Questions: Why would changing BigQueryCreateEmptyTableOperator to BigQueryCreateTableOperator cause all DAGs to disappear from the UI, instead of just showing a specific error in the affected DAG? What is the correct way to perform this operator replacement to resolve the initial ImportError without causing this new issue? Could there be a problem with how the new operator is instantiated that causes a global parsing failure? Any help would be greatly appreciated. Continue reading...
