[Python] Trigger DBT core jobs for Snowflake using Airflow

Airflow is installed using Docker and it is running fine. Now I am trying to add dbt-core, dbt-snowflake and astronomer-cosmos python packages to the Airflow image in order to run the DBT core jobs using Airflow.

I am trying to add the Python libraries using requirements.txt file. The contents of the requirements.txt file are:

astronomer-cosmos==1.9.0
dbt-core==1.9.0
dbt-snowflake==1.9.0
psycopg2-binary==2.9.9


In order to replicate my issue: The contents of docker-compose.yaml are:

version: '3.8'

x-airflow-common:
&airflow-common
build:
context: .
dockerfile: Dockerfile
env_file: .env
environment:
AIRFLOW__CORE__EXECUTOR: LocalExecutor
AIRFLOW__CORE__FERNET_KEY: ''
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOW__API__AUTH_BACKENDS: 'airflow.api.auth.backend.basic_auth'
AIRFLOW__WEBSERVER__SECRET_KEY: 'mysecretkey'
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- ./plugins:/opt/airflow/plugins
- ./dbt:/opt/airflow/dbt

services:
postgres:
image: postgres:15
environment:
POSTGRES_USER: airflow
POSTGRES_PASSWORD: airflow
POSTGRES_DB: airflow
volumes:
- postgres-db-volume:/var/lib/postgresql/data

airflow-webserver:
<<: *airflow-common
command: webserver
ports:
- "8080:8080"
depends_on:
- postgres

airflow-scheduler:
<<: *airflow-common
command: scheduler
depends_on:
- postgres

airflow-init:
<<: *airflow-common
command: >
bash -c "airflow db migrate &&
airflow users create --username airflow --password airflow --firstname Air --lastname Flow --role Admin --email airflow@example.com"
depends_on:
- postgres
volumes:
- ./wait-for-postgres.sh:/wait-for-postgres.sh




volumes:
postgres-db-volume:


The contents of the Dockerfile are:

FROM apache/airflow:2.10.5-python3.9

USER root

RUN apt-get update && apt-get install -y \
build-essential \
unixodbc-dev \
gcc \
g++ \
curl \
&& apt-get clean

USER airflow

COPY requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt


The contents of the wait-for-postgres.sh are:

#!/bin/bash
set -e

host="$1"
shift
cmd="$@"

until pg_isready -h "$host" -p 5432 -U airflow; do
>&2 echo "Postgres is unavailable - sleeping"
sleep 2
done

>&2 echo "Postgres is up - executing command"
exec $cmd


Now I am doing trial and error with the version of the astronomer-cosmos, dbt-core and dbt-snowflake if I increase the version to the latest or decrease the version to the older ones, it's breaking some where.

The error is:

File "/home/airflow/.local/lib/python3.9/site-packages/cosmos/dbt/runner.py", line 109, in handle_exception_if_needed
raise CosmosDbtRunError(f"dbt invocation did not complete with unhandled error: {result.exception}")
cosmos.exceptions.CosmosDbtRunError: dbt invocation did not complete with unhandled error: No such option: --no-static-parser
[2025-08-24, 07:19:24 UTC] {taskinstance.py:1226} INFO - Marking task as FAILED. dag_id=dev_test_dag, task_id=dev_test_dag.example_1_run,


The other error is:

The conflict is caused by:
1404.9 dbt-core 1.9.0 depends on protobuf<6.0 and >=5.0
1404.9 dbt-adapters 1.16.5 depends on protobuf<7.0 and >=6.0
1404.9 dbt-core 1.9.0 depends on protobuf<6.0 and >=5.0
1404.9 dbt-adapters 1.16.4 depends on protobuf<7.0 and >=6.0
1404.9
1404.9 To fix this you could try to:
1404.9 1. loosen the range of package versions you've specified
1404.9 2. remove package versions to allow pip to attempt to solve the dependency conflict


Now I need to know which versions should I use in the requirements.txt file?

Continue reading...