[Python] Stream Apache Airflow (3.0.2) Logs to Azure Blob

here I'm trying to stream (pushed) my airflow logs to azure blob instead of keeping it in container folder.

Current Issue that I'm facing: even after putting everything as mentioned in official document and following online threads, nothing works for my project.

current folder structure:

project_03/
├── config/
├── dags/
│ └── hello_world_dag.py
├── logs/
├── plugins/
├── requirements.txt
├── Dockerfile
├── .env
└── docker-compose.yml


DockerFile contain:

FROM apache/airflow:3.0.2
COPY requirements.txt /
RUN pip install --no-cache-dir -r /requirements.txt


here is my docker-compose.yml code:

x-airflow-common:
&airflow-common

build: .
environment:
&airflow-common-env
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__AUTH_MANAGER: airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__BROKER_URL: redis://redis:6379/0
AIRFLOW__CORE__FERNET_KEY: ''
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
AIRFLOW__CORE__EXECUTION_API_SERVER_URL: 'http://airflow-apiserver:8080/execution/'

AIRFLOW__SCHEDULER__ENABLE_HEALTH_CHECK: 'true'

_PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}

AIRFLOW__LOGGING__REMOTE_LOGGING: 'true'
AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: 'azure_blob_conn'
AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: 'wasb://airflow-logs@ccmytest.blob.core.windows.net/airflow/logs'
AIRFLOW_CONN_AZURE_BLOB_CONN: 'wasb://ccmytest:<account-key>@ccmytest.blob.core.windows.net'
AIRFLOW__LOGGING__LOGGING_LEVEL: 'INFO'

AIRFLOW_CONFIG: '/opt/airflow/config/airflow.cfg'


not sure what I'm missing but every other config that mentioned online I have put everything, but still it's not pushing the logs. if any workaround for this issue, that will be really helpful.

Continue reading...