[Python] Docker Using Airflow and Conda

I have a Conda environment that I am using to create a couple of DAGS in Airflow. Much of what I am working with is in a directory above Airflow, similar to this:

.
├── environment.yml
├── airflow
│ ├── dags
│ ├── Dockerfile
│ ├── docker-compose.yml
│ └── config
└── my_module
├── __init__.py
└── file1.py


My plan is to use Taskflow API using functions I made in the my_module directory into the dags folder since there are going to be other functions in this module.

I've been asking GPT a lot to help write the Dockerfile but it's not super helpful anymore, here's what I'm working with below:

# Stage 1: Build stage using Miniconda image
FROM continuumio/miniconda3:latest AS miniconda_stage

# Set working directory inside the container
WORKDIR /opt/airflow

# Install dependencies using Conda
COPY ../environment.yml /tmp/environment.yml
RUN conda env create -f /tmp/environment.yml

# Stage 2: Final stage using the Airflow image
FROM apache/airflow:2.10.5 AS airflow_stage

# Set working directory inside the container
WORKDIR /opt/airflow

# Copy the Conda environment from the first stage
COPY --from=miniconda_stage /opt/conda /opt/conda

# Activate the Conda environment
ENV PATH="/opt/conda/envs/readme/bin:$PATH"

# Copy the custom code and DAGs into the container
COPY ../codebase /opt/codebase
COPY ./airflow/dags /opt/airflow/dags

# Set the PYTHONPATH
ENV PYTHONPATH="/opt/codebase:$PYTHONPATH"

# Ensure the correct permissions
# RUN chmod -R 777 /opt/airflow/dags /opt/codebase

# Start the Airflow webserver and scheduler
CMD ["bash", "-c", "airflow db init && airflow webserver & airflow scheduler"]


I'm using the docker-compose.yml found on the Airflow site by running this command: curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.10.5/docker-compose.yaml'

I modified the yaml file to include the build, changed the context, and mounted volumes. It works whenever I uncomment the image: line, but the environment is loaded like it should be in the Dockerfile. Modified parts of the yaml file:

x-airflow-common:
&airflow-common-env
# image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.10.5}
build:
context: ..
dockerfile: ./airflow/Dockerfile
...
volumes:
- ${AIRFLOW_PROJ_DIR:-.}/dags:/opt/airflow/dags
- ${AIRFLOW_PROJ_DIR:-.}/logs:/opt/airflow/logs
- ${AIRFLOW_PROJ_DIR:-.}/config:/opt/airflow/config
- ${AIRFLOW_PROJ_DIR:-.}/plugins:/opt/airflow/plugins
- ${AIRFLOW_PROJ_DIR:-.}/../codebase:/opt/codebase


So the first issue is that it takes a while to build the image, and from what I understand it's a multi-stage build. The second issue is that I get an error saying that /endpoint isn't found with exit code 1.

I still feel very new to Docker and any help/guidance would be much appreciated!!!

Continue reading...