I am facing a quite strange problem. Context : I am trying to build 2 Docker images to use in JupyuterHub using multistage : base base-sparkmagic, built from base When I add package psycopg2 to Python dependencies in image base, base-sparkmagic image do not build anymore. I get the error below : 40.36 Looking for: ["pandas[version='<2.0.0']", 'sparkmagic==0.22.0', <'ipywidgets==8.1.6', 'apache-sedona==1.7.1', 'pytz==2025.2', 'dateutils==0.6.12'] 40.36 40.36 40.36 Pinned packages: 40.36 - python 3.11.* 40.36 - python 3.11.4 40.36 40.36 40.36 Could not solve for environment specs 40.36 The following packages are incompatible 40.36 ├─ apache-sedona 1.7.1 does not exist (perhaps a typo or a missing channel); 40.36 ├─ ipywidgets 8.1.6 does not exist (perhaps a typo or a missing channel); 40.36 ├─ pytz 2025.2 does not exist (perhaps a typo or a missing channel); 40.36 └─ sparkmagic 0.22.0 does not exist (perhaps a typo or a missing channel). Dockerfile:164 163 | COPY --chown=${NB_UID} requirements-sparkmagic.txt /tmp/requirements-sparkmagic.txt 164 | >>> RUN mamba install -y --file /tmp/requirements-sparkmagic.txt && \ 165 | >>> mamba clean --all -f -y && \ 166 | >>> fix-permissions "${CONDA_DIR}" && \ 167 | >>> fix-permissions "/home/${NB_USER}" 168 | ERROR: failed to solve: process "/bin/bash -o pipefail -c mamba install -y --file /tmp/requirements-sparkmagic.txt && mamba clean --all -f -y && fix-permissions \"${CONDA_DIR}\" && fix-permissions \"/home/${NB_USER}\"" did not complete successfully: exit code: 1 I am quite confident that the problem comes from psycopg2 package because if I install kafka dependency package before psycopg2, image base builds, and if I install psycopg2 before kafka, the image do not build anymore. It does not seem to be related to libpq-dev apt package. Here is my Dockerfile : # Base image ARG REGISTRY=docker.io ARG OWNER=jupyter ARG BASE_CONTAINER=$REGISTRY/$OWNER/pyspark-notebook # --------------------------------------------------------------- # FROM $BASE_CONTAINER:spark-3.4.0 as base-linux FROM docker.io/jupyter/pyspark-notebook:spark-3.4.0 as base-linux # Fix: https://github.com/hadolint/hadolint/wiki/DL4006 # Fix: https://github.com/koalaman/shellcheck/wiki/SC3014 USER 0 ARG SCALA_VERSION="2.12" ARG MAVEN_REPO="https://repo1.maven.org/maven2" ARG HADOOP_VERSION_LIB="3.3.4" ARG AWS_JAVA_SDK_VERSION="1.12.262" ARG HADOOP_SUM="59907e790ce713441955015d79f670bc" ARG AWS_JAVA_SDK_SUM="8a22f2d30b7e8eee9ea44f04fb13b35a" ARG SPARK_HOME="/usr/local/spark" SHELL ["/bin/bash", "-o", "pipefail", "-c"] # Install paquets RUN apt-get update && \ apt-get install -y --no-install-recommends --assume-yes --allow-downgrades \ curl=7.81.0-1ubuntu1.20 \ # fix os vuln linux-libc-dev=5.15.0-138.148 \ libc-devtools=2.35-0ubuntu3.9 \ libc-bin=2.35-0ubuntu3.9 \ libc6=2.35-0ubuntu3.9 \ locales=2.35-0ubuntu3.9 \ libcurl3-gnutls=7.81.0-1ubuntu1.20 \ libnghttp2-14=1.43.0-1ubuntu0.2 \ libvpx7=1.11.0-2ubuntu2.3 \ openssh-client=1:8.9p1-3ubuntu0.13 \ && \ apt-get autoremove --yes && \ rm -rf /var/lib/apt/lists/* && \ apt-get clean -y USER ${NB_UID} # --------------------------------------------------------------- FROM base-linux as base-builder USER 0 ARG SCALA_VERSION="2.12" ARG MAVEN_REPO="https://repo1.maven.org/maven2" ARG HADOOP_VERSION_LIB="3.3.4" ARG AWS_JAVA_SDK_VERSION="1.12.262" ARG HADOOP_SUM="59907e790ce713441955015d79f670bc" ARG AWS_JAVA_SDK_SUM="8a22f2d30b7e8eee9ea44f04fb13b35a" ARG SPARK_HOME="/usr/local/spark" ARG WORK="/home/jovyan/work" ARG DELTA_CORE_VERSION="2.4.0" ARG DELTA_CORE_SUM="65b8dec752d4984b7958d644848e3978" ARG DELTA_STORAGE_VERSION="2.4.0" ARG DELTA_STORAGE_SUM="a83011a52c66e081d4f53a7dc5c9708a" SHELL ["/bin/bash", "-o", "pipefail", "-c"] # # Update certificate for Minio # COPY cert/HXD_PKI_CA_SubCA_1.crt /usr/local/share/ca-certificates/HXD_PKI_CA_SubCA_1.crt # RUN sudo update-ca-certificates # Install jars for Minio and Delta RUN curl -o "${SPARK_HOME}/jars/aws-java-sdk-bundle-${AWS_JAVA_SDK_VERSION}.jar" -C - "${MAVEN_REPO}/com/amazonaws/aws-java-sdk-bundle/${AWS_JAVA_SDK_VERSION}/aws-java-sdk-bundle-${AWS_JAVA_SDK_VERSION}.jar" && \ echo "Computed md5sum: $(md5sum "${SPARK_HOME}/jars/aws-java-sdk-bundle-${AWS_JAVA_SDK_VERSION}.jar")" && \ echo "${AWS_JAVA_SDK_SUM} ${SPARK_HOME}/jars/aws-java-sdk-bundle-${AWS_JAVA_SDK_VERSION}.jar" | md5sum -c - && \ ## Add Hadoop AWS curl -o "${SPARK_HOME}/jars/hadoop-aws-${HADOOP_VERSION_LIB}.jar" -C - "${MAVEN_REPO}/org/apache/hadoop/hadoop-aws/${HADOOP_VERSION_LIB}/hadoop-aws-${HADOOP_VERSION_LIB}.jar" && \ echo "Computed md5sum: $(md5sum "${SPARK_HOME}/jars/hadoop-aws-${HADOOP_VERSION_LIB}.jar")" && \ echo "${HADOOP_SUM} ${SPARK_HOME}/jars/hadoop-aws-${HADOOP_VERSION_LIB}.jar" | md5sum -c - && \ ## Add delta-core curl -o "${SPARK_HOME}/jars/delta-core_2.12-${DELTA_CORE_VERSION}.jar" -C - "${MAVEN_REPO}/io/delta/delta-core_2.12/${DELTA_CORE_VERSION}/delta-core_2.12-${DELTA_CORE_VERSION}.jar" && \ echo "Computed md5sum: $(md5sum "${SPARK_HOME}/jars/delta-core_2.12-${DELTA_CORE_VERSION}.jar")" && \ echo "${DELTA_CORE_SUM} ${SPARK_HOME}/jars/delta-core_2.12-${DELTA_CORE_VERSION}.jar" | md5sum -c - && \ ## Add delta-storage curl -o "${SPARK_HOME}/jars/delta-storage-${DELTA_STORAGE_VERSION}.jar" -C - "${MAVEN_REPO}/io/delta/delta-storage/${DELTA_STORAGE_VERSION}/delta-storage-${DELTA_STORAGE_VERSION}.jar" && \ echo "Computed md5sum: $(md5sum "${SPARK_HOME}/jars/delta-storage-${DELTA_STORAGE_VERSION}.jar")" && \ echo "${DELTA_STORAGE_SUM} ${SPARK_HOME}/jars/delta-storage-${DELTA_STORAGE_VERSION}.jar" | md5sum -c - USER ${NB_UID} # --------------------------------------------------------------- FROM base-linux as base USER 0 ARG SCALA_VERSION="2.12" ARG MAVEN_REPO="https://repo1.maven.org/maven2" ARG HADOOP_VERSION_LIB="3.3.4" ARG AWS_JAVA_SDK_VERSION="1.12.262" ARG HADOOP_SUM="59907e790ce713441955015d79f670bc" ARG AWS_JAVA_SDK_SUM="8a22f2d30b7e8eee9ea44f04fb13b35a" ARG SPARK_HOME="/usr/local/spark" ARG WORK="/home/jovyan/work" ARG DELTA_CORE_VERSION="2.4.0" ARG DELTA_STORAGE_VERSION="2.4.0" ENV PATH="$SPARK_HOME/bin:$SPARK_HOME/sbin:${PATH}" #ENV PYTHONPATH ="/usr/local/spark/python" #ENV PYSPARK_PYTHON="/usr/bin/python3" #ENV PYSPARK_DRIVER_PYTHON="/usr/bin/python3" COPY --chown="${NB_UID}:${NB_UID}" --from=base-builder ${WORK} ${WORK} COPY --from=base-builder "${SPARK_HOME}/jars/hadoop-aws-${HADOOP_VERSION_LIB}.jar" "${SPARK_HOME}/jars/hadoop-aws-${HADOOP_VERSION_LIB}.jar" COPY --from=base-builder "${SPARK_HOME}/jars/aws-java-sdk-bundle-${AWS_JAVA_SDK_VERSION}.jar" "${SPARK_HOME}/jars/aws-java-sdk-bundle-${AWS_JAVA_SDK_VERSION}.jar" COPY --from=base-builder "${SPARK_HOME}/jars/delta-core_2.12-${DELTA_CORE_VERSION}.jar" "${SPARK_HOME}/jars/delta-core_2.12-${DELTA_CORE_VERSION}.jar" COPY --from=base-builder "${SPARK_HOME}/jars/delta-storage-${DELTA_STORAGE_VERSION}.jar" "${SPARK_HOME}/jars/delta-storage-${DELTA_STORAGE_VERSION}.jar" # COPY --from=base-builder /usr/local/share/ca-certificates/HXD_PKI_CA_SubCA_1.crt /usr/local/share/ca-certificates/HXD_PKI_CA_SubCA_1.crt USER ${NB_UID} # Install additionnal packages. COPY --chown=${NB_UID} requirements.txt /tmp/requirements.txt RUN mamba install -y --file /tmp/requirements.txt && \ mamba clean --all -f -y && \ fix-permissions "${CONDA_DIR}" && \ fix-permissions "/home/${NB_USER}" COPY --chown=${NB_UID} kafka.txt /tmp/kafka.txt RUN mamba install -y --file /tmp/kafka.txt && \ mamba clean --all -f -y && \ fix-permissions "${CONDA_DIR}" && \ fix-permissions "/home/${NB_USER}" COPY --chown=${NB_UID} psycopg2.txt /tmp/psycopg2.txt RUN mamba install -y --file /tmp/psycopg2.txt && \ mamba clean --all -f -y && \ fix-permissions "${CONDA_DIR}" && \ fix-permissions "/home/${NB_USER}" # COPY --chown=${NB_UID} sqlalchemy.txt /tmp/sqlalchemy.txt # RUN mamba install -y --file /tmp/sqlalchemy.txt && \ # mamba clean --all -f -y && \ # fix-permissions "${CONDA_DIR}" && \ # fix-permissions "/home/${NB_USER}" FROM base as base-sparkmagic # Install additionnal packages. COPY --chown=${NB_UID} requirements-pyspark.txt /tmp/requirements-pyspark.txt RUN mamba install -y --file /tmp/requirements-pyspark.txt && \ mamba clean --all -f -y && \ fix-permissions "${CONDA_DIR}" && \ fix-permissions "/home/${NB_USER}" COPY --chown=${NB_UID} requirements-sparkmagic.txt /tmp/requirements-sparkmagic.txt RUN mamba install -y --file /tmp/requirements-sparkmagic.txt && \ mamba clean --all -f -y && \ fix-permissions "${CONDA_DIR}" && \ fix-permissions "/home/${NB_USER}" and my requirements file : requirements.txt : minio==7.2.15 scikit-learn trino-python-client==0.333.0 psycopg2.txt psycopg2==2.9.10 kafka.txt kafka-python==2.2.12 requirements-sparkmagic.txt pandas<2.0.0 sparkmagic==0.22.0 ipywidgets==8.1.6 apache-sedona==1.7.1 pytz==2025.2 dateutils==0.6.12 requirements-pyspark.txt pyspark==3.4.0 py4j==0.10.9.7 delta-spark==2.4.0 Continue reading...
