[Python] pip install -r requirements.txt is extremely slow in Docker build using NVIDIA...

I'm trying to build a Docker image using the base image nvcr.io/nvidia/pytorch:23.05-py3. Everything seems fine until the step where it installs Python packages from requirements.txt using pip: pip install --no-cache-dir -r requirements.txt This step takes forever — over 3 hours — and doesn't complete. This feels abnormally slow.

Here is the full Dockerfile snippet:

FROM nvcr.io/nvidia/pytorch:23.05-py3

ENV LANG=C.UTF-8
ENV PYTHONUNBUFFERED=TRUE
ENV PYTHONDONTWRITEBYTECODE=TRUE
ENV NVIDIA_VISIBLE_DEVICES="all"

#install ffmpeg and sudo
RUN apt update && \
apt install -y software-properties-common && \
add-apt-repository -y ppa:jonathonf/ffmpeg-4 && \
apt install -y ffmpeg && \
apt install -y sudo && \
rm -rf /var/lib/apt/lists/*

#install git-lfs
RUN curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash && \
apt-get install -y git-lfs && \
git lfs install

#install requirements
COPY requirements.txt requirements.txt
RUN pip install --no-cache-dir -r requirements.txt && \
pip uninstall -y jupyterlab_widgets && \
pip install jupyterlab_widgets


here is my requirements.txt

boto3==1.38.32
librosa==0.11.0
evaluate>=0.4.0
jiwer>=3.10
gradio>=5.3.0
bitsandbytes==0.46.0
datasets==3.5.0
accelerate==1.6.0
loralib==0.1.2
transformers==4.51.3
peft==0.15.2
sagemaker-pytorch-training>=2.8.0


During the build, pip seems to be doing a lot of backtracking and retrying versions, like:

=> [5/5] RUN pip install --no-cache-dir -r requirements.txt && pip uninstall -y jupyterlab_widgets && pip install jupyterlab_widgets 25323.0s
=> => # Downloading charset_normalizer-2.1.0-py3-none-any.whl (39 kB)
=> => # INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can p
=> => # ress Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
=> => # Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
=> => # Downloading charset_normalizer-2.0.11-py3-none-any.whl (39 kB)
=> => # Downloading charset_normalizer-2.0.10-py3-none-any.whl (39 kB)

Continue reading...