[Python] PyTorch with Docker issues: torch.cuda.is_available() = False

I'm having an issue with PyTorch in a Docker container where returns False, but the same PyTorch version works correctly outside the container. torch.cuda.is_available()

Environment

• Host: Debian 12
• GPU: NVIDIA GeForce RTX 3060 Ti (Driver Version: 550.54.14)
• CUDA Version: 12.1 (both host and container report the same version via nvcc --version)
• Docker version: 28.0.4, build b8034c0
• Docker Compose version v2.34.0

Docker Configuration


docker-compose.yml

services:
bert:
build:
context: ./bert
environment:
- BERT_MODEL_DIR=/app/model
- BERT_MAX_SEQ_LEN=256
volumes:
- ./bert:/app
- model:/app/model
ports:
- "8000:8000"
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
runtime: nvidia


Dockerfile

FROM nvidia/cuda:12.1.0-devel-ubuntu22.04

# Verify installation

RUN nvcc --version

ENV PATH=/usr/local/cuda/bin:$PATH
ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

# Update and install system dependencies

RUN apt-get update && apt-get install -y --no-install-recommends \
python3 \
python3-pip \
python3-dev \
python3-venv \
wget \
curl \
libc-dev \
git \
build-essential \
ldconfig \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/\*

# Ensure `python3` points to `python`

RUN ln -s /usr/bin/python3 /usr/bin/python \
&& python --version \
&& pip --version

# Set the working directory

WORKDIR /app

COPY ./requirements.txt /app/requirements.txt

# Install Python libraries

RUN python3 -m venv /opt/venv \
&& . /opt/venv/bin/activate \
&& pip install --no-cache-dir -r /app/requirements.txt

# Activate virtual environment by default

ENV PATH="/opt/venv/bin:$PATH"

COPY ./ /app

# Expose port for the API

EXPOSE 8000

# Run FastAPI app

CMD \["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000", "--reload"\]


requirements.txt

--extra-index-url https://download.pytorch.org/whl/cu121

# List of dependencies
fastapi>=0.103.1
loguru>=0.7.2
numpy<2
pydantic>=2.4.0
scikit-learn
sentencepiece>=0.1.99
torch==2.2.2+cu121
transformers>=4.35.0
uvicorn>=0.23.2

What I've tried

• Confirmed that on both host and container it shows the same GPU with nvidia-smi
• Confirmed that the same PyTorch version returns True when running outside the container torch.cuda.is_available()
• Added NVIDIA container toolkit to Docker
• Used in docker-compose.yml runtime: nvidia
• Set correct CUDA environment variables

All of these did not help, any suggestion would be welcome.

Continue reading...