[Python] Installing torch with GPU support but without downloading 3 GB of duplicated siloed...

I'm trying to make CUDA containers for DNN less heavy. PyTorch goes against my efforts, as it seems to come bundled with its own siloed copy of a large subset of CUDA libraries. But what if we have them already (possibly newer) and want to install just torch?

And bonus question: why cannot PyTorch detect or accept your system CUDA libraries which match its own desired (and bundled in) major and minor CUDA version (e.g. 11.8)? Why does it have to force pip to download its own hard-coded CUDA 11.8.x when your system already has 11.8.y and y>x (i.e. slightly newer build of the same version)? After all, tensorflow can accept such minor differences in CUDA builds, and avoid unwanted duplication of these heavy dependencies (measured in gigabytes).


A more concrete illustration of the problem

(run under the reasonably new GPU driver - supporting the latest CUDA 12.2 - and the official nvidia/cuda-11.8-cudnn8-devel-ubuntu22.04:latest container with CUDA 11.8.0 installed inside):

!nvidia-smi
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
[..]


!env | grep CU | sort
CUDA_VERSION=11.8.0
[..]
NV_CUDA_LIB_VERSION=11.8.0-1
[..]
NV_CUDNN_PACKAGE=libcudnn8=8.9.0.131-1+cuda11.8
NV_CUDNN_PACKAGE_DEV=libcudnn8-dev=8.9.0.131-1+cuda11.8
[..]
NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1
[..]
NV_LIBCUBLAS_VERSION=11.11.3.6-1
NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1
NV_LIBCUSPARSE_VERSION=11.7.5.86-1


If you just intuitively try to install pip install torch, it will not download CUDA itself, but it will download the remaining NVIDIA libraries: its own (older) cuDNN (0.5 GB) and (older) NCCL, as well as various cu11* packages, including CUDA runtime (for an older version of CUDA - 11.7 instead of 11.8):

!pip install torch
Collecting torch
Downloading torch-2.0.1-cp310-cp310-manylinux1_x86_64.whl (619.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 619.9/619.9 MB 5.3 MB/s eta 0:00:0000:0100:02
[..]
Collecting nvidia-cudnn-cu11==8.5.0.96
Downloading nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 557.1/557.1 MB 5.1 MB/s eta 0:00:0000:0100:02
Collecting nvidia-cufft-cu11==10.9.0.58
Downloading nvidia_cufft_cu11-10.9.0.58-py3-none-manylinux1_x86_64.whl (168.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 168.4/168.4 MB 8.7 MB/s eta 0:00:0000:0100:01
Collecting nvidia-cusolver-cu11==11.4.0.1
Downloading nvidia_cusolver_cu11-11.4.0.1-2-py3-none-manylinux1_x86_64.whl (102.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 102.6/102.6 MB 9.7 MB/s eta 0:00:0000:0100:01
[..]
Collecting nvidia-cublas-cu11==11.10.3.66
Downloading nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl (317.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 317.1/317.1 MB 7.2 MB/s eta 0:00:0000:0100:01
Collecting nvidia-curand-cu11==10.2.10.91
Downloading nvidia_curand_cu11-10.2.10.91-py3-none-manylinux1_x86_64.whl (54.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.6/54.6 MB 10.2 MB/s eta 0:00:0000:0100:01
[..]
Collecting nvidia-nccl-cu11==2.14.3
Downloading nvidia_nccl_cu11-2.14.3-py3-none-manylinux1_x86_64.whl (177.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 177.1/177.1 MB 8.7 MB/s eta 0:00:0000:0100:01
Collecting nvidia-cuda-cupti-cu11==11.7.101
Downloading nvidia_cuda_cupti_cu11-11.7.101-py3-none-manylinux1_x86_64.whl (11.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.8/11.8 MB 11.4 MB/s eta 0:00:0000:0100:01
[..]
Collecting nvidia-nvtx-cu11==11.7.91
Downloading nvidia_nvtx_cu11-11.7.91-py3-none-manylinux1_x86_64.whl (98 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.6/98.6 KB 10.7 MB/s eta 0:00:00
Collecting nvidia-cuda-runtime-cu11==11.7.99
Downloading nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (849 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 849.3/849.3 KB 11.0 MB/s eta 0:00:00a 0:00:01
[..]
Successfully installed cmake-3.27.5 filelock-3.12.4 lit-17.0.1 mpmath-1.3.0 networkx-3.1 nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-cupti-cu11-11.7.101 nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-8.5.0.96 nvidia-cufft-cu11-10.9.0.58 nvidia-curand-cu11-10.2.10.91 nvidia-cusolver-cu11-11.4.0.1 nvidia-cusparse-cu11-11.7.4.91 nvidia-nccl-cu11-2.14.3 nvidia-nvtx-cu11-11.7.91 sympy-1.12 torch-2.0.1 triton-2.0.0



But if you try to install torch for CUDA 11.8 (specifying --index-url https://download.pytorch.org/whl/cu118), then torch will make pip download it's own CUDA 11.8 bundle (2.3 GB, nearly 4 times larger than the torch wheel alone), probably not even checking if one is already installed (and available e.g. for tensorflow):

!pip install torch --index-url https://download.pytorch.org/whl/cu118
Looking in indexes: https://download.pytorch.org/whl/cu118
Collecting torch
Downloading https://download.pytorch.org/whl/cu118/torch-2.0.1%2Bcu118-cp310-cp310-linux_x86_64.whl (2267.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 GB 2.3 MB/s eta 0:00:00:00:0100:06
[..]
Installing collected packages: mpmath, lit, cmake, sympy, networkx, filelock, triton, torch
Successfully installed cmake-3.25.0 filelock-3.9.0 lit-15.0.7 mpmath-1.3.0 networkx-3.0 sympy-1.12 torch-2.0.1+cu118 triton-2.0.0


So what was inside torch-2.0.1%2Bcu118-cp310-cp310-linux_x86_64.whl? We have some 3 GB (unpacked) of goodies there: cuDNN and several dynamic libraries with "cu" in their names, including the biggest one called libtorch_cuda.so, which - judging by its name and the sheer size - is PyTorch's own distribution of CUDA (or just a subset?):

148K libcudnn.so.8
680K libcudart-d0da41ae.so.11.0
1.3M libc10_cuda.so
72M libcudnn_ops_train.so.8
91M libcublas.so.11
94M libcudnn_ops_infer.so.8
98M libcudnn_cnn_train.so.8
116M libcudnn_adv_train.so.8
125M libcudnn_adv_infer.so.8
241M libtorch_cuda_linalg.so
548M libcublasLt.so.11
621M libcudnn_cnn_infer.so.8
1.3G libtorch_cuda.so


So it seems we don't need to install CUDA if it's PyTorch alone we are after, as it will download its own. Unless we make torch notice the pre-installed CUDA and cuDNN, the duplication of these libraries is unavoidable in other scenarios, because we will need the official NVIDIA version for the remaining DNN packages, such as Tensorflow or Hugging Face transformers.

Note: adding --no-deps argument to pip install would miss even the required C++ dynamic libraries (not just the bundled CUDA).

Continue reading...