[Python] Is there a way to use symlink in site-packages to reduce disk space usage?

Every time I create a new project, it copies the entire package codebase into the project folder, which I find wasteful. For example, I don't want each of my projects to occupy 1 gigabyte worth of Tensorflow v2.8 disk space.

In other languages, we can easily avoid this. For instance, in Node.js, we can use pnpm or yarn berry. In Dart, this functionality is built-in by default. They utilize a global cache directory and the engine refers directly to ~/cache/.some-language/some-package/version/files (or its symlink in pnpm).

However, I can't seem to find a way to do this in Python. I read about all the dozens of Python package manager and I tried using uv because they advertise on their GitHub page:

Disk-space efficient, with a global cache for dependency deduplication.
​

Unfortunately, it turns out that this was completely false and misleading. I tried it, and it only caches the package to reduce network usage the next time it is installed, but it still copies an entire instance into each project's .venv folder. This does not improve disk space usage at all.

For example:

myproject/.venv/site-packages % du -sh * | sort -hr | head -20

1.0G tensorflow
326M jaxlib
106M mediapipe
106M cv2
81M scipy
72M clang
46M numpy


no-go answers:

• using a global interpreter and installing packages on that
• reusing environments (it slows down indexing of projects that don't require heavy packages like Tensorflow)

Continue reading...