[Python] How to create a system-wide environment to install pip packages when using pip inside...

I am trying to build an existing Dockerfile, which needs to be migrated from an older ubuntu version base image to a more up-to-date base image.

The existing Dockerfile contains commands such as

pip install pandas


There are many such pip commands, each of which triggers the following error message.

error: externally-managed-environment


This is not unexpected. Recent versions of Ubuntu produce this error when the user attempts to install pip packages without first activating a virtual environment.

This can be fixed by creating an activating an virtual environment. The disadvantage is that inside a Docker container, this shouldn't really be needed, since a container is its own isolated environment. In addition, it creates an additional layer which is slightly inconvenient. RUN python3 my_file.py no longer works directly, as the venv has to be activated first. (There are two ways to do this, the easiest of which is to do RUN /path/to/.venv/bin/python3 /path/to/my_file.py.)

The error could also be "fixed" by passing the --break-system-packages argument. I do not know in detail what the consequences of this are, so I do not know if this could be a recommended solution in this context.

There is a third possibility, which would be to install python3-pandas (assuming it exists). This is an apt package which provides an installation of pandas via apt. I would prefer not to use this method, since not all pip packages are available as apt packages. I aim to try and avoid a fragmented install whereby some packages are provided through one method and other packages are provided through a different method.

To review:

• What does the --break-system-packages command line option do? How "safe" is this inside a Docker container? (Rather than frequently creating and destroying this particular container, it tends to persist for a significant period of time. Typically several weeks to a few months.)
• If this isn't a suitable or recommended approach, is there a way that I can conveniently create a system-wide virtual environment, and somehow cause it to be "permanently" activated. (In other words, to create some kind of "transparent" virtual environment, which isn't noticeable to the user - so that running python3 main.py will run main.py with the virtual environment active, automatically. Can this be done?)

Continue reading...