TL;DR: When pip install builds and installs wheels, how do I determine which versions of build tool packages (e.g. Cython) it used? How do I force it to use the same versions of those packages in the future? Besides pip, wheel, and cython, what packages do I need to care about in the first place? I maintain a project which uses Docker containers running Python. These containers use an image which is based on the python:3.11.2-slim-bullseye official image, with the various Python packages and non-Python tools we require installed. One of our concerns is making our builds repeatable and consistent, and we've taken a number of measures to that end: Our base image, rather than being the 3.11.2-slim-bullseye tag, is a specific hash, so even if a new version of that tag is released, the image we are building from will be the same Before installing our Python packages, we make sure to install a specific version of pip and wheel Our requirement.txt file lists not only what packages we require, but the exact versions to install, and the versions of their dependencies to use Despite all this, we recently suffered a broken build: a package, metrohash, started to fail during the compilation step. After investigation, we discovered the problem: The metrohash module was written using Cython, a package that allows for writing code in a sort of hybrid of Python & C or C++. Code in this format looks mostly like ordinary Python, but Cython can transform it into C or C++; the compiled code can then be called as if it were the Python module it resembled The metrohash code used the Python 2 type 'long' to represent data that would, in the C++ code, be stored in an actual long int When metrohash was installed, pip (or possibly wheel) would automatically download the latest version of Cython in order to transform and build the code The latest version of Cython, 3.10, release just a week or so ago, dropped support for a bunch of deprecated Python 2 features, including the Python long type Thus we were getting an error when trying to transform the code into copilable form, so the build failed We manage to find a workaround for that issue, but the fact that it happened in the first place revealed a serious weakness in our build system: we were implicitly depending on a package, Cython, for which were not controlling what version to use. Obviously, I want to change that, and make the version used consistent, but I have no idea how: I went back and checked the logs from the last successful build, and nothing in pip's output mentions cython at all, let alone what particular version it is using. Nor does cython show up in the output of pip freeze --all, either before or after the install of metrohash. So how do I check what version is actually being used? Once I know what version we are currently using, how do I ensure that particular version continues to be used in the future? I could try installing the version in question in a separate pip install step before the main install, but given the package doesn't show up in the output of pip freeze --all, I'm not at all sure it would actually use the installed package for the build. Most importantly, are there any other packages which may be implicitly used in the install process like this? How can I tell? Continue reading...
