[Python] Trouble migrating to numpy 2: numpy.core.multiarray failed to import

I maintain openTSNE and would like to support numpy2. The package depends on numpy and uses Cython, so the build process is a bit more involved.

When building the package on Azure's CI servers, I previously depended on oldest-supported-numpy for maximum numpy backwards compatibility, but with numpy2, my understanding is that I don't need to do this anymore. Therefore, my pyproject.toml changed to

[build-system]
requires = ["setuptools", "wheel", "cython", "numpy"] # was `oldest-supported-numpy` before
build-backend = "setuptools.build_meta"


As far as I can tell, this works fine.

However, I am now getting a new error on my CI servers. You can find the full logs here.

ImportError while importing test module '/Users/runner/work/1/s/tests/test_utils.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_utils.py:4: in <module>
from openTSNE.utils import clip_point_to_disc
/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/openTSNE/__init__.py:1: in <module>
from .tsne import TSNE, TSNEEmbedding, PartialTSNEEmbedding, OptimizationInterrupt
/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/openTSNE/tsne.py:11: in <module>
from openTSNE import _tsne
openTSNE/_tsne.pyx:1: in init openTSNE._tsne
???
E ImportError: numpy.core.multiarray failed to import (auto-generated because you didn't call 'numpy.import_array()' after cimporting numpy; use '<void>numpy._import_array' to disable if you are certain you don't need it).


The numpy docs state that I should now explicitly call np.import_array() in my cython files (https://numpy.org/devdocs/numpy_2_0_migration_guide.html#c-api-changes) and the cython docs provide a simple example. However, I am still running into this error. I have tried to remedy this by adding the import at the beginning of all my .pyx and .pxd files, as indicated by the cython example, e.g.,

cimport numpy as cnp
cnp.import_array() # <--- HERE
import numpy as np
...
# function definitions


This didn't resolve the issue, so I also added these calls into every cpdef function in the file, e.g.,

cpdef double[:, ::1] compute_gaussian_perplexity(
double[:, :] distances,
double[:] desired_perplexities,
double perplexity_tol=1e-8,
Py_ssize_t max_iter=200,
Py_ssize_t num_threads=1,
):
cnp.import_array() # <--- HERE
cdef:
Py_ssize_t n_samples = distances.shape[0]
Py_ssize_t n_scales = desired_perplexities.shape[0]
...


but nothing seems to resolve the issue.

I would note that this only happens in my Release pipeline and not in my Test pipeline. In my Test pipeline, I essentially run only two commands:

pip install .
pytest -v


My Release pipeline, however is more involved, and involves first building the wheels, installing the produced wheel, and then running the test suite on this wheel. The error only occurs in this second Release pipeline. The sequence of commands is roughly

python -m cibuildwheel --output-dir dist/
python -m pip install --force-reinstall --find-links dist openTSNE
pytest # <--- FAILS


I am somewhat at a loss of what else to try at this point, so any help would be appreciated.

Continue reading...