[Python] pip install . fails, looking into /var/private/folders

pip install . fails, while python3 ./setup.py install succeeds.

I am trying to compile and install a module that is purely local for now, written in C++. The name of the module is tcore. The module uses python C API and numpy C API.

pip install complains it cannot find numpy which is imported by my setup.py, however the import is actually correct and the setup.py import is very trivial.

numpy is correctly installed and can be imported:

pierre@pierre-maxm3 tcore % pip install numpy
Requirement already satisfied: numpy in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (1.26.2)
pierre@pierre-maxm3 tcore % python3
Python 3.12.3 (v3.12.3:f6650f9ad7, Apr 9 2024, 08:18:47) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy_path = numpy.get_include()
>>> print(numpy_path)
/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/numpy/core/include


Of course python3 setup.py install works and install my module properly.

But pip install fails to load numpy. It looks into /var/private/folders directory that doesn't exist... this is pip 24.3.1. Any idea? What is the problem with pip?

pierre@pierre-maxm3 tcore % pip install .
DEPRECATION: Loading egg at /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/tcore-1.0.0-py3.12-macosx-10.9-universal2.egg is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at https://github.com/pypa/pip/issues/12330
Processing /Users/pierre/src/trade/t-bot/c/modules/tcore
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [33 lines of output]
3.12.3 (v3.12.3:f6650f9ad7, Apr 9 2024, 08:18:47) [Clang 13.0.0 (clang-1300.0.29.30)]
system paths:
/Users/pierre/src/trade/t-bot/c/modules/tcore
/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process
/private/var/folders/9g/wy6m2s4d705fpzyr78yx0s9w0000gn/T/pip-build-env-1m0z9fie/site
/Library/Frameworks/Python.framework/Versions/3.12/lib/python312.zip
/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12
/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/lib-dynload
/private/var/folders/9g/wy6m2s4d705fpzyr78yx0s9w0000gn/T/pip-build-env-1m0z9fie/overlay/lib/python3.12/site-packages
/private/var/folders/9g/wy6m2s4d705fpzyr78yx0s9w0000gn/T/pip-build-env-1m0z9fie/normal/lib/python3.12/site-packages
/private/var/folders/9g/wy6m2s4d705fpzyr78yx0s9w0000gn/T/pip-build-env-1m0z9fie/overlay/lib/python3.12/site-packages/setuptools/_vendor

*******will import numpy
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
main()
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
^^^^^^^^^^^^^^^^^^^^^
File "/private/var/folders/9g/wy6m2s4d705fpzyr78yx0s9w0000gn/T/pip-build-env-1m0z9fie/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 334, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=[])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/private/var/folders/9g/wy6m2s4d705fpzyr78yx0s9w0000gn/T/pip-build-env-1m0z9fie/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 304, in _get_build_requires
self.run_setup()
File "/private/var/folders/9g/wy6m2s4d705fpzyr78yx0s9w0000gn/T/pip-build-env-1m0z9fie/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 522, in run_setup
super().run_setup(setup_script=setup_script)
File "/private/var/folders/9g/wy6m2s4d705fpzyr78yx0s9w0000gn/T/pip-build-env-1m0z9fie/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 320, in run_setup
exec(code, locals())
File "<string>", line 13, in <module>
ModuleNotFoundError: No module named 'numpy'
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
pierre@pierre-maxm3 tcore % ls -la /private/var/folders/9g/wy6m2s4d705fpzyr78yx0s9w0000gn/T/pip*
zsh: no matches found: /private/var/folders/9g/wy6m2s4d705fpzyr78yx0s9w0000gn/T/pip*
pierre@pierre-maxm3 tcore %


pip setup.py install :

pierre@pierre-maxm3 tcore % python3 setup.py install
['/Users/pierre/src/trade/t-bot/c/modules/tcore', '/Library/Frameworks/Python.framework/Versions/3.12/lib/python312.zip', '/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12', '/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/lib-dynload', '/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages', '/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/tcore-1.0.0-py3.12-macosx-10.9-universal2.egg']
3.12.3 (v3.12.3:f6650f9ad7, Apr 9 2024, 08:18:47) [Clang 13.0.0 (clang-1300.0.29.30)]
/Users/pierre/src/trade/t-bot/c/modules/tcore
/Library/Frameworks/Python.framework/Versions/3.12/lib/python312.zip
/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12
/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/lib-dynload
/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages
/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/tcore-1.0.0-py3.12-macosx-10.9-universal2.egg

*******will import numpy
done numpy
root_path /Users/pierre/src/trade/t-bot/c/modules/tcore
src_path /Users/pierre/src/trade/t-bot/c/modules/tcore
main() root_path /Users/pierre/src/trade/t-bot/c/modules/tcore
running install
/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

********************************************************************************
Please avoid running ``setup.py`` directly.
Instead, use pypa/build, pypa/installer or other
standards-based tools.

See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
********************************************************************************

!!
self.initialize_options()
/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/_distutils/cmd.py:66: EasyInstallDeprecationWarning: easy_install command is deprecated.
!!

********************************************************************************
Please avoid running ``setup.py`` and ``easy_install``.
Instead, use pypa/build, pypa/installer or other
standards-based tools.

See https://github.com/pypa/setuptools/issues/917 for details.
********************************************************************************

!!
self.initialize_options()
running bdist_egg
running egg_info
writing tcore.egg-info/PKG-INFO
writing dependency_links to tcore.egg-info/dependency_links.txt
writing top-level names to tcore.egg-info/top_level.txt
dependency /Users/pierre/src/trade/t-bot/c/src/ticker_database/src/TickerDatabase.h won't be automatically included in the manifest: the path must be relative
dependency /Users/pierre/src/trade/t-bot/c/src/ticker_database/src/Tickert.h won't be automatically included in the manifest: the path must be relative
dependency /Users/pierre/src/trade/t-bot/c/src/logger/src/TmonLogger.h won't be automatically included in the manifest: the path must be relative
reading manifest file 'tcore.egg-info/SOURCES.txt'
writing manifest file 'tcore.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.9-universal2/egg
running install_lib
running build_ext
building 'tcore' extension
clang -fno-strict-overflow -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -O3 -Wall -arch arm64 -arch x86_64 -g -I/Library/Frameworks/Python.framework/Versions/3.12/include/python3.12 -c /Users/pierre/src/trade/t-bot/c/src/logger/src/TmonLogger.cpp -o build/temp.macosx-10.9-universal2-cpython-312/Users/pierre/src/trade/t-bot/c/src/logger/src/TmonLogger.o -O3 -std=c++23 -I/Users/pierre/src/trade/t-bot/c/src/ticker_database/src -I/Users/pierre/src/trade/t-bot/c/src/logger/src -I/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/numpy/core/include
clang -fno-strict-overflow -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -O3 -Wall -arch arm64 -arch x86_64 -g -I/Library/Frameworks/Python.framework/Versions/3.12/include/python3.12 -c /Users/pierre/src/trade/t-bot/c/src/ticker_database/src/Ticker.cpp -o build/temp.macosx-10.9-universal2-cpython-312/Users/pierre/src/trade/t-bot/c/src/ticker_database/src/Ticker.o -O3 -std=c++23 -I/Users/pierre/src/trade/t-bot/c/src/ticker_database/src -I/Users/pierre/src/trade/t-bot/c/src/logger/src -I/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/numpy/core/include
clang -fno-strict-overflow -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -O3 -Wall -arch arm64 -arch x86_64 -g -I/Library/Frameworks/Python.framework/Versions/3.12/include/python3.12 -c /Users/pierre/src/trade/t-bot/c/src/ticker_database/src/TickerDatabase.cpp -o build/temp.macosx-10.9-universal2-cpython-312/Users/pierre/src/trade/t-bot/c/src/ticker_database/src/TickerDatabase.o -O3 -std=c++23 -I/Users/pierre/src/trade/t-bot/c/src/ticker_database/src -I/Users/pierre/src/trade/t-bot/c/src/logger/src -I/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/numpy/core/include
clang -fno-strict-overflow -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -O3 -Wall -arch arm64 -arch x86_64 -g -I/Library/Frameworks/Python.framework/Versions/3.12/include/python3.12 -c /Users/pierre/src/trade/t-bot/c/src/ticker_database/src/day_offsets.cpp -o build/temp.macosx-10.9-universal2-cpython-312/Users/pierre/src/trade/t-bot/c/src/ticker_database/src/day_offsets.o -O3 -std=c++23 -I/Users/pierre/src/trade/t-bot/c/src/ticker_database/src -I/Users/pierre/src/trade/t-bot/c/src/logger/src -I/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/numpy/core/include
clang -fno-strict-overflow -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -O3 -Wall -arch arm64 -arch x86_64 -g -I/Library/Frameworks/Python.framework/Versions/3.12/include/python3.12 -c tcore.cpp -o build/temp.macosx-10.9-universal2-cpython-312/tcore.o -O3 -std=c++23 -I/Users/pierre/src/trade/t-bot/c/src/ticker_database/src -I/Users/pierre/src/trade/t-bot/c/src/logger/src -I/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/numpy/core/include
clang++ -bundle -undefined dynamic_lookup -arch arm64 -arch x86_64 -g build/temp.macosx-10.9-universal2-cpython-312/Users/pierre/src/trade/t-bot/c/src/logger/src/TmonLogger.o build/temp.macosx-10.9-universal2-cpython-312/Users/pierre/src/trade/t-bot/c/src/ticker_database/src/Ticker.o build/temp.macosx-10.9-universal2-cpython-312/Users/pierre/src/trade/t-bot/c/src/ticker_database/src/TickerDatabase.o build/temp.macosx-10.9-universal2-cpython-312/Users/pierre/src/trade/t-bot/c/src/ticker_database/src/day_offsets.o build/temp.macosx-10.9-universal2-cpython-312/tcore.o -o build/lib.macosx-10.9-universal2-cpython-312/tcore.cpython-312-darwin.so -std=c++23
creating build/bdist.macosx-10.9-universal2/egg
copying build/lib.macosx-10.9-universal2-cpython-312/tcore.cpython-312-darwin.so -> build/bdist.macosx-10.9-universal2/egg
creating stub loader for tcore.cpython-312-darwin.so
byte-compiling build/bdist.macosx-10.9-universal2/egg/tcore.py to tcore.cpython-312.pyc
creating build/bdist.macosx-10.9-universal2/egg/EGG-INFO
copying tcore.egg-info/PKG-INFO -> build/bdist.macosx-10.9-universal2/egg/EGG-INFO
copying tcore.egg-info/SOURCES.txt -> build/bdist.macosx-10.9-universal2/egg/EGG-INFO
copying tcore.egg-info/dependency_links.txt -> build/bdist.macosx-10.9-universal2/egg/EGG-INFO
copying tcore.egg-info/top_level.txt -> build/bdist.macosx-10.9-universal2/egg/EGG-INFO
writing build/bdist.macosx-10.9-universal2/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
__pycache__.tcore.cpython-312: module references __file__
creating 'dist/tcore-1.0.0-py3.12-macosx-10.9-universal2.egg' and adding 'build/bdist.macosx-10.9-universal2/egg' to it
removing 'build/bdist.macosx-10.9-universal2/egg' (and everything under it)
Processing tcore-1.0.0-py3.12-macosx-10.9-universal2.egg
removing '/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/tcore-1.0.0-py3.12-macosx-10.9-universal2.egg' (and everything under it)
creating /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/tcore-1.0.0-py3.12-macosx-10.9-universal2.egg
Extracting tcore-1.0.0-py3.12-macosx-10.9-universal2.egg to /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages
Adding tcore 1.0.0 to easy-install.pth file

Installed /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/tcore-1.0.0-py3.12-macosx-10.9-universal2.egg
Processing dependencies for tcore==1.0.0
Finished processing dependencies for tcore==1.0.0
pierre@pierre-maxm3 tcore %



Here is my setup.py:

from setuptools import setup, Extension
import os
import sys
import site

print(sys.version)

print("system paths:")
for p in sys.path:
print(p)

print("\n*******will import numpy")
import numpy
print("done numpy")

root_path = os.path.abspath(".")
print(f"root_path {root_path}")

p = root_path
# look for top path of the .git repo
while ".git" not in os.listdir(p):
p = os.path.dirname(p)

src_path = os.path.join(p, "c", "src")
print(f"src_path {root_path}")

src_files = ["tcore.cpp",\
os.path.join(src_path, "ticker_database", "src", "TickerDatabase.cpp"),\
os.path.join(src_path, "ticker_database", "src", "day_offsets.cpp"),\
os.path.join(src_path, "ticker_database", "src", "Ticker.cpp"),\
os.path.join(src_path, "logger", "src", "TmonLogger.cpp"),\
]

h_files = [os.path.join(src_path, "ticker_database", "src", "TickerDatabase.h"),\
os.path.join(src_path, "ticker_database", "src", "Tickert.h"),\
os.path.join(src_path, "logger", "src", "TmonLogger.h"),\
]


sp = site.getsitepackages()[0]


numpy_path = numpy.get_include()

include_path = [os.path.join(src_path, "ticker_database", "src"),\
os.path.join(src_path, "logger", "src"),
numpy_path]

include_options = [f"-I{p}" for p in include_path]

compile_options = ["-O3", "-std=c++23"]

compile_options += include_options

def main():
print(f"main() root_path {root_path}")

src_path = "../"
setup(name="tcore",
version="1.0.0",
description="Python interface for tcore C++",
author="Pierre Vandwalle",
include_package_data=True,
ext_modules=[Extension("tcore",
src_files,
extra_compile_args=compile_options,
extra_link_args=["-std=c++23"],
depends=h_files,
)],
setup_requires=['numpy']
)

if __name__ == "__main__":
main()



Added pyproject.toml:

[project]
name = "tcore"

dependencies = [
"numpy==1.26.2",
]

Continue reading...