[Python] How to use pyInstaller to package PyTorch code that includes JIT?

The spec file I used is as follows, based on this discussion.

# -*- mode: python ; coding: utf-8 -*-
from PyInstaller.utils.hooks import collect_data_files

datas = []
datas += collect_data_files('triton', include_py_files=True)
datas += collect_data_files('torchvision', include_py_files=True)

a = Analysis(
['python/sglang/launch_server.py'],
pathex=[],
binaries=[],
datas=datas,
hiddenimports=['triton'],
hookspath=[],
hooksconfig={},
runtime_hooks=[],
excludes=[],
noarchive=False,
optimize=0,
)
pyz = PYZ(a.pure)

exe = EXE(
pyz,
a.scripts,
[],
exclude_binaries=True,
name='launch_server',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
console=True,
disable_windowed_traceback=False,
argv_emulation=False,
target_arch=None,
codesign_identity=None,
entitlements_file=None,
)
coll = COLLECT(
exe,
a.binaries,
a.datas,
strip=False,
upx=True,
upx_exclude=[],
name='launch_server',
)


The packaging process had no errors, but the following error occurred at runtime:

Traceback (most recent call last):
File "sglang/launch_server.py", line 6, in <module>
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
...
File "PyInstaller/loader/pyimod02_importers.py", line 457, in exec_module
File "sglang/srt/layers/quantization/int8_utils.py", line 5, in <module>
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "PyInstaller/loader/pyimod02_importers.py", line 457, in exec_module
File "sglang/srt/layers/quantization/int8_kernel.py", line 21, in <module>
File "xxx/dist/launch_server/_internal/triton/runtime/jit.py", line 852, in jit
return decorator(fn)
File "xxx/dist/launch_server/_internal/triton/runtime/jit.py", line 840, in decorator
return JITFunction(
File "xxx/dist/launch_server/_internal/triton/runtime/jit.py", line 668, in __init__
self.starting_line_number = inspect.getsourcelines(fn)[1]
File "inspect.py", line 1121, in getsourcelines
File "inspect.py", line 958, in findsource
OSError: could not get source code
[PYI-195669:ERROR] Failed to execute script 'launch_server' due to unhandled exception!


It seems to be a JIT-related issue. Could it be that the JIT code in Triton, which exists as a string, was not packaged correctly? However, I have explicitly added Triton to the spec file and forced the loading of all Python files (through the collect_data_files interface). Are there any other methods to resolve this issue?

The Python project I packaged is sglang, based on the Ubuntu environment, with PyInstaller version 6.14.1 and Python version 3.10.12.

Continue reading...