[Python] Service in a virtualenv calling nodes in their own virtualenv's

I have a situation where I have a large code base, that I did not author, only have a surface level understanding of and cannot substantially change. There is a parent service running in it's own virtualenv. This is importing various third party nodes, each with their own code base and dependencies. The nodes are imported using importlib.

As is normal, different nodes require different versions of the same packages and their dependencies. This is causing an issue as the parent service is importing one node after another. If diffusers v0.24 has already been imported and the next node needs diffusers v0.32, as diffusers has already been imported, the package by default will not be re imported.

Steps taken so far:

1. Parent service and nodes all in their own virtualenv's with their own package requirements installed.
2. Sys.path dynamically defined iteratively to always have the site package dir for the virtualenv first in the path and the location of the node second.
3. A shebang line added to the init.py file for each node to be imported.
4. Have changed the init.py file's chmod permissions to x+ and the chmod permissions for the node directory to 755.
5. Using the example of diffusers again, added the line print(importlib.metadata.version('diffusers')) in the node code to check that v0.32 is available where it should be and v0.24 likewise.
6. Iterated over the key, value pairs of sys.module in the virtualenv hosting the parent service to see where diffusers was imported from. If v0.24 has already been loaded in the parent virtualenv, v0.32 will not be loaded and I will get an error saying the code in the the node that needs v0.32 cant find what it needs from the diffusers library.

I've had partial success using the below code:

import sys

del_modules = ['diffusers']
del_list = []

print(sys.path)
print(sys.executable)

if (node_var != 'cogvideox' and diffusers_vers != '0.24.0') or (node_var == 'cogvideox' and diffusers_vers != '0.32.2'):
for module in del_modules:
for key, value in sys.modules.items():
#print(key, value)
key2 = key.split('.')[0]
if module in key2:
del_list.append(key)

for d in del_list:
print(f"Unloading {d}")
#reload(d)
del sys.modules[d]
if d in globals():
print(f"Unloading {d} globals")
del globals()[d]


...this feels a bit hacky to say the least, but it does at least achieve the effect of reloading diffusers for me. However, it seems like this has had a knock on effect on some other installed modules that are not behaving as expected.

I am also aware of the reload module from importlib and have tried something of the order of:

from importlib import reload

for key, value in sys.modules.items():
if '/usr' in str(value) and 'built-in' not in str(value) and 'dist-packages' in str(value) and 'importlib' not in str(value) and 'google' not in str(value) and 'ipy' not in str(value):
reload(sys.modules[key])


...this works, but when I have tried it in a sandbox type of way so far keeps causing my Google Colab session to crash.

I understand all the arguments about not having multiple versions of the same package used in the same code base etc, but I am not in a position to rewrite volumous and complex AI code.

Have I missed something really basic, or is this just an inbuilt limitation of Python? To recap, the parent service imports all the slave nodes and their dependencies in a loop as it's booting up and there is nothing I can do to change that.

I am basically looking for a more straightforward, reliable way of being able to import two nodes using two different versions of the same package into my parent service...

Continue reading...