[Python] Fixing Python import paths for a package in src used both from entry points outside...

I have a model package that runs in Sagemaker. It's structure looks something like this (domain-specific stuff redacted):

<My project root>
--> src
|--> potato
|----> a bunch of nested modules
|--> utils
|----> some modules
|--> datastuff
|----> more nested modules
--> test/
| ....
--> entrypoint1.py
--> entrypoint2.py
--> config and other stuff


Right now, the code is filled with a bunch of from src.potato.some_module import some_class, some_method etc. , both from code within src and from the entrypoints which aren't in src. This was working fine until now, but now we are trying to vend stuff within potato and datastuff to other packages (by publishing this package to a CA repo). Problems cropped up because in those dependent packages, I try to import something like from potato.blarg import some_method , but get errors about being unable to find a module called src.datastuff.some_module that exists in the blarg module's imports.

So my next step was to try and get rid of all the src pieces from the imports in the potato and datastuff packages. When I did this, VSCode was resolving the imports fine. But as soon as I tried to actually run either of the entrypoints (which live outside of src), I get the error:

Traceback (most recent call last):
File "<...>/entrypoint.py", line 1, in <module>
from src.potato.cli.blarg_cli import blarg_main
File "<...>/src/potato/cli/blarg_cli.py", line 10, in <module>
from potato.cli.cli_utils import ( <--- this used to be src.potato.<...>
ModuleNotFoundError: No module named 'potato'


I don't think I'm understanding something about how the imports are supposed to work and how I can fix this issue. I have considered doing a refactoring and moving the entrypoints into src, or moving stuff out of src, etc. but the team I am working on really wants to keep the structure like this (with the code in src, and the entrypoints out of it) and I think there are valid reasons to keep things this way, so I'm feeling a bit stuck and trying to see if I can make this work with a minimal change.

Any help here would be much appreciated!

Continue reading...