[Python] How to fix this python import shadowing problem?

Problem Overview


I'm facing an import shadowing problem with a python 3.12 project that uses google protobuf.

Below is the structure of my project.

Project
├── proto (generated by protoc)
│ ├── google
│ │ └── api
│ │ └── ...
│ └── my_proto (depends on google.api protos)
│ └── ...
└── main.py

Virtualenv
└── lib
└── python3.12
└── site-packages
└── google
└── protobuf


I'm using a virtual environment to manage dependencies and protoc to generate python code. The problem is that in some of the generated code proto/my_proto, it attempts to do both import google.protobuf (which is the protobuf package) and the generated proto code module import google.api.xxxx. One of the imports is guaranteed to fail because the proto/google module shadows the one in the virtual environment. How can I fix this problem?

Minimal Repro

Get the Googleapi protos


clone this repo here: https://github.com/googleapis/googleapis

Create a directory with the following content


.
├── project
│ ├── proto
│ │ └── __init__.py
│ ├── __init__.py
│ └── __main__.py
├── proto
│ └── my_proto.proto
├── build.sh
└── pyproject.toml

__main__.py


from .proto.my_proto_pb2 import *

def main():
pass

my_proto.proto


syntax = "proto3";
import "google/api/annotations.proto";

message MyMessage {
string example = 1;
}

pyproject.toml


[tool.poetry]
name = "project"
version = "0.1.0"
description = ""
authors = []
readme = "README.md"

[tool.poetry.scripts]
main = "project.__main__:main"

[tool.poetry.dependencies]
python = "^3.12"
grpcio-tools = "1.66.1"
grpcio = "1.66.1"
protobuf = "5.27.2"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

build.sh


proto_folder="./proto"
google_proto_folder="<dir containg this repo: https://github.com/googleapis/googleapis>"

poetry lock
poetry install
poetry run python -m grpc_tools.protoc \
-I"$proto_folder" \
-I"$google_proto_folder/googleapis" \
--experimental_allow_proto3_optional \
--python_out="./project/proto" \
--grpc_python_out="./project/proto" \
$google_proto_folder/googleapis/google/api/annotations.proto \
$proto_folder/my_proto.proto

poetry install

Build and Run


change the working directory to the project root and run the following:

./build.sh
poetry run main


the following error message will be shown:

Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 995, in exec_module
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File "/home/tongweid/Documents/repos/avsim-service/_local/repro/project/project/__main__.py", line 1, in <module>
from .proto.my_proto_pb2 import *
File "/home/tongweid/Documents/repos/avsim-service/_local/repro/project/project/proto/my_proto_pb2.py", line 25, in <module>
from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
ModuleNotFoundError: No module named 'google.api'


I suspect this is because google.protobuf shadows the generated google.api.

Continue reading...