We use protobuf and the buf CLI to create and generate the shared files between a Go and a Python project (for ensuring structured data to be transmitted between the projects in a common way). We want to build the generator commands as simple as possible, as the generation is done using GitHub workflows and taskfiles etc. and therefore do not want (to the greatest extent possible), to maintain specific file paths every time a new .proto file is added. In addition we also use the buf linting to ensure the files are valid... We have tried different things, but don't think we can quite get what we want (without having to do post-processing by renaming and moving files and folders)... buf.yaml version: v2 modules: - path: proto lint: use: - STANDARD breaking: use: - FILE As mentioned we use the .proto files as the base for both a Go and a Python project - the Go project. The file structure (output) of the Go project is not as relevant or important as it is in our other component, which uses Python. buf.gen.yaml from the Python project (separate repo, fetching files from the other repo using the github input - this have however been removed from the below file listing as it has nothing to do with this issue) version: v2 managed: enabled: true plugins: - remote: buf.build/protocolbuffers/python out: components - remote: buf.build/protocolbuffers/pyi out: components The remote python plugin does not support the options (opt) argument - and the python protoc_builtin plugin seems to require each file to be lisited (no wildcards etc are supported). We have tried different structures of the proto files, but if I use a folder name with dots (like com.company.app1) the linter is not satisfied. proto/com/company/app1/entity1/v1/entity1.proto proto/com/company/app1/entity1/v2/entity1.proto proto/com/company/app2/entity2/v1/entity2.proto This requires the packages to be something like this: package com.company.app1.entity1.v1; message Entity1 { string name = 2; } (if we add dots in a folder name, then linter says that the package name used is not correct - and you cannot use underscore or hyphen to replace the dots in the folder name) What we want to achieve (output) is the generated folders in the python projects to be something like this: components/com.company.app1/entity1/v1/entity1.proto components/com.company.app1/entity1/v2/entity1.proto components/com.company.app2/entity2/v1/entity2.proto The Go code has not the same requirements regarding the folder structure and can be handled by the individual imports Continue reading...
