[Python] How can I efficiently share schemas and utils between micro services in a monorepo...

Hi I am trying to build live prediction with the YOLO. The goal is to stream the data with some kind of transformation from the inference to a final frontend.

The flow should be like this:

1. model training(separate process)
2. Inference which saves the data in the Postgres
3. ELT process which reads the data from the postgres, transforms it and saves it to the a new table in the postgres
4. An API exposes the final table
5. Frontend uses the API to show the data in a dashboard

The idea is to put everything into a micro service sturcture to have independent and scalable solutions. I know in bigger scale a architecture with Kafka and Spark would be the more efficient way, but for this project I want to put in a micro service architrcture.

My problem now is that I want to share some utils and also some schemas between the services. My idea is to use a base container which I am building so that I can use this container as base image for all the container which need the schemas. Due to the reason that everything should end up in one product I also want to have it in a monorepo.

I also know schema sharing for micro services is not best practice but for this usecase it would really help.

Here shortly a simplyfied idea for our structure(here with videos which get live calculated):

.
├── data/
│ ├── weights
│ ├── model_data
│ └── inference_tests
├── model_training/
│ ├── train.py
│ ├── prep.py
│ └── eval.py
├── services/
│ ├── shared/
│ │ ├── Dockerfile
│ │ ├── schemas/
│ │ │ ├── stats.py
│ │ │ └── raw_data.py
│ │ └── db_utils
│ ├── inference/
│ │ ├── Dockerfile
│ │ ├── pyproject.toml
│ │ ├── main.py
│ │ └── src/
│ │ └── all_stuff.py
│ ├── etl_process/
│ │ ├── Dockerfile
│ │ ├── pyproject.toml
│ │ ├── main.py
│ │ └── src/
│ │ └── all_stuff.py
│ ├── backend_for_frontend/
│ │ ├── Dockerfile
│ │ ├── pyproject.toml
│ │ ├── main.py
│ │ └── src/
│ │ └── all_stuff.py
│ └── frontend/
│ ├── Dockerfile
│ ├── pyproject.toml
│ ├── main.py
│ └── src/
│ └── all_stuff.py
└── docker-compose.yaml


In the end I want to combine everything with docker-compose like this:

version: "3.8"

services:
# Base image for shared code
shared-base:
build:
context: ./services/shared
dockerfile: Dockerfile.base
image: shared-base-image

db:
image: postgres:13
volumes:
- db_data:.local/postgresql/data # .local is in the .gitignore
environment:
POSTGRES_USER: myuser
POSTGRES_PASSWORD: mypassword
POSTGRES_DB: mydb
ports:
- "5432:5432"

inference:
build:
context: ./services/inference
dockerfile: Dockerfile
depends_on:
- shared-base
- db
volumes:
- ./data/video:/input_videos # not stream yet
environment:
DB_HOST: db
DB_USER: myuser
DB_PASSWORD: mypassword
DB_NAME: mydb

etl-process:
build:
context: ./services/etl-process
dockerfile: Dockerfile
depends_on:
- shared-base
- db
environment:
DB_HOST: db
DB_USER: myuser
DB_PASSWORD: mypassword
DB_NAME: mydb

backend:
build:
context: ./services/backend_for_frontend
dockerfile: Dockerfile
depends_on:
- shared-base
- db
ports:
- "8000:8000"
environment:
DB_HOST: db
DB_USER: myuser
DB_PASSWORD: mypassword
DB_NAME: mydb

frontend:
build:
context: ./services/frontend
dockerfile: Dockerfile
ports:
- "3000:3000"

volumes:
db_data:


To have the shared modules and schemas I want to use the use the base container I am building as base image for others container that need the shared schemas and utils.

Here how I want to implement it:

FROM python:3.9-slim-buster
WORKDIR /app
COPY . /shared


Next file that depends on it:

FROM shared-base-image

RUN pip install uv

COPY . .

ENRTRYPOINT["uv", "run", "main.py"]


Now my final question: What would be the final structure for this workflow and design? Are there some design patterns which are really helpful?

With this sturucture I also face the issues that I cannot easiyl run the scripst and modules without the container. Does it make sense to append a paths based on if the path exists?

I mean I could also have only one bif src folder but then all services would have the same dependencies which woul also be overhead.

Thx already for your help and I hope you have some inut to improve the structure.

I hope you can give me some idea how to structure it effectively. It is mainly about design and design pattern.

Continue reading...