[Python] Postgres database missing in docker container using DBT

I'm following this data engineering course and I'm having trouble on the DBT part. I configured two databases, a source_db and a destination_db, using this docker-compose.yml file :

# version: '3'

services:
source_postgres:
image: postgres:latest
ports:
- "5433:5432"
networks:
- elt_network
environment:
POSTGRES_DB: source_db
POSTGRES_USER: postgres
POSTGRES_PASSWORD: secret
volumes:
- ./source_db_init/init.sql:/docker-entrypoint-initdb.d/init.sql

destination_postgres:
image: postgres:latest
ports:
- "5434:5432"
networks:
- elt_network
environment:
POSTGRES_DB: destination_db
POSTGRES_USER: postgres
POSTGRES_PASSWORD: secret

elt_script:
build:
context: ./elt
dockerfile: Dockerfile
command: ["python3", "elt_script.py"]
networks:
- elt_network
depends_on:
- source_postgres
- destination_postgres

dbt:
image: ghcr.io/dbt-labs/dbt-postgres:1.8.1
command:
[
"run",
"--profiles-dir",
"/root",
"--project-dir",
"/dbt"
]
networks:
- elt_network
volumes:
- ./custom_postgres:/dbt
- ~/.dbt:/root
depends_on:
- elt_script
environment:
DBT_PROFILE: custom_postgres
DBT_TARGET: dev

networks:
elt_network:
driver: bridge



When I used the command docker compose up I end up with this error in the docker logs:

2024-06-03 18:40:32 16:40:32 Running with dbt=1.8.1
2024-06-03 18:40:32 16:40:32 Registered adapter: postgres=1.8.1
2024-06-03 18:40:32 16:40:32 Found 4 models, 20 data tests, 3 sources, 413 macros
2024-06-03 18:40:32 16:40:32
2024-06-03 18:40:33 16:40:33 Concurrency: 1 threads (target='dev')
2024-06-03 18:40:33 16:40:33
2024-06-03 18:40:33 16:40:33 1 of 4 START sql table model public.actors ..................................... [RUN]
2024-06-03 18:40:33 16:40:33 1 of 4 ERROR creating sql table model public.actors ............................ [ERROR in 0.10s]
2024-06-03 18:40:33 16:40:33 2 of 4 START sql table model public.film_actors ................................ [RUN]
2024-06-03 18:40:33 16:40:33 2 of 4 ERROR creating sql table model public.film_actors ....................... [ERROR in 0.06s]
2024-06-03 18:40:33 16:40:33 3 of 4 START sql table model public.films ...................................... [RUN]
2024-06-03 18:40:33 16:40:33 3 of 4 ERROR creating sql table model public.films ............................. [ERROR in 0.06s]
2024-06-03 18:40:33 16:40:33 4 of 4 SKIP relation public.film_ratings ....................................... [SKIP]
2024-06-03 18:40:33 16:40:33
2024-06-03 18:40:33 16:40:33 Finished running 4 table models in 0 hours 0 minutes and 0.46 seconds (0.46s).
2024-06-03 18:40:33 16:40:33
2024-06-03 18:40:33 16:40:33 Completed with 3 errors and 0 warnings:
2024-06-03 18:40:33 16:40:33
2024-06-03 18:40:33 16:40:33 Database Error in model actors (models/example/actors.sql)
2024-06-03 18:40:33 relation "public.actors" does not exist
2024-06-03 18:40:33 LINE 12: SELECT * FROM "destination_db"."public"."actors"
2024-06-03 18:40:33 ^
2024-06-03 18:40:33 compiled Code at target/run/custom_postgres/models/example/actors.sql
2024-06-03 18:40:33 16:40:33
2024-06-03 18:40:33 16:40:33 Database Error in model film_actors (models/example/film_actors.sql)
2024-06-03 18:40:33 relation "public.film_actors" does not exist
2024-06-03 18:40:33 LINE 12: SELECT * FROM "destination_db"."public"."film_actors"
2024-06-03 18:40:33 ^
2024-06-03 18:40:33 compiled Code at target/run/custom_postgres/models/example/film_actors.sql
2024-06-03 18:40:33 16:40:33
2024-06-03 18:40:33 16:40:33 Database Error in model films (models/example/films.sql)
2024-06-03 18:40:33 relation "public.films" does not exist
2024-06-03 18:40:33 LINE 12: SELECT * FROM "destination_db"."public"."films"
2024-06-03 18:40:33 ^
2024-06-03 18:40:33 compiled Code at target/run/custom_postgres/models/example/films.sql
2024-06-03 18:40:33 16:40:33
2024-06-03 18:40:33 16:40:33 Done. PASS=0 WARN=0 ERROR=3 SKIP=1 TOTAL=4



It's pointing that some relations does not exist but I investigated in the destination_db docker container doing so :

connect to container command line: docker exec -it <id_of_my_container> bash

switch to the postgres user inside my container: su - postgres

enable postgreSQL command line: psql

connection to destination_db: \c destination_db

list of tables inside destination_db: \dt

All the tables listed in the error logs were here. I don't understand what is going wrong.

Continue reading...