[Python] The problem is when connecting the mlflow container to s3 storage

This is my docker-compose.yml

version: '3.8'

services:
postgres_mlflow:
image: postgres:16-alpine
environment:
- POSTGRES_USER=${MLFLOW_POSTGRES_USER}
- POSTGRES_PASSWORD=${MLFLOW_POSTGRES_PASSWORD}
- POSTGRES_DB=${MLFLOW_POSTGRES_DB}
volumes:
- ./mlflow/postgres_data:/var/lib/postgresql/data
ports:
- "5432:5432"

# MLflow Tracking Server
mlflow:
build: ./mlflow
depends_on:
- postgres_mlflow
environment:
- AWS_ACCESS_KEY_ID=${S3_ACCESS_KEY_ID}
- AWS_SECRET_ACCESS_KEY=${S3_SECRET_ACCESS_KEY}
- MLFLOW_S3_ENDPOINT_URL=https://storage.yandexcloud.net
command: >
mlflow server
--backend-store-uri postgresql://${MLFLOW_POSTGRES_USER}:${MLFLOW_POSTGRES_PASSWORD}@postgres_mlflow/${MLFLOW_POSTGRES_DB}
--default-artifact-root s3://my_bucket/
--host 0.0.0.0
--port 5000
ports:
- "5000:5000"


At the moment mlflow.path.log_model(self.model, "best_model"), an error occurs: botocore.exceptions.NoCredentialsError: Unable to locate credentials. The training is happening on the host

I decided to add the following lines to the training code (on the host):

load_dotenv()
os.environ["AWS_ACCESS_KEY_ID"] = os.getenv("S3_ACCESS_KEY_ID")
os.environ["AWS_SECRET_ACCESS_KEY"] = os.getenv("S3_SECRET_ACCESS_KEY")
os.environ["MLFLOW_S3_ENDPOINT_URL"] = ("https://storage.yandexcloud.net")


After this, the error disappeared.

However, in the graphical editor, in the Artifacts section, I see the error: Loading artifact failed INTERNAL_ERROR

Continue reading...