[Python] map flat environment variables to nested Pydantic models using pydantic-settings

I have a YAML configuration and environment variables for my application, and I'm using pydantic-settings to load both YAML and environment variables into my Pydantic models. Pydantic shall prefer the env var over the yaml.

However, I'm facing an issue with nested models where the environment variable structure doesn't align with my nested Pydantic model fields.

Here's my config.yaml:

```
mongodb:
host: "mongo"
port: 27017
username: "USER"
password: "123456"
name: "db"
collection: "tasks"

s3:
bucket_name: "ai"
```


My code looks like this:

```
import os
from pydantic_settings import YamlConfigSettingsSource, BaseSettings, PydanticBaseSettingsSource, EnvSettingsSource
from typing import Tuple, Type
from pydantic import BaseModel


class DatabaseSettings(BaseModel):
host: str
port: int
username: str
password: str
name: str
collection: str


class S3Settings(BaseModel):
bucket_name: str


class Settings(BaseSettings):
mongodb: DatabaseSettings
s3: S3Settings

@classmethod
def settings_customise_sources(
cls, settings_cls: Type[BaseSettings], **kwargs
) -> Tuple[PydanticBaseSettingsSource, ...]:
return (
EnvSettingsSource(
settings_cls,
env_nested_delimiter="_",
case_sensitive=False,
env_prefix="PREFIX_"
),
YamlConfigSettingsSource(settings_cls, yaml_file="config.yaml"),
)


db_host = os.environ.get('PREFIX_MONGODB_HOST')
if db_host is not None:
print(f"PREFIX_MONGODB_HOST is set to: {db_host}")
else:
print("PREFIX_MONGODB_HOST is not set")

s3_bucket_name = os.environ.get('PREFIX_S3_BUCKET_NAME')
if s3_bucket_name is not None:
print(f"PREFIX_S3_BUCKET_NAME is set to: {s3_bucket_name}")
else:
print("PREFIX_S3_BUCKET_NAME is not set")

settings = Settings()
print(settings.model_dump())
```


I defined the following environment variables:

PREFIX_MONGODB_HOST=localhost

PREFIX_S3_BUCKET_NAME=some-bucket

When I run this code, the output is:

```
PREFIX_MONGODB_HOST is set to: localhost
PREFIX_S3_BUCKET_NAME is set to: some-bucket
{'mongodb': {'host': 'localhost', 'port': 27017, 'username': 'USER', 'password': '123456', 'name': 'db', 'collection': 'tasks'}, 's3': {'bucket_name': 'ai'}}
```


The Problem:

PREFIX_MONGODB_HOST correctly overwrites the mongodb.host key in the YAML. However, PREFIX_S3_BUCKET_NAME doesn't overwrite s3.bucket_name. The issue seems to be that pydantic-settings parse the environment variable PREFIX_S3_BUCKET_NAME to the following json:

{
"s3": {
"bucket": {
"name": "some-bucket"
}
}
}


which isn't corrosponsing to my pydantic model.

I would like it to parse the env var to:

{
"s3": {
"bucket_name": "some-bucket"
}
}


which is corrosponsing to the pydantic model. I cannot change the environment variable name (PREFIX_S3_BUCKET_NAME), so I'm looking for a pydantic-settings-oriented solution to map this environment variable to the bucket_name field in the S3Settings model.

I've tried using aliases and different configurations, but nothing has worked so far.

Would appritaite any help

Continue reading...