[Python] How can I handle initial settings with Pydantic Settings?

I have an app that is largely configured by environment variables. I use Pydantic Settings to define the settings available, and validate them. I have an initial set of settings, and the regular app settings.

The initial settings are ones that should not fail validation, and contain essential settings for starting the app.

For example, when my app starts up, if the regular Settings() can't be initialized because something in them failed validation, I still want to be able to send the error to Sentry. For that, I need SENTRY_DSN to configure Sentry. SENTRY_DSN can't be part of the regular settings, because if something unrelated in Settings fails validation, I won't have access to SENTRY_DNS either.

Right now, my settings look like this:

class InitialSettings(BaseSettings):
model_config = SettingsConfigDict(
env_file="settings.env",
env_file_encoding="utf-8",
extra="ignore",
env_ignore_empty=True,
env_nested_delimiter="__",
case_sensitive=True,
)

SENTRY_DSN: Annotated[
Optional[str],
Field(None),
]

class Settings(BaseSettings):
model_config = SettingsConfigDict(
env_file="settings.env",
env_file_encoding="utf-8",
extra="ignore",
env_ignore_empty=True,
env_nested_delimiter="__",
case_sensitive=True,
)

STORAGE: Annotated[
LocalStorageSettings | S3StorageSettings,
Field(..., discriminator="STORAGE_TYPE"),
]
DEBUG: Annotated[DebugSettings, Field(default_factory=DebugSettings)]
...


This works. When my app starts up, I first initialize InitialSettings(), and then try to initialize Settings(). If Settings() fails, I can still use the SENTRY_DSN setting to send the error to Sentry.

The issues comes when I try to have both settings use the same env file (settings.env), AND enable the extra="forbid" feature on Settings().

I like the idea of having extra="forbid" enabled, but that also means that if I enable it on Settings(), it will always fail, because the env file will contain an entry for SENTRY_DSN, which Settings doesn't know about.

To fix, this I tried to add InitialSettings to Settings like this:

class Settings(BaseSettings):
model_config = SettingsConfigDict(
env_file="settings.env",
env_file_encoding="utf-8",
extra="forbid",
env_ignore_empty=True,
env_nested_delimiter="__",
case_sensitive=True,
)

STORAGE: Annotated[
LocalStorageSettings | S3StorageSettings,
Field(..., discriminator="STORAGE_TYPE"),
]
DEBUG: Annotated[DebugSettings, Field(default_factory=DebugSettings)]
INIT: Annotated[InitialSettings, Field(default_factory=InitialSettings)]
...


Now Settings should know about all the settings defined in InitialSettings, and if there's any extra settings in the env file that aren't defined in either class, it should fail.

This almost works.

The problem is that when you call InitialSettings, the SENTRY_DSN in the env file is expected to just be called SENTRY_DSN. When you call InitialSettings, because InitialSettings is nested under INIT, it expects the sentry variable to be called INIT__SENTRY_DSN.

How do I configure Pydantic Settings so that all settings under InitialSettings always look for SENTRY_DSN, no matter if they are initialized using InitialSettings(), or Settings()?

Note: I still want the other nested settings classes under Settings, like STORAGE, to work the same - be prefixed with STORAGE__ in the env file.

Continue reading...