# pydantic-file-secrets 🔑
> Use file secrets in nested [Pydantic Settings](https://docs.pydantic.dev/latest/concepts/pydantic_settings/) models, drop-in replacement for `SecretsSettingsSource`.
[](https://github.com/makukha/pydantic-file-secrets)
[](https://github.com/makukha/pydantic-file-secrets)
[](https://github.com/astral-sh/ruff)
[](https://github.com/psf/black) \
[](https://pypi.python.org/pypi/pydantic-file-secrets)
[](https://pypi.org/project/pydantic-file-secrets)
[](https://pydantic.dev)
This project is inspired by discussions in Pydantic Settings and solves problems in issues [#3](https://github.com/pydantic/pydantic-settings/issues/3), [#30](https://github.com/pydantic/pydantic-settings/issues/30), [#154](https://github.com/pydantic/pydantic-settings/issues/154).
This package unties secrets from environment variables config options and implements other long waited features.
## Features
* Use secret file source in nested settings models
* Drop-in replacement of standard `SecretsSettingsSource` (up to monkey patching)
* Plain or nested directory layout: `/run/secrets/dir__key` or `/run/secrets/dir/key`
* Respects `env_prefix`, `env_nested_delimiter` and other [config options](https://github.com/makukha/pydantic-file-secrets?tab=readme-ov-file#configuration-options)
* Has `secrets_prefix`, `secrets_nested_delimiter`, [etc.](https://github.com/makukha/pydantic-file-secrets?tab=readme-ov-file#configuration-options) to configure secrets and env vars separately
* Use multiple `secrets_dir` directories
* Pure Python thin wrapper over standard `EnvSettingsSource`
* No third party dependencies except `pydantic-settings`
* 100% test coverage
## Motivation
Nested Pydantic config can contain nested models with secret entries, as well as secrets in top level config. In dockerized environment, these entries may be read from file system, e.g. `/run/secrets` when using Docker Secrets:
```python
from pydantic import BaseModel, Secret
from pydantic_settings import BaseSettings, SettingsConfigDict
class DbSettings(BaseModel):
user: str
password: Secret[str] # secret in nested model
class Settings(BaseSettings):
db: DbSettings
app_key: Secret[str] # secret in root config
model_config = SettingsConfigDict(
secrets_dir='/run/secrets',
)
```
Pydantic Settings has a corresponding data source, [`SecretsSettingsSource`](https://docs.pydantic.dev/latest/api/pydantic_settings/#pydantic_settings.SecretsSettingsSource), but it does not load secrets in nested models. For things that DO NOT work in original Pydantic Settings, see [test_pydantic_motivation.py](https://github.com/makukha/pydantic-file-secrets/blob/main/tests/test_pydantic_motivation.py).
## Solution
The new `FileSecretsSettingsSource` is a drop-in replacement of stock `SecretsSettingsSource`.
### Installation
```shell
$ pip install pydantic-file-secrets
```
### Plain secrets directory layout
```text
# /run/secrets/app_key
secret1
# /run/secrets/db__password
secret2
```
```python
from pydantic import BaseModel, Secret
from pydantic_file_secrets import FileSecretsSettingsSource
from pydantic_settings import BaseSettings, SettingsConfigDict
class DbSettings(BaseModel):
user: str
password: Secret[str]
class Settings(BaseSettings):
db: DbSettings
app_key: Secret[str]
model_config = SettingsConfigDict(
secrets_dir='/run/secrets',
env_nested_delimiter='__',
)
@classmethod
def settings_customise_sources(
cls,
settings_cls,
init_settings,
env_settings,
dotenv_settings,
file_secret_settings,
):
return (
init_settings,
env_settings,
dotenv_settings,
FileSecretsSettingsSource(file_secret_settings),
)
```
### Nested secrets directory layout
Config option `secrets_nested_delimiter` overrides `env_nested_delimiter` for files. In particular, this allows to use nested directory layout along with environmemt variables for other non-secret settings:
```text
# /run/secrets/app_key
secret1
# /run/secrets/db/password
secret2
```
```python
...
model_config = SettingsConfigDict(
secrets_dir='/run/secrets',
secrets_nested_subdir=True,
)
...
```
### Multiple `secrets_dir`
When passing `list` to `secrets_dir`, last match wins.
```python
...
model_config = SettingsConfigDict(
secrets_dir=['/run/configs/', '/run/secrets'],
)
...
```
## Configuration options
### secrets_dir
Path to secrets directory. Same as `SecretsSettingsSource.secrets_dir` if `str` or `Path`. If `list`, the last match wins. If `secrets_dir` is passed in both source constructor and model config, values are not merged (constructor takes priority).
### secrets_dir_missing
If `secrets_dir` does not exist, original `SecretsSettingsSource` issues a warning. However, this may be undesirable, for example if we don't mount Docker Secrets in e.g. dev environment. Now you have a choice:
* `'ok'` — do nothing if `secrets_dir` does not exist
* `'warn'` (default) — print warning, same as `SecretsSettingsSource`
* `'error'` — raise `SettingsError`
If multiple `secrets_dir` passed, the same `secrets_dir_missing` action applies to each of them.
### secrets_dir_max_size
Limit the size of `secrets_dir` for security reasons, defaults to 8 MiB.
`FileSecretsSettingsSource` is a thin wrapper around [`EnvSettingsSource`](https://docs.pydantic.dev/latest/api/pydantic_settings/#pydantic_settings.EnvSettingsSource), which loads all potential secrets on initialization. This could lead to `MemoryError` if we mount a large file under `secrets_dir`.
If multiple `secrets_dir` passed, the limit applies to each directory independently.
### secrets_case_sensitive
Same as [`case_sensitive`](https://docs.pydantic.dev/latest/concepts/pydantic_settings/#case-sensitivity), but works for secrets only. If not specified, defaults to `case_sensitive`.
### secrets_nested_delimiter
Same as [`env_nested_delimiter`](https://docs.pydantic.dev/latest/concepts/pydantic_settings/#parsing-environment-variable-values), but works for secrets only. If not specified, defaults to `env_nested_delimiter`. This option is used to implement [nested secrets directory layout](https://github.com/makukha/pydantic-file-secrets?tab=readme-ov-file#nested-secrets-directory-layout) and allows to do even nastier things like `/run/secrets/model/delim/nested1/delim/nested2`.
### secrets_nested_subdir
Boolean flag to turn on nested secrets directory mode, `False` by default. If `True`, sets `secrets_nested_delimiter` to [`os.sep`](https://docs.python.org/3/library/os.html#os.sep). Raises `SettingsError` if `secrets_nested_delimiter` is already specified.
### secrets_prefix
Secret path prefix, similar to `env_prefix`, but works for secrets only. Defaults to `env_prefix` if not specified. Works in both plain and nested directory modes, like `'/run/secrets/prefix_model__nested'` and `'/run/secrets/prefix_model/nested'`.
### Not supported config options
Some config options that are declared in [`SecretsSettingsSource`](https://docs.pydantic.dev/latest/api/pydantic_settings/#pydantic_settings.SecretsSettingsSource) interface are actually [not working](https://github.com/makukha/pydantic-file-secrets/blob/main/tests/test_pydantic_source.py) and are not supported in `FileSecretsSettingsSource`:
* `env_ignore_empty`
* `env_parse_none_str`
* `env_parse_enums`
However, we [make sure](https://github.com/makukha/pydantic-file-secrets/blob/main/tests/test_ignored_options.py) that the behaviour of `FileSecretsSettingsSource` matches `SecretsSettingsSource` to provide a drop-in replacement, although it is somewhat wierd (e.g. `env_parse_enums` is always `True`).
## Testing
We [ensure](https://raw.githubusercontent.com/makukha/pydantic-file-secrets/main/tox.ini) 100% test coverage for latest Python release (3.12).
We [test](https://raw.githubusercontent.com/makukha/pydantic-file-secrets/main/tox.ini) all minor Pydantic Settings v2 versions and all minor Python 3 versions supported by Pydantic Settings:
* Python 3.{8,9,10,11,12,13} + pydantic-settings 2.{0,1,2,3,4}
## Roadmap
* Support `_FILE` environment variables to set secret file name.
* Per-field secret file name override.
## Authors
* [Michael Makukha](https://github.com/makukha)
## License
[MIT License](https://github.com/makukha/pydantic-file-secrets/blob/main/LICENSE)
Raw data
{
"_id": null,
"home_page": null,
"name": "pydantic-file-secrets",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "python, python3, pydantic, pydantic-v2, pydantic-settings, settings, configuration, config, validation, secrets, docker-secret, docker, file-secrets",
"author": null,
"author_email": "Michael Makukha <m.makukha@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/7d/24/388ad216f6ab76676b2f856f21a244f1be29b83dccac80cbde4d72da627f/pydantic_file_secrets-0.3.0.tar.gz",
"platform": null,
"description": "# pydantic-file-secrets \ud83d\udd11\n\n> Use file secrets in nested [Pydantic Settings](https://docs.pydantic.dev/latest/concepts/pydantic_settings/) models, drop-in replacement for `SecretsSettingsSource`.\n\n\n[](https://github.com/makukha/pydantic-file-secrets)\n[](https://github.com/makukha/pydantic-file-secrets)\n[](https://github.com/astral-sh/ruff)\n[](https://github.com/psf/black) \\\n[](https://pypi.python.org/pypi/pydantic-file-secrets)\n[](https://pypi.org/project/pydantic-file-secrets)\n[](https://pydantic.dev)\n\n\nThis project is inspired by discussions in Pydantic Settings and solves problems in issues [#3](https://github.com/pydantic/pydantic-settings/issues/3), [#30](https://github.com/pydantic/pydantic-settings/issues/30), [#154](https://github.com/pydantic/pydantic-settings/issues/154).\n\nThis package unties secrets from environment variables config options and implements other long waited features.\n\n\n## Features\n\n* Use secret file source in nested settings models\n* Drop-in replacement of standard `SecretsSettingsSource` (up to monkey patching)\n* Plain or nested directory layout: `/run/secrets/dir__key` or `/run/secrets/dir/key`\n* Respects `env_prefix`, `env_nested_delimiter` and other [config options](https://github.com/makukha/pydantic-file-secrets?tab=readme-ov-file#configuration-options)\n* Has `secrets_prefix`, `secrets_nested_delimiter`, [etc.](https://github.com/makukha/pydantic-file-secrets?tab=readme-ov-file#configuration-options) to configure secrets and env vars separately\n* Use multiple `secrets_dir` directories\n* Pure Python thin wrapper over standard `EnvSettingsSource`\n* No third party dependencies except `pydantic-settings`\n* 100% test coverage\n\n\n## Motivation\n\nNested Pydantic config can contain nested models with secret entries, as well as secrets in top level config. In dockerized environment, these entries may be read from file system, e.g. `/run/secrets` when using Docker Secrets:\n\n```python\nfrom pydantic import BaseModel, Secret\nfrom pydantic_settings import BaseSettings, SettingsConfigDict\n\nclass DbSettings(BaseModel):\n user: str\n password: Secret[str] # secret in nested model\n\nclass Settings(BaseSettings):\n db: DbSettings\n app_key: Secret[str] # secret in root config\n\n model_config = SettingsConfigDict(\n secrets_dir='/run/secrets',\n )\n```\n\nPydantic Settings has a corresponding data source, [`SecretsSettingsSource`](https://docs.pydantic.dev/latest/api/pydantic_settings/#pydantic_settings.SecretsSettingsSource), but it does not load secrets in nested models. For things that DO NOT work in original Pydantic Settings, see [test_pydantic_motivation.py](https://github.com/makukha/pydantic-file-secrets/blob/main/tests/test_pydantic_motivation.py).\n\n\n## Solution\n\nThe new `FileSecretsSettingsSource` is a drop-in replacement of stock `SecretsSettingsSource`.\n\n### Installation\n\n```shell\n$ pip install pydantic-file-secrets\n```\n\n### Plain secrets directory layout\n\n```text\n# /run/secrets/app_key\nsecret1\n\n# /run/secrets/db__password\nsecret2\n```\n\n```python\nfrom pydantic import BaseModel, Secret\nfrom pydantic_file_secrets import FileSecretsSettingsSource\nfrom pydantic_settings import BaseSettings, SettingsConfigDict\n\nclass DbSettings(BaseModel):\n user: str\n password: Secret[str]\n\nclass Settings(BaseSettings):\n db: DbSettings\n app_key: Secret[str]\n\n model_config = SettingsConfigDict(\n secrets_dir='/run/secrets',\n env_nested_delimiter='__',\n )\n \n @classmethod\n def settings_customise_sources(\n cls,\n settings_cls,\n init_settings,\n env_settings,\n dotenv_settings,\n file_secret_settings,\n ):\n return (\n init_settings,\n env_settings,\n dotenv_settings,\n FileSecretsSettingsSource(file_secret_settings),\n )\n\n```\n\n### Nested secrets directory layout\n\nConfig option `secrets_nested_delimiter` overrides `env_nested_delimiter` for files. In particular, this allows to use nested directory layout along with environmemt variables for other non-secret settings:\n\n```text\n# /run/secrets/app_key\nsecret1\n\n# /run/secrets/db/password\nsecret2\n```\n\n```python\n...\n model_config = SettingsConfigDict(\n secrets_dir='/run/secrets',\n secrets_nested_subdir=True,\n )\n...\n```\n\n### Multiple `secrets_dir`\n\nWhen passing `list` to `secrets_dir`, last match wins.\n\n```python\n...\n model_config = SettingsConfigDict(\n secrets_dir=['/run/configs/', '/run/secrets'],\n )\n...\n```\n\n## Configuration options\n\n### secrets_dir\n\nPath to secrets directory. Same as `SecretsSettingsSource.secrets_dir` if `str` or `Path`. If `list`, the last match wins. If `secrets_dir` is passed in both source constructor and model config, values are not merged (constructor takes priority).\n\n### secrets_dir_missing\n\nIf `secrets_dir` does not exist, original `SecretsSettingsSource` issues a warning. However, this may be undesirable, for example if we don't mount Docker Secrets in e.g. dev environment. Now you have a choice:\n\n* `'ok'` \u2014 do nothing if `secrets_dir` does not exist\n* `'warn'` (default) \u2014 print warning, same as `SecretsSettingsSource`\n* `'error'` \u2014 raise `SettingsError`\n\nIf multiple `secrets_dir` passed, the same `secrets_dir_missing` action applies to each of them.\n\n### secrets_dir_max_size\n\nLimit the size of `secrets_dir` for security reasons, defaults to 8 MiB.\n\n`FileSecretsSettingsSource` is a thin wrapper around [`EnvSettingsSource`](https://docs.pydantic.dev/latest/api/pydantic_settings/#pydantic_settings.EnvSettingsSource), which loads all potential secrets on initialization. This could lead to `MemoryError` if we mount a large file under `secrets_dir`.\n\nIf multiple `secrets_dir` passed, the limit applies to each directory independently.\n\n### secrets_case_sensitive\n\nSame as [`case_sensitive`](https://docs.pydantic.dev/latest/concepts/pydantic_settings/#case-sensitivity), but works for secrets only. If not specified, defaults to `case_sensitive`.\n\n### secrets_nested_delimiter\n\nSame as [`env_nested_delimiter`](https://docs.pydantic.dev/latest/concepts/pydantic_settings/#parsing-environment-variable-values), but works for secrets only. If not specified, defaults to `env_nested_delimiter`. This option is used to implement [nested secrets directory layout](https://github.com/makukha/pydantic-file-secrets?tab=readme-ov-file#nested-secrets-directory-layout) and allows to do even nastier things like `/run/secrets/model/delim/nested1/delim/nested2`.\n\n### secrets_nested_subdir\n\nBoolean flag to turn on nested secrets directory mode, `False` by default. If `True`, sets `secrets_nested_delimiter` to [`os.sep`](https://docs.python.org/3/library/os.html#os.sep). Raises `SettingsError` if `secrets_nested_delimiter` is already specified.\n\n### secrets_prefix\n\nSecret path prefix, similar to `env_prefix`, but works for secrets only. Defaults to `env_prefix` if not specified. Works in both plain and nested directory modes, like `'/run/secrets/prefix_model__nested'` and `'/run/secrets/prefix_model/nested'`.\n\n\n### Not supported config options\n\nSome config options that are declared in [`SecretsSettingsSource`](https://docs.pydantic.dev/latest/api/pydantic_settings/#pydantic_settings.SecretsSettingsSource) interface are actually [not working](https://github.com/makukha/pydantic-file-secrets/blob/main/tests/test_pydantic_source.py) and are not supported in `FileSecretsSettingsSource`:\n\n* `env_ignore_empty`\n* `env_parse_none_str`\n* `env_parse_enums`\n\nHowever, we [make sure](https://github.com/makukha/pydantic-file-secrets/blob/main/tests/test_ignored_options.py) that the behaviour of `FileSecretsSettingsSource` matches `SecretsSettingsSource` to provide a drop-in replacement, although it is somewhat wierd (e.g. `env_parse_enums` is always `True`).\n\n\n## Testing\n\nWe [ensure](https://raw.githubusercontent.com/makukha/pydantic-file-secrets/main/tox.ini) 100% test coverage for latest Python release (3.12).\n\nWe [test](https://raw.githubusercontent.com/makukha/pydantic-file-secrets/main/tox.ini) all minor Pydantic Settings v2 versions and all minor Python 3 versions supported by Pydantic Settings:\n\n* Python 3.{8,9,10,11,12,13} + pydantic-settings 2.{0,1,2,3,4}\n\n\n## Roadmap\n\n* Support `_FILE` environment variables to set secret file name.\n* Per-field secret file name override.\n\n\n## Authors\n\n* [Michael Makukha](https://github.com/makukha)\n\n## License\n\n[MIT License](https://github.com/makukha/pydantic-file-secrets/blob/main/LICENSE)\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Use file secrets in nested Pydantic Settings models, drop-in replacement for SecretsSettingsSource.",
"version": "0.3.0",
"project_urls": {
"Changelog": "https://github.com/makukha/pydantic-file-secrets/releases",
"Documentation": "https://github.com/makukha/pydantic-file-secrets#readme",
"Issues": "https://github.com/makukha/pydantic-file-secrets/issues",
"Source": "https://github.com/makukha/pydantic-file-secrets"
},
"split_keywords": [
"python",
" python3",
" pydantic",
" pydantic-v2",
" pydantic-settings",
" settings",
" configuration",
" config",
" validation",
" secrets",
" docker-secret",
" docker",
" file-secrets"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "7f3b634b02f735e03303cec3513f3a95f04d26103bd15438c1e89861b79f5ad5",
"md5": "0e01359ec3b24c690751e6da23216b96",
"sha256": "ee112993fee85f9f868af04caa662c8598b0b211edabfef64e38785d2626d10b"
},
"downloads": -1,
"filename": "pydantic_file_secrets-0.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0e01359ec3b24c690751e6da23216b96",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 7241,
"upload_time": "2024-09-03T19:07:43",
"upload_time_iso_8601": "2024-09-03T19:07:43.181852Z",
"url": "https://files.pythonhosted.org/packages/7f/3b/634b02f735e03303cec3513f3a95f04d26103bd15438c1e89861b79f5ad5/pydantic_file_secrets-0.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "7d24388ad216f6ab76676b2f856f21a244f1be29b83dccac80cbde4d72da627f",
"md5": "87e54908524b67cd9839f21f3aaf0174",
"sha256": "aeb313f4fffe853bf843b13ba1abefbe4dc96d9e13b3a3fbe80cf1ebd015f8a2"
},
"downloads": -1,
"filename": "pydantic_file_secrets-0.3.0.tar.gz",
"has_sig": false,
"md5_digest": "87e54908524b67cd9839f21f3aaf0174",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 12214,
"upload_time": "2024-09-03T19:07:46",
"upload_time_iso_8601": "2024-09-03T19:07:46.218107Z",
"url": "https://files.pythonhosted.org/packages/7d/24/388ad216f6ab76676b2f856f21a244f1be29b83dccac80cbde4d72da627f/pydantic_file_secrets-0.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-03 19:07:46",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "makukha",
"github_project": "pydantic-file-secrets",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"tox": true,
"lcname": "pydantic-file-secrets"
}
JSON
