# nested-config <!-- omit in toc -->
<span style="font-size: larger">If you've ever wanted to have the option of replacing part
of a configuration file with a path to another configuration file that contains those
sub-parameters, then _nested-config_ might be for you.</span>
_nested-config_ allows you to parse configuration files that contain references to other
configuration files using a series of [models](#model). If a model includes a [nested
model](#nested-model) as one of its attributes and _nested-config_ finds a string value
for that parameter in the configuration file instead of an associative
array[^assoc-array], then it assumes that this string is a path to another configuration
file that should be parsed and whose contents should replace the string in the main
configuration file. If the string appears to be a relative path, it is assumed to be
relative to the path of its parent configuration file.
## Contents
- [Contents](#contents)
- [Basic Usage](#basic-usage)
- [Nomenclature](#nomenclature)
- [loader](#loader)
- [model](#model)
- [nested model](#nested-model)
- [config dict](#config-dict)
- [API](#api)
- [`nested_config.expand_config(config_path, model, *, default_suffix = None)`](#nested_configexpand_configconfig_path-model--default_suffix--none)
- [`nested_config.config_dict_loaders`](#nested_configconfig_dict_loaders)
- [Included loaders](#included-loaders)
- [Adding loaders](#adding-loaders)
- [_Deprecated features in v2.1.0, to be removed in v3.0.0_](#deprecated-features-in-v210-to-be-removed-in-v300)
- [Pydantic 1.0/2.0 Compatibility](#pydantic-1020-compatibility)
- [Footnotes](#footnotes)
## Basic Usage
Given the following configuration files `/tmp/house.toml` and `/tmp/tmp2/dimensions.toml`:
<figure>
<figcaption>Figure 1: /tmp/house.toml</figcaption>
```toml
name = "my house"
dimensions = "tmp2/dimensions.toml"
```
</figure>
<figure>
<figcaption>Figure 2: /tmp/tmp2/dimensions.toml</figcaption>
```toml
length = 10
width = 20
```
</figure>
You can expand these into a single dict with the following:
<figure>
<figcaption>Figure 3: Expand /tmp/house.toml</figcaption>
```python
import nested_config
class Dimensions:
length: int
width: int
class House:
name: str
dimensions: Dimensions
house_dict = nested_config.expand_config("/tmp/house.toml", House)
print(house_dict)
# {'name': 'my house', 'dimensions': {'length': 10, 'width': 20}}
```
Note that in `/tmp/house.toml`, `dimensions` is not a mapping but is a path to another
toml file at a path relative to `house.toml`.
See [tests](https://gitlab.com/osu-nrsg/nested-config/-/tree/master/tests) for more
detailed use-cases, such as where the root model contains lists or dicts of other models
and when those may be included in the root config file or specified as paths to sub-config
files.
## Nomenclature
### loader
A _loader_ is a function that reads a config file and returns a `dict` containing the
key-value pairs from the file. _nested-config_ includes loaders for JSON, TOML, and (if
PyYAML is installed) YAML. For example, the JSON loader looks like this:
```python
import json
def json_load(path):
with open(path, "rb") as fobj:
return json.load(fobj)
```
### model
_nested-config_ uses the term _model_ to refer to a class definition that includes
annotated attributes. For example, this model, `Dimensions`, includes three attributes,
each of float type, `x`, `y`, and `z`:
```python
class Dimensions:
x: float
y: float
z: float
```
A model can be decorated as a [dataclass][dataclasses] or using [`attrs.define`][attrs] or
can subclass [`pydantic.BaseModel`][pydantic] to provide some method for instantiating an
object instance of the model but they aren't necessary to use _nested-config_.
The only criterion for a type to be a model is that is has a `__dict__` attribute that
includes an `__annotations__` member. _Note: This does **not** mean that **instances** of
the model must have a `__dict__` attribute. For example, instances of classes with
`__slots__` and `NamedTuple` instances may not have a `__dict__` attribute._
### nested model
A _nested model_ is a model that is included within another model as one of its class
attributes. For example, the below model `House` includes an `name` of string type, and an
attribute `dimensions` of `Dimensions` type (defined above). Since `Dimensions` is a
_model_ type, this is an example of a _nested model_.
```python
class House:
name: str
dimensions: Dimensions
```
### config dict
A _config dict_ is simply a `dict` with string keys such as may be obtained by reading in
configuration text. For example reading in a string of TOML text with `tomllib.loads`
returns a _config dict_.
```python
import tomllib
config = "x = 2\ny = 3"
print(tomllib.loads(config))
# {'x': 2, 'y': 3}
```
## API
### `nested_config.expand_config(config_path, model, *, default_suffix = None)`
This function first loads the config file at `config_path` into a [config
dict](#config-dict) using the appropriate [loader](#loader). It then uses the attribute
annotations of [`model`](#model) and/or any [nested models](#nested-model) within `model`
to see if any of the string values in the configuration file correspond to a nested model.
For each such case, the string is assumed to be a path and is loaded into another config
dict which replaces the string value in the parent config dict. This continues until all
paths are converted and then the fully-expanded config dict is returned.
Note that all non-absolute string paths are assumed to be relative to the path of their
parent config file.
The loader for a given config file is determined by file extension (AKA suffix). If
`default_suffix` is specified, any config file with an unknown suffix or no suffix will be
assumed to be of that type, e.g. `".toml"`. (Otherwise this is an error.) It is possible
for one config file to include a path to a config file of a different format, so long as
each file has the appropriate suffix and there is a loader for that suffix.
### `nested_config.config_dict_loaders`
`config_dict_loaders` is a `dict` that maps file suffixes to [loaders](#loader).
#### Included loaders
_nested-config_ automatically loads the following files based on extension:
| Format | Extensions(s) | Library |
| ------ | ------------- | ------------------------------------------ |
| JSON | .json | `json` (stdlib) |
| TOML | .toml | `tomllib` (Python 3.11+ stdlib) or `tomli` |
| YAML | .yaml, .yml | `pyyaml` (extra dependency[^yaml-extra]) |
#### Adding loaders
To add a loader for another file extension, simply update `config_dict_loaders`:
```python
import nested_config
from nested_config import ConfigDict # alias for dict[str, Any]
def dummy_loader(config_path: Path | str) -> ConfigDict:
return {"a": 1, "b": 2}
nested_config.config_dict_loaders[".dmy"] = dummy_loader
# or add another extension for an existing loader
nested_config.config_dict_loaders[".jsn"] = nested_config.config_dict_loaders[".json"]
# or use a different library to replace an existing loader
import rtoml
def rtoml_load(path) -> ConfigDict:
with open(path, "rb") as fobj:
return rtoml.load(fobj)
nested_config.config_dict_loaders[".toml"] = rtoml_load
```
### _Deprecated features in v2.1.0, to be removed in v3.0.0_
The following functionality is available only if Pydantic is installed:
- `nested_config.validate_config()` expands a configuration file according to a Pydantic
model and then validates the config dictionary into an instance of the Pydantic model.
- `nested_config.BaseModel` can be used as a replacement for `pydantic.BaseModel` to
include a `from_config()` classmethod on all models that uses
`nested_config.validate_config()` to create an instance of the model.
- By importing `nested_config`, `PurePath` validators and JSON encoders are added to
`pydantic` in Pydantic 1.8-1.10 (they are included in Pydantic 2.0+)
## Pydantic 1.0/2.0 Compatibility
The [pydantic functionality](#deprecated-features-in-v210-to-be-removed-in-v300) in
nested-config is runtime compatible with Pydantic 1.8+ and Pydantic 2.0.
The follow table gives info on how to configure the [mypy](https://www.mypy-lang.org/) and
[Pyright](https://microsoft.github.io/pyright) type checkers to properly work, depending
on the version of Pydantic you are using.
| Pydantic Version | [mypy config][1] | mypy cli | [Pyright config][2] |
|------------------|-----------------------------|-----------------------------|---------------------------------------------|
| 2.0+ | `always_false = PYDANTIC_1` | `--always-false PYDANTIC_1` | `defineConstant = { "PYDANTIC_1" = false }` |
| 1.8-1.10 | `always_true = PYDANTIC_1` | `--always-true PYDANTIC_1` | `defineConstant = { "PYDANTIC_1" = true }` |
## Footnotes
[^yaml-extra]: Install `pyyaml` separately with `pip` or install _nested-config_ with
`pip install nested-config[yaml]`.
[^assoc-array]: Each language uses one or more names for an associative arrays. JSON calls
it an _object_, YAML calls is a _mapping_, and TOML calls is a _table_.
Any of course in Python it's a _dictionary_, or `dict`.
[1]: https://mypy.readthedocs.io/en/latest/config_file.html
[2]: https://microsoft.github.io/pyright/#/configuration
[dataclasses]: https://docs.python.org/3/library/dataclasses.html
[attrs]: https://www.attrs.org
[pydantic]: https://pydantic.dev
Raw data
{
"_id": null,
"home_page": "https://gitlab.com/osu-nrsg/nested-config",
"name": "nested-config",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.8",
"maintainer_email": null,
"keywords": "config, configuration files",
"author": "Randall Pittman",
"author_email": "pittmara@oregonstate.edu",
"download_url": "https://files.pythonhosted.org/packages/c4/4c/8e0fdb10f823819b659800ef6352160ceabf66f7aef21d1ea4160b7ea208/nested_config-2.1.1.tar.gz",
"platform": null,
"description": "# nested-config <!-- omit in toc -->\n\n<span style=\"font-size: larger\">If you've ever wanted to have the option of replacing part\n of a configuration file with a path to another configuration file that contains those\nsub-parameters, then _nested-config_ might be for you.</span>\n\n_nested-config_ allows you to parse configuration files that contain references to other\nconfiguration files using a series of [models](#model). If a model includes a [nested\nmodel](#nested-model) as one of its attributes and _nested-config_ finds a string value\nfor that parameter in the configuration file instead of an associative\narray[^assoc-array], then it assumes that this string is a path to another configuration\nfile that should be parsed and whose contents should replace the string in the main\nconfiguration file. If the string appears to be a relative path, it is assumed to be\nrelative to the path of its parent configuration file.\n\n## Contents\n\n- [Contents](#contents)\n- [Basic Usage](#basic-usage)\n- [Nomenclature](#nomenclature)\n - [loader](#loader)\n - [model](#model)\n - [nested model](#nested-model)\n - [config dict](#config-dict)\n- [API](#api)\n - [`nested_config.expand_config(config_path, model, *, default_suffix = None)`](#nested_configexpand_configconfig_path-model--default_suffix--none)\n - [`nested_config.config_dict_loaders`](#nested_configconfig_dict_loaders)\n - [Included loaders](#included-loaders)\n - [Adding loaders](#adding-loaders)\n - [_Deprecated features in v2.1.0, to be removed in v3.0.0_](#deprecated-features-in-v210-to-be-removed-in-v300)\n- [Pydantic 1.0/2.0 Compatibility](#pydantic-1020-compatibility)\n- [Footnotes](#footnotes)\n\n## Basic Usage\n\nGiven the following configuration files `/tmp/house.toml` and `/tmp/tmp2/dimensions.toml`:\n\n<figure>\n<figcaption>Figure 1: /tmp/house.toml</figcaption>\n\n```toml\nname = \"my house\"\ndimensions = \"tmp2/dimensions.toml\"\n```\n\n</figure>\n\n<figure>\n<figcaption>Figure 2: /tmp/tmp2/dimensions.toml</figcaption>\n\n```toml\nlength = 10\nwidth = 20\n```\n\n</figure>\n\nYou can expand these into a single dict with the following:\n\n<figure>\n<figcaption>Figure 3: Expand /tmp/house.toml</figcaption>\n\n```python\nimport nested_config\n\nclass Dimensions:\n length: int\n width: int\n\n\nclass House:\n name: str\n dimensions: Dimensions\n\n\nhouse_dict = nested_config.expand_config(\"/tmp/house.toml\", House)\nprint(house_dict)\n# {'name': 'my house', 'dimensions': {'length': 10, 'width': 20}}\n```\n\nNote that in `/tmp/house.toml`, `dimensions` is not a mapping but is a path to another\ntoml file at a path relative to `house.toml`.\n\nSee [tests](https://gitlab.com/osu-nrsg/nested-config/-/tree/master/tests) for more\ndetailed use-cases, such as where the root model contains lists or dicts of other models\nand when those may be included in the root config file or specified as paths to sub-config\nfiles.\n\n## Nomenclature\n\n### loader\n\nA _loader_ is a function that reads a config file and returns a `dict` containing the\nkey-value pairs from the file. _nested-config_ includes loaders for JSON, TOML, and (if\nPyYAML is installed) YAML. For example, the JSON loader looks like this:\n\n```python\nimport json\n\ndef json_load(path):\n with open(path, \"rb\") as fobj:\n return json.load(fobj)\n```\n\n### model\n\n_nested-config_ uses the term _model_ to refer to a class definition that includes\nannotated attributes. For example, this model, `Dimensions`, includes three attributes,\neach of float type, `x`, `y`, and `z`:\n\n```python\nclass Dimensions:\n x: float\n y: float\n z: float\n```\n\nA model can be decorated as a [dataclass][dataclasses] or using [`attrs.define`][attrs] or\ncan subclass [`pydantic.BaseModel`][pydantic] to provide some method for instantiating an\nobject instance of the model but they aren't necessary to use _nested-config_.\n\nThe only criterion for a type to be a model is that is has a `__dict__` attribute that\nincludes an `__annotations__` member. _Note: This does **not** mean that **instances** of\nthe model must have a `__dict__` attribute. For example, instances of classes with\n`__slots__` and `NamedTuple` instances may not have a `__dict__` attribute._\n\n### nested model\n\nA _nested model_ is a model that is included within another model as one of its class\nattributes. For example, the below model `House` includes an `name` of string type, and an\nattribute `dimensions` of `Dimensions` type (defined above). Since `Dimensions` is a\n_model_ type, this is an example of a _nested model_.\n\n```python\nclass House:\n name: str\n dimensions: Dimensions\n```\n\n### config dict\n\nA _config dict_ is simply a `dict` with string keys such as may be obtained by reading in\nconfiguration text. For example reading in a string of TOML text with `tomllib.loads`\nreturns a _config dict_.\n\n```python\nimport tomllib\n\nconfig = \"x = 2\\ny = 3\"\nprint(tomllib.loads(config))\n# {'x': 2, 'y': 3}\n```\n\n## API\n\n### `nested_config.expand_config(config_path, model, *, default_suffix = None)`\n\nThis function first loads the config file at `config_path` into a [config\ndict](#config-dict) using the appropriate [loader](#loader). It then uses the attribute\nannotations of [`model`](#model) and/or any [nested models](#nested-model) within `model`\nto see if any of the string values in the configuration file correspond to a nested model.\nFor each such case, the string is assumed to be a path and is loaded into another config\ndict which replaces the string value in the parent config dict. This continues until all\npaths are converted and then the fully-expanded config dict is returned.\n\nNote that all non-absolute string paths are assumed to be relative to the path of their\nparent config file.\n\nThe loader for a given config file is determined by file extension (AKA suffix). If\n`default_suffix` is specified, any config file with an unknown suffix or no suffix will be\nassumed to be of that type, e.g. `\".toml\"`. (Otherwise this is an error.) It is possible\nfor one config file to include a path to a config file of a different format, so long as\neach file has the appropriate suffix and there is a loader for that suffix.\n\n### `nested_config.config_dict_loaders`\n\n`config_dict_loaders` is a `dict` that maps file suffixes to [loaders](#loader).\n\n#### Included loaders\n\n_nested-config_ automatically loads the following files based on extension:\n\n| Format | Extensions(s) | Library |\n| ------ | ------------- | ------------------------------------------ |\n| JSON | .json | `json` (stdlib) |\n| TOML | .toml | `tomllib` (Python 3.11+ stdlib) or `tomli` |\n| YAML | .yaml, .yml | `pyyaml` (extra dependency[^yaml-extra]) |\n\n#### Adding loaders\n\nTo add a loader for another file extension, simply update `config_dict_loaders`:\n\n```python\nimport nested_config\nfrom nested_config import ConfigDict # alias for dict[str, Any]\n\ndef dummy_loader(config_path: Path | str) -> ConfigDict:\n return {\"a\": 1, \"b\": 2}\n\nnested_config.config_dict_loaders[\".dmy\"] = dummy_loader\n\n# or add another extension for an existing loader\nnested_config.config_dict_loaders[\".jsn\"] = nested_config.config_dict_loaders[\".json\"]\n\n# or use a different library to replace an existing loader\nimport rtoml\n\ndef rtoml_load(path) -> ConfigDict:\n with open(path, \"rb\") as fobj:\n return rtoml.load(fobj)\n\nnested_config.config_dict_loaders[\".toml\"] = rtoml_load\n```\n\n### _Deprecated features in v2.1.0, to be removed in v3.0.0_\n\nThe following functionality is available only if Pydantic is installed:\n\n- `nested_config.validate_config()` expands a configuration file according to a Pydantic\n model and then validates the config dictionary into an instance of the Pydantic model.\n- `nested_config.BaseModel` can be used as a replacement for `pydantic.BaseModel` to\n include a `from_config()` classmethod on all models that uses\n `nested_config.validate_config()` to create an instance of the model.\n- By importing `nested_config`, `PurePath` validators and JSON encoders are added to\n `pydantic` in Pydantic 1.8-1.10 (they are included in Pydantic 2.0+)\n\n## Pydantic 1.0/2.0 Compatibility\n\nThe [pydantic functionality](#deprecated-features-in-v210-to-be-removed-in-v300) in\nnested-config is runtime compatible with Pydantic 1.8+ and Pydantic 2.0.\n\nThe follow table gives info on how to configure the [mypy](https://www.mypy-lang.org/) and\n[Pyright](https://microsoft.github.io/pyright) type checkers to properly work, depending\non the version of Pydantic you are using.\n\n| Pydantic Version | [mypy config][1] | mypy cli | [Pyright config][2] |\n|------------------|-----------------------------|-----------------------------|---------------------------------------------|\n| 2.0+ | `always_false = PYDANTIC_1` | `--always-false PYDANTIC_1` | `defineConstant = { \"PYDANTIC_1\" = false }` |\n| 1.8-1.10 | `always_true = PYDANTIC_1` | `--always-true PYDANTIC_1` | `defineConstant = { \"PYDANTIC_1\" = true }` |\n\n## Footnotes\n\n[^yaml-extra]: Install `pyyaml` separately with `pip` or install _nested-config_ with\n `pip install nested-config[yaml]`.\n\n[^assoc-array]: Each language uses one or more names for an associative arrays. JSON calls\n it an _object_, YAML calls is a _mapping_, and TOML calls is a _table_.\n Any of course in Python it's a _dictionary_, or `dict`.\n\n[1]: https://mypy.readthedocs.io/en/latest/config_file.html\n[2]: https://microsoft.github.io/pyright/#/configuration\n[dataclasses]: https://docs.python.org/3/library/dataclasses.html\n[attrs]: https://www.attrs.org\n[pydantic]: https://pydantic.dev\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Parse configuration files that include paths to other config files into a singleconfiguration object",
"version": "2.1.1",
"project_urls": {
"Changes": "https://gitlab.com/osu-nrsg/nested-config/-/blob/master/CHANGELOG.md",
"Homepage": "https://gitlab.com/osu-nrsg/nested-config",
"Repository": "https://gitlab.com/osu-nrsg/nested-config"
},
"split_keywords": [
"config",
" configuration files"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "56e7efd95dfd3ff7e0e1a3119d8e64c8491395e01d5bc5750c73c535d8d667f9",
"md5": "cbab17ffba81a379b92e55c8320dd1bd",
"sha256": "eeac03a333fdc7f08863a0dbb49edc82785874898750a484061ef10e398bf08b"
},
"downloads": -1,
"filename": "nested_config-2.1.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "cbab17ffba81a379b92e55c8320dd1bd",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.8",
"size": 14170,
"upload_time": "2024-04-19T16:45:54",
"upload_time_iso_8601": "2024-04-19T16:45:54.621564Z",
"url": "https://files.pythonhosted.org/packages/56/e7/efd95dfd3ff7e0e1a3119d8e64c8491395e01d5bc5750c73c535d8d667f9/nested_config-2.1.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c44c8e0fdb10f823819b659800ef6352160ceabf66f7aef21d1ea4160b7ea208",
"md5": "c9fffa35475db730de97f5efef950b61",
"sha256": "5e5a8bf52b68fcee687eeed6afb9c1c9249469f42cd697f978c5878640832854"
},
"downloads": -1,
"filename": "nested_config-2.1.1.tar.gz",
"has_sig": false,
"md5_digest": "c9fffa35475db730de97f5efef950b61",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.8",
"size": 16248,
"upload_time": "2024-04-19T16:45:56",
"upload_time_iso_8601": "2024-04-19T16:45:56.338909Z",
"url": "https://files.pythonhosted.org/packages/c4/4c/8e0fdb10f823819b659800ef6352160ceabf66f7aef21d1ea4160b7ea208/nested_config-2.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-19 16:45:56",
"github": false,
"gitlab": true,
"bitbucket": false,
"codeberg": false,
"gitlab_user": "osu-nrsg",
"gitlab_project": "nested-config",
"lcname": "nested-config"
}
JSON