# python-configuration
> A library to load configuration parameters hierarchically from multiple sources and formats
[](https://pypi.org/project/python-configuration/)
[](https://travis-ci.org/tr11/python-configuration)
[](https://codecov.io/gh/tr11/python-configuration)
[](https://python-configuration.readthedocs.io/en/latest/?badge=latest)
This library is intended as a helper mechanism to load configuration files hierarchically. Supported format types are:
* Python files
* Dictionaries
* Environment variables
* Filesystem paths
* JSON files
* INI files
* dotenv type files
and optionally
* YAML files
* TOML files
* Azure Key Vault credentials
* AWS Secrets Manager credentials
* GCP Secret Manager credentials
## Installing
To install the library:
```shell
pip install python-configuration
```
To include the optional TOML and/or YAML loaders, install the optional dependencies `toml` and ` yaml`. For example,
```shell
pip install python-configuration[toml,yaml]
```
## Getting started
`python-configuration` converts the various config types into dictionaries with dotted-based keys. For example, given this JSON configuration
```json
{
"a": {
"b": "value"
}
}
```
We can use the `config_from_json` method to parse it:
```python
from config import config_from_json
cfg = config_from_json("my_config_file.json", read_from_file=True)
```
(Similar methods exist for all the other supported configuration formats (eg. `config_from_toml`, etc.).)
We are then able to refer to the parameters in the config above using any of:
```python
cfg['a.b']
cfg['a']['b']
cfg['a'].b
cfg.a.b
```
and extract specific data types such as dictionaries:
```python
cfg['a'].as_dict == {'b': 'value'}
```
This is particularly useful in order to isolate group parameters.
For example, with the JSON configuration
```json
{
"database.host": "something",
"database.port": 12345,
"database.driver": "name",
"app.debug": true,
"app.environment": "development",
"app.secrets": "super secret",
"logging": {
"service": "service",
"token": "token",
"tags": "tags"
}
}
```
one can retrieve the dictionaries as
```python
cfg.database.as_dict()
cfg.app.as_dict()
cfg.logging.as_dict()
```
or simply as
```python
dict(cfg.database)
dict(cfg.app)
dict(cfg.logging)
```
## Configuration
There are two general types of objects in this library. The first one is the `Configuration`, which represents a single config source. The second is a `ConfigurationSet` that allows for multiple `Configuration` objects to be specified.
### Single Config
#### Python Files
To load a configuration from a Python module, the `config_from_python` can be used.
The first parameter must be a Python module and can be specified as an absolute path to the Python file or as an importable module.
Optional parameters are the `prefix` and `separator`. The following call
```python
config_from_python('foo.bar', prefix='CONFIG', separator='__')
```
will read every variable in the `foo.bar` module that starts with `CONFIG__` and replace every occurrence of `__` with a `.`. For example,
```python
# foo.bar
CONFIG__AA__BB_C = 1
CONFIG__AA__BB__D = 2
CONF__AA__BB__D = 3
```
would result in the configuration
```python
{
'aa.bb_c': 1,
'aa.bb.d': 2,
}
```
Note that the single underscore in `BB_C` is not replaced and the last line is not prefixed by `CONFIG`.
#### Dictionaries
Dictionaries are loaded with `config_from_dict` and are converted internally to a flattened `dict`.
```python
{
'a': {
'b': 'value'
}
}
```
becomes
```python
{
'a.b': 'value'
}
```
#### Environment Variables
Environment variables starting with `prefix` can be read with `config_from_env`:
```python
config_from_env(prefix, separator='_')
```
#### Filesystem Paths
Folders with files named as `xxx.yyy.zzz` can be loaded with the `config_from_path` function. This format is useful to load mounted Kubernetes [ConfigMaps](https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#populate-a-volume-with-data-stored-in-a-configmap) or [Secrets](https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#create-a-pod-that-has-access-to-the-secret-data-through-a-volume).
#### JSON, INI, .env, YAML, TOML
JSON, INI, YAML, TOML files are loaded respectively with
`config_from_json`,
`config_from_ini`,
`config_from_dotenv`,
`config_from_yaml`, and
`config_from_toml`.
The parameter `read_from_file` controls whether a string should be interpreted as a filename.
###### Caveats
In order for `Configuration` objects to act as `dict` and allow the syntax `dict(cfg)`, the `keys()` method is implemented as the typical `dict` keys. If `keys` is an element in the configuration `cfg` then the `dict(cfg)` call will fail. In that case, it's necessary to use the `cfg.as_dict()` method to retrieve the `dict` representation for the `Configuration` object.
The same applies to the methods `values()` and `items()`.
### Configuration Sets
Configuration sets are used to hierarchically load configurations and merge settings. Sets can be loaded by constructing a `ConfigurationSet` object directly or using the simplified `config` function.
To construct a `ConfigurationSet`, pass in as many of the simple `Configuration` objects as needed:
```python
cfg = ConfigurationSet(
config_from_env(prefix=PREFIX),
config_from_json(path, read_from_file=True),
config_from_dict(DICT),
)
```
The example above will read first from Environment variables prefixed with `PREFIX`, and fallback first to the JSON file at `path`, and finally use the dictionary `DICT`.
The `config` function simplifies loading sets by assuming some defaults.
The example above can also be obtained by
```python
cfg = config(
('env', PREFIX),
('json', path, True),
('dict', DICT),
)
```
or, even simpler if `path` points to a file with a `.json` suffix:
```python
cfg = config('env', path, DICT, prefix=PREFIX)
```
The `config` function automatically detects the following:
* extension `.py` for python modules
* dot-separated python identifiers as a python module (e.g. `foo.bar`)
* extension `.json` for JSON files
* extension `.yaml` for YAML files
* extension `.toml` for TOML files
* extension `.ini` for INI files
* extension `.env` for dotenv type files
* filesystem folders as Filesystem Paths
* the strings `env` or `environment` for Environment Variables
#### Merging Values
`ConfigurationSet` instances are constructed by inspecting each configuration source, taking into account nested dictionaries, and merging at the most granular level.
For example, the instance obtained from `cfg = config(d1, d2)` for the dictionaries below
```python
d1 = {'sub': {'a': 1, 'b': 4}}
d2 = {'sub': {'b': 2, 'c': 3}}
```
is such that `cfg['sub']` equals
```python
{'a': 1, 'b': 4, 'c': 3}
```
Note that the nested dictionaries of `'sub'` in each of `d1` and `d2` do not overwrite each other, but are merged into a single dictionary with keys from both `d1` and `d2`, giving priority to the values of `d1` over those from `d2`.
###### Caveats
As long as the data types are consistent across all the configurations that are part of a `ConfigurationSet`, the behavior should be straightforward. When different configuration objects are specified with competing data types, the first configuration to define the elements sets its datatype. For example, if in the example above `element` is interpreted as a `dict` from environment variables, but the JSON file specifies it as anything else besides a mapping, then the JSON value will be dropped automatically.
## Other Features
###### String Interpolation
When setting the `interpolate` parameter in any `Configuration` instance, the library will perform a string interpolation step using the [str.format](https://docs.python.org/3/library/string.html#formatstrings) syntax. In particular, this allows to format configuration values automatically:
```python
cfg = config_from_dict({
"percentage": "{val:.3%}",
"with_sign": "{val:+f}",
"val": 1.23456}, interpolate=True)
assert cfg.val == 1.23456
assert cfg.with_sign == "+1.234560"
assert cfg.percentage == "123.456%"
```
## Extras
The `config.contrib` package contains extra implementations of the `Configuration` class used for special cases. Currently the following are implemented:
* `AzureKeyVaultConfiguration` in `config.contrib.azure`, which takes Azure Key Vault
credentials into a `Configuration`-compatible instance. To install the needed dependencies
execute
```shell
pip install python-configuration[azure]
```
* `AWSSecretsManagerConfiguration` in `config.contrib.aws`, which takes AWS Secrets Manager
credentials into a `Configuration`-compatible instance. To install the needed dependencies
execute
```shell
pip install python-configuration[aws]
```
* `GCPSecretManagerConfiguration` in `config.contrib.gcp`, which takes GCP Secret Manager
credentials into a `Configuration`-compatible instance. To install the needed dependencies
execute
```shell
pip install python-configuration[gcp]
```
## Features
* Load multiple configuration types
* Hierarchical configuration
* Ability to override with environment variables
* Merge parameters from different configuration types
## Contributing :tada:
If you'd like to contribute, please fork the repository and use a feature branch. Pull requests are welcome.
See [`CONTRIBUTING.md`](CONTRIBUTING.md) for the details.
## Links
- Repository: https://github.com/tr11/python-configuration
- Issue tracker: https://github.com/tr11/python-configuration/issues
- Documentation: https://python-configuration.readthedocs.io
## Licensing
The code in this project is licensed under MIT license.
Raw data
{
"_id": null,
"home_page": "https://github.com/tr11/python-configuration",
"name": "lamini-configuration",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6.2,<4.0.0",
"maintainer_email": "",
"keywords": "configuration,settings,json,yaml,toml,ini",
"author": "Tiago Requeijo",
"author_email": "tiago.requeijo.dev@gmail.com",
"download_url": "",
"platform": null,
"description": "# python-configuration\n> A library to load configuration parameters hierarchically from multiple sources and formats\n\n[](https://pypi.org/project/python-configuration/)\n\n\n\n[](https://travis-ci.org/tr11/python-configuration)\n[](https://codecov.io/gh/tr11/python-configuration)\n[](https://python-configuration.readthedocs.io/en/latest/?badge=latest)\n\nThis library is intended as a helper mechanism to load configuration files hierarchically. Supported format types are:\n\n* Python files\n* Dictionaries\n* Environment variables\n* Filesystem paths\n* JSON files\n* INI files\n* dotenv type files\n\nand optionally\n\n* YAML files\n* TOML files\n* Azure Key Vault credentials\n* AWS Secrets Manager credentials\n* GCP Secret Manager credentials\n\n## Installing\n\nTo install the library:\n\n```shell\npip install python-configuration\n```\n\nTo include the optional TOML and/or YAML loaders, install the optional dependencies `toml` and ` yaml`. For example,\n\n```shell\npip install python-configuration[toml,yaml]\n```\n\n## Getting started\n\n`python-configuration` converts the various config types into dictionaries with dotted-based keys. For example, given this JSON configuration\n\n```json\n{\n \"a\": {\n \"b\": \"value\"\n }\n}\n```\n\nWe can use the `config_from_json` method to parse it:\n\n```python\nfrom config import config_from_json\n\ncfg = config_from_json(\"my_config_file.json\", read_from_file=True)\n```\n\n(Similar methods exist for all the other supported configuration formats (eg. `config_from_toml`, etc.).)\n\nWe are then able to refer to the parameters in the config above using any of:\n\n```python\ncfg['a.b']\ncfg['a']['b']\ncfg['a'].b\ncfg.a.b\n```\n\nand extract specific data types such as dictionaries:\n\n```python\ncfg['a'].as_dict == {'b': 'value'}\n```\n\nThis is particularly useful in order to isolate group parameters.\nFor example, with the JSON configuration\n\n```json\n{\n \"database.host\": \"something\",\n \"database.port\": 12345,\n \"database.driver\": \"name\",\n \"app.debug\": true,\n \"app.environment\": \"development\",\n \"app.secrets\": \"super secret\",\n \"logging\": {\n \"service\": \"service\",\n \"token\": \"token\",\n \"tags\": \"tags\"\n }\n}\n```\n\none can retrieve the dictionaries as\n\n```python\ncfg.database.as_dict()\ncfg.app.as_dict()\ncfg.logging.as_dict()\n```\n\nor simply as\n\n```python\ndict(cfg.database)\ndict(cfg.app)\ndict(cfg.logging)\n```\n\n## Configuration\n\nThere are two general types of objects in this library. The first one is the `Configuration`, which represents a single config source. The second is a `ConfigurationSet` that allows for multiple `Configuration` objects to be specified.\n\n### Single Config\n\n#### Python Files\n\nTo load a configuration from a Python module, the `config_from_python` can be used.\nThe first parameter must be a Python module and can be specified as an absolute path to the Python file or as an importable module.\n\nOptional parameters are the `prefix` and `separator`. The following call\n\n```python\nconfig_from_python('foo.bar', prefix='CONFIG', separator='__')\n```\n\nwill read every variable in the `foo.bar` module that starts with `CONFIG__` and replace every occurrence of `__` with a `.`. For example,\n\n```python\n# foo.bar\nCONFIG__AA__BB_C = 1\nCONFIG__AA__BB__D = 2\nCONF__AA__BB__D = 3\n```\n\nwould result in the configuration\n\n```python\n{\n 'aa.bb_c': 1,\n 'aa.bb.d': 2,\n}\n```\n\nNote that the single underscore in `BB_C` is not replaced and the last line is not prefixed by `CONFIG`.\n\n#### Dictionaries\n\nDictionaries are loaded with `config_from_dict` and are converted internally to a flattened `dict`.\n\n```python\n{\n 'a': {\n 'b': 'value'\n }\n}\n```\n\nbecomes\n\n```python\n{\n 'a.b': 'value'\n}\n```\n\n#### Environment Variables\n\nEnvironment variables starting with `prefix` can be read with `config_from_env`:\n\n```python\nconfig_from_env(prefix, separator='_')\n```\n\n#### Filesystem Paths\n\nFolders with files named as `xxx.yyy.zzz` can be loaded with the `config_from_path` function. This format is useful to load mounted Kubernetes [ConfigMaps](https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#populate-a-volume-with-data-stored-in-a-configmap) or [Secrets](https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#create-a-pod-that-has-access-to-the-secret-data-through-a-volume).\n\n#### JSON, INI, .env, YAML, TOML\n\nJSON, INI, YAML, TOML files are loaded respectively with\n`config_from_json`,\n`config_from_ini`,\n`config_from_dotenv`,\n`config_from_yaml`, and\n`config_from_toml`.\nThe parameter `read_from_file` controls whether a string should be interpreted as a filename.\n\n###### Caveats\n\nIn order for `Configuration` objects to act as `dict` and allow the syntax `dict(cfg)`, the `keys()` method is implemented as the typical `dict` keys. If `keys` is an element in the configuration `cfg` then the `dict(cfg)` call will fail. In that case, it's necessary to use the `cfg.as_dict()` method to retrieve the `dict` representation for the `Configuration` object.\n\nThe same applies to the methods `values()` and `items()`.\n\n\n### Configuration Sets\n\nConfiguration sets are used to hierarchically load configurations and merge settings. Sets can be loaded by constructing a `ConfigurationSet` object directly or using the simplified `config` function.\n\nTo construct a `ConfigurationSet`, pass in as many of the simple `Configuration` objects as needed:\n\n```python\ncfg = ConfigurationSet(\n config_from_env(prefix=PREFIX),\n config_from_json(path, read_from_file=True),\n config_from_dict(DICT),\n)\n```\nThe example above will read first from Environment variables prefixed with `PREFIX`, and fallback first to the JSON file at `path`, and finally use the dictionary `DICT`.\n\nThe `config` function simplifies loading sets by assuming some defaults.\nThe example above can also be obtained by\n\n```python\ncfg = config(\n ('env', PREFIX),\n ('json', path, True),\n ('dict', DICT),\n)\n```\n\nor, even simpler if `path` points to a file with a `.json` suffix:\n\n```python\ncfg = config('env', path, DICT, prefix=PREFIX)\n```\n\nThe `config` function automatically detects the following:\n\n* extension `.py` for python modules\n* dot-separated python identifiers as a python module (e.g. `foo.bar`)\n* extension `.json` for JSON files\n* extension `.yaml` for YAML files\n* extension `.toml` for TOML files\n* extension `.ini` for INI files\n* extension `.env` for dotenv type files\n* filesystem folders as Filesystem Paths\n* the strings `env` or `environment` for Environment Variables\n\n#### Merging Values\n\n`ConfigurationSet` instances are constructed by inspecting each configuration source, taking into account nested dictionaries, and merging at the most granular level.\nFor example, the instance obtained from `cfg = config(d1, d2)` for the dictionaries below\n\n```python\nd1 = {'sub': {'a': 1, 'b': 4}}\nd2 = {'sub': {'b': 2, 'c': 3}}\n```\n\nis such that `cfg['sub']` equals\n\n```python\n{'a': 1, 'b': 4, 'c': 3}\n```\n\nNote that the nested dictionaries of `'sub'` in each of `d1` and `d2` do not overwrite each other, but are merged into a single dictionary with keys from both `d1` and `d2`, giving priority to the values of `d1` over those from `d2`.\n\n\n###### Caveats\n\nAs long as the data types are consistent across all the configurations that are part of a `ConfigurationSet`, the behavior should be straightforward. When different configuration objects are specified with competing data types, the first configuration to define the elements sets its datatype. For example, if in the example above `element` is interpreted as a `dict` from environment variables, but the JSON file specifies it as anything else besides a mapping, then the JSON value will be dropped automatically.\n\n## Other Features\n\n###### String Interpolation\n\nWhen setting the `interpolate` parameter in any `Configuration` instance, the library will perform a string interpolation step using the [str.format](https://docs.python.org/3/library/string.html#formatstrings) syntax. In particular, this allows to format configuration values automatically:\n\n```python\ncfg = config_from_dict({\n \"percentage\": \"{val:.3%}\",\n \"with_sign\": \"{val:+f}\",\n \"val\": 1.23456}, interpolate=True)\n\nassert cfg.val == 1.23456\nassert cfg.with_sign == \"+1.234560\"\nassert cfg.percentage == \"123.456%\"\n```\n\n## Extras\n\nThe `config.contrib` package contains extra implementations of the `Configuration` class used for special cases. Currently the following are implemented:\n\n* `AzureKeyVaultConfiguration` in `config.contrib.azure`, which takes Azure Key Vault\n credentials into a `Configuration`-compatible instance. To install the needed dependencies\n execute\n\n ```shell\n pip install python-configuration[azure]\n ```\n\n* `AWSSecretsManagerConfiguration` in `config.contrib.aws`, which takes AWS Secrets Manager\n credentials into a `Configuration`-compatible instance. To install the needed dependencies\n execute\n\n ```shell\n pip install python-configuration[aws]\n ```\n\n* `GCPSecretManagerConfiguration` in `config.contrib.gcp`, which takes GCP Secret Manager\n credentials into a `Configuration`-compatible instance. To install the needed dependencies\n execute\n\n ```shell\n pip install python-configuration[gcp]\n ```\n\n## Features\n\n* Load multiple configuration types\n* Hierarchical configuration\n* Ability to override with environment variables\n* Merge parameters from different configuration types\n\n## Contributing :tada:\n\nIf you'd like to contribute, please fork the repository and use a feature branch. Pull requests are welcome.\n\nSee [`CONTRIBUTING.md`](CONTRIBUTING.md) for the details.\n\n## Links\n\n- Repository: https://github.com/tr11/python-configuration\n- Issue tracker: https://github.com/tr11/python-configuration/issues\n- Documentation: https://python-configuration.readthedocs.io\n\n## Licensing\n\nThe code in this project is licensed under MIT license.\n\n",
"bugtrack_url": null,
"license": "",
"summary": "A library to load configuration parameters from multiple sources and formats",
"version": "0.8.3",
"project_urls": {
"Homepage": "https://github.com/tr11/python-configuration",
"Repository": "https://github.com/lamini-ai/lamini-configuration"
},
"split_keywords": [
"configuration",
"settings",
"json",
"yaml",
"toml",
"ini"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a2e7f8769b3b4e7250431b040a476414a20f915e3988470efde916f0a117ceb6",
"md5": "877191cb485ab575400896312a63bb0e",
"sha256": "670775ddb85d818428afffe840705e115ab360084c83c7c513d8e8eaed1b9dd0"
},
"downloads": -1,
"filename": "lamini_configuration-0.8.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "877191cb485ab575400896312a63bb0e",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6.2,<4.0.0",
"size": 22601,
"upload_time": "2023-07-17T21:56:31",
"upload_time_iso_8601": "2023-07-17T21:56:31.890613Z",
"url": "https://files.pythonhosted.org/packages/a2/e7/f8769b3b4e7250431b040a476414a20f915e3988470efde916f0a117ceb6/lamini_configuration-0.8.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-07-17 21:56:31",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "tr11",
"github_project": "python-configuration",
"travis_ci": true,
"coveralls": false,
"github_actions": false,
"lcname": "lamini-configuration"
}
JSON
