[](https://pepy.tech/project/pyaml-env)
[](https://github.com/mkaranasou/pyaml_env/actions/workflows/python-app.yml)
[](https://github.com/mkaranasou/pyaml_env/actions/workflows/codeql-analysis.yml)
[](https://opensource.org/licenses/MIT)
[](https://github.com/mkaranasou/pyaml_env/actions/workflows/python-publish.yml)
# Python YAML configuration with environment variables parsing
## TL;DR
A very small library that parses a yaml configuration file and it resolves the environment variables,
so that no secrets are kept in text.
### Install
```bash
pip install pyaml-env
```
### How to use:
---
#### Basic Usage: Environment variable parsing
This yaml file:
```yaml
database:
name: test_db
username: !ENV ${DB_USER}
password: !ENV ${DB_PASS}
url: !ENV 'http://${DB_BASE_URL}:${DB_PORT}'
```
given that we've set these:
```bash
export $DB_USER=super_secret_user
export $DB_PASS=extra_super_secret_password
export $DB_BASE_URL=localhost
export $DB_PORT=5432
```
becomes this:
```python
from pyaml_env import parse_config
config = parse_config('path/to/config.yaml')
print(config)
# outputs the following, with the environment variables resolved
{
'database': {
'name': 'test_db',
'username': 'super_secret_user',
'password': 'extra_super_secret_password',
'url': 'http://localhost:5432',
}
}
```
---
#### Attribute Access using `BaseConfig`
Which can also become this:
```python
from pyaml_env import parse_config, BaseConfig
config = BaseConfig(parse_config('path/to/config.yaml'))
# you can then access the config properties as atrributes
# I'll explain why this might be useful in a bit.
print(config.database.url)
```
---
#### Default Values with `:`
You can also set default values for when the environment variables are not set for some reason,
using the `default_sep` kwarg (**which is `:` by default**) like this:
```yaml
databse:
name: test_db
username: !ENV ${DB_USER:paws}
password: !ENV ${DB_PASS:meaw2}
url: !ENV 'http://${DB_BASE_URL:straight_to_production}:${DB_PORT}'
```
And if no environment variables are found then we get:
```python
from pyaml_env import parse_config
config = parse_config('path/to/config.yaml')
print(config)
{
'database': {
'name': 'test_db',
'username': 'paws',
'password': 'meaw2',
'url': 'http://straight_to_production:N/A',
}
}
```
**NOTE 0**: Special characters like `*`, `{` etc. are not currently supported as separators. Let me know if you'd like them handled also.
**NOTE 1**: If you set `tag` to `None`, then, the current behavior is that environment variables in all places in the yaml will be resolved (if set).
---
#### Datatype parsing with yaml's tag:yaml.org,2002:<datatype>
```python
# because this is not allowed:
# data1: !TAG !!float ${ENV_TAG2:27017}
# use tag:yaml.org,2002:datatype to convert value:
test_data = '''
data0: !TAG ${ENV_TAG1}
data1: !TAG tag:yaml.org,2002:float ${ENV_TAG2:27017}
data2: !!float 1024
data3: !TAG ${ENV_TAG2:some_value}
data4: !TAG tag:yaml.org,2002:bool ${ENV_TAG2:false}
'''
```
Will become:
```python
os.environ['ENV_TAG1'] = "1024"
config = parse_config(data=test_data, tag='!TAG')
print(config)
{
'data0': '1024',
'data1': 27017.0,
'data2': 1024.0,
'data3': 'some_value',
'data4': False
}
```
[reference in yaml code](https://github.com/yaml/pyyaml/blob/master/lib/yaml/parser.py#L78)
---
#### If nothing matches: `N/A` as `default_value`:
If no defaults are found and no environment variables, the `default_value` (**which is `N/A` by default**) is used:
```python
{
'database': {
'name': 'test_db',
'username': 'N/A',
'password': 'N/A',
'url': 'http://N/A:N/A',
}
}
```
Which, of course, means something went wrong and we need to set the correct environment variables.
If you want this process to fail if a *default value* is not found, you can set the `raise_if_na` flag to `True`.
For example:
```yaml
test1:
data0: !TEST ${ENV_TAG1:has_default}/somethingelse/${ENV_TAG2:also_has_default}
data1: !TEST ${ENV_TAG2}
```
will raise a `ValueError` because `data1: !TEST ${ENV_TAG2}` there is no default value for `ENV_TAG2` in this line.
---
#### Using a different loader:
The default yaml loader is `yaml.SafeLoader`. If you need to work with serialized Python objects,
you can specify a different loader.
So given a class:
```python
class OtherLoadTest:
def __init__(self):
self.data0 = 'it works!'
self.data1 = 'this works too!'
```
Which has become a yaml output like the following using `yaml.dump(OtherLoadTest())`:
```yaml
!!python/object:__main__.OtherLoadTest
data0: it works!
data1: this works too!
```
You can use `parse_config` to load the object like this:
```python
import yaml
from pyaml_env import parse_config
other_load_test = parse_config(path='path/to/config.yaml', loader=yaml.UnsafeLoader)
print(other_load_test)
<__main__.OtherLoadTest object at 0x7fc38ccd5470>
```
---
## Long story: Load a YAML configuration file and resolve any environment variables
If you’ve worked with Python projects, you’ve probably have stumbled across the many ways to provide configuration. I am not going to go through all the ways here, but a few of them are:
* using .ini files
* using a python class
* using .env files
* using JSON or XML files
* using a yaml file
And so on. I’ve put some useful links about the different ways below, in case you are interested in digging deeper.
My preference is working with yaml configuration because I usually find very handy and easy to use and I really like that yaml files are also used in e.g. docker-compose configuration so it is something most are familiar with.
For yaml parsing I use the [PyYAML](https://pyyaml.org/wiki/PyYAMLDocumentation) Python library.
In this article we’ll talk about the yaml file case and more specifically what you can do to **avoid keeping your secrets, e.g. passwords, hosts, usernames etc, directly on it**.
Let’s say we have a very simple example of a yaml file configuration:
database:
name: database_name
user: me
password: very_secret_and_complex
host: localhost
port: 5432
ws:
user: username
password: very_secret_and_complex_too
host: localhost
When you come to a point where you need to deploy your project, it is not really safe to have passwords and sensitive data in a plain text configuration file lying around on your production server. That’s where [**environment variables](https://medium.com/dataseries/hiding-secret-info-in-python-using-environment-variables-a2bab182eea) **come in handy. So the goal here is to be able to easily replace the very_secret_and_complex password with input from an environment variable, e.g. DB_PASS, so that this variable only exists when you set it and run your program instead of it being hardcoded somewhere.
For PyYAML to be able to resolve environment variables, we need three main things:
* A regex pattern for the environment variable identification e.g. pattern = re.compile(‘.*?\${(\w+)}.*?’)
* A tag that will signify that there’s an environment variable (or more) to be parsed, e.g. !ENV.
* And a function that the loader will use to resolve the environment variables
```python
def constructor_env_variables(loader, node):
"""
Extracts the environment variable from the node's value
:param yaml.Loader loader: the yaml loader
:param node: the current node in the yaml
:return: the parsed string that contains the value of the environment
variable
"""
value = loader.construct_scalar(node)
match = pattern.findall(value)
if match:
full_value = value
for g in match:
full_value = full_value.replace(
f'${{{g}}}', os.environ.get(g, g)
)
return full_value
return value
```
Example of a YAML configuration with environment variables:
database:
name: database_name
user: !ENV ${DB_USER}
password: !ENV ${DB_PASS}
host: !ENV ${DB_HOST}
port: 5432
ws:
user: !ENV ${WS_USER}
password: !ENV ${WS_PASS}
host: !ENV ‘[https://${CURR_ENV}.ws.com.local'](https://${CURR_ENV}.ws.com.local')
This can also work **with more than one environment variables** declared in the same line for the same configuration parameter like this:
ws:
user: !ENV ${WS_USER}
password: !ENV ${WS_PASS}
host: !ENV '[https://${CURR_ENV}.ws.com.](https://${CURR_ENV}.ws.com.local')[${MODE}](https://${CURR_ENV}.ws.com.local')' # multiple env var
And how to use this:
First set the environment variables. For example, for the DB_PASS :
export DB_PASS=very_secret_and_complex
Or even better, so that the password is not echoed in the terminal:
read -s ‘Database password: ‘ db_pass
export DB_PASS=$db_pass
```python
# To run this:
# export DB_PASS=very_secret_and_complex
# python use_env_variables_in_config_example.py -c /path/to/yaml
# do stuff with conf, e.g. access the database password like this: conf['database']['DB_PASS']
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='My awesome script')
parser.add_argument(
"-c", "--conf", action="store", dest="conf_file",
help="Path to config file"
)
args = parser.parse_args()
conf = parse_config(path=args.conf_file)
```
Then you can run the above script:
```bash
python use_env_variables_in_config_example.py -c /path/to/yaml
```
And in your code, do stuff with conf, e.g. access the database password like this: `conf['database']['DB_PASS']`
I hope this was helpful. Any thoughts, questions, corrections and suggestions are very welcome :)
## Useful links
[**The Many Faces and Files of Python Configs**
*As we cling harder and harder to Dockerfiles, Kubernetes, or any modern preconfigured app environment, our dependency…*hackersandslackers.com](https://hackersandslackers.com/simplify-your-python-projects-configuration/)
[**4 Ways to manage the configuration in Python**
*I’m not a native speaker. Sorry for my english. Please understand.*hackernoon.com](https://hackernoon.com/4-ways-to-manage-the-configuration-in-python-4623049e841b)
[**Python configuration files**
*A common need when writing an application is loading and saving configuration values in a human-readable text format…*www.devdungeon.com](https://www.devdungeon.com/content/python-configuration-files)
[**Configuration files in Python**
*Most interesting programs need some kind of configuration: Content Management Systems like WordPress blogs, WikiMedia…*martin-thoma.com](https://martin-thoma.com/configuration-files-in-python/)
<a href="https://www.buymeacoffee.com/mkaranasou" target="_blank" style="background: #40DCA5;"><img src="https://cdn.buymeacoffee.com/buttons/default-orange.png" alt="Buy Me A Coffee" height="41" width="174"></a>
Raw data
{
"_id": null,
"home_page": "https://github.com/mkaranasou/pyaml_env",
"name": "pyaml-env",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": "",
"keywords": "",
"author": "Maria Karanasou",
"author_email": "karanasou@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/28/72/7a65881cb83e8af22b36259436fb87507e9399e7251831dc9248303faaae/pyaml_env-1.2.1.tar.gz",
"platform": null,
"description": "[](https://pepy.tech/project/pyaml-env)\n[](https://github.com/mkaranasou/pyaml_env/actions/workflows/python-app.yml)\n[](https://github.com/mkaranasou/pyaml_env/actions/workflows/codeql-analysis.yml)\n[](https://opensource.org/licenses/MIT)\n[](https://github.com/mkaranasou/pyaml_env/actions/workflows/python-publish.yml)\n\n# Python YAML configuration with environment variables parsing\n\n## TL;DR\nA very small library that parses a yaml configuration file and it resolves the environment variables, \nso that no secrets are kept in text.\n\n### Install\n```bash\npip install pyaml-env\n```\n### How to use:\n\n--- \n\n#### Basic Usage: Environment variable parsing\nThis yaml file:\n```yaml\ndatabase:\n name: test_db\n username: !ENV ${DB_USER}\n password: !ENV ${DB_PASS}\n url: !ENV 'http://${DB_BASE_URL}:${DB_PORT}'\n```\ngiven that we've set these:\n```bash\nexport $DB_USER=super_secret_user\nexport $DB_PASS=extra_super_secret_password\nexport $DB_BASE_URL=localhost\nexport $DB_PORT=5432\n```\n\nbecomes this:\n```python\nfrom pyaml_env import parse_config\nconfig = parse_config('path/to/config.yaml')\n\nprint(config)\n# outputs the following, with the environment variables resolved\n{\n 'database': {\n 'name': 'test_db',\n 'username': 'super_secret_user',\n 'password': 'extra_super_secret_password',\n 'url': 'http://localhost:5432',\n }\n}\n\n```\n\n\n---\n\n#### Attribute Access using `BaseConfig`\nWhich can also become this:\n```python\nfrom pyaml_env import parse_config, BaseConfig\nconfig = BaseConfig(parse_config('path/to/config.yaml'))\n# you can then access the config properties as atrributes\n# I'll explain why this might be useful in a bit.\nprint(config.database.url)\n```\n---\n\n\n#### Default Values with `:`\nYou can also set default values for when the environment variables are not set for some reason,\nusing the `default_sep` kwarg (**which is `:` by default**) like this:\n```yaml\ndatabse:\n name: test_db\n username: !ENV ${DB_USER:paws}\n password: !ENV ${DB_PASS:meaw2}\n url: !ENV 'http://${DB_BASE_URL:straight_to_production}:${DB_PORT}'\n```\n\nAnd if no environment variables are found then we get:\n```python\nfrom pyaml_env import parse_config\nconfig = parse_config('path/to/config.yaml')\n\nprint(config)\n{\n 'database': {\n 'name': 'test_db',\n 'username': 'paws',\n 'password': 'meaw2',\n 'url': 'http://straight_to_production:N/A',\n }\n}\n```\n**NOTE 0**: Special characters like `*`, `{` etc. are not currently supported as separators. Let me know if you'd like them handled also.\n\n**NOTE 1**: If you set `tag` to `None`, then, the current behavior is that environment variables in all places in the yaml will be resolved (if set).\n\n---\n#### Datatype parsing with yaml's tag:yaml.org,2002:<datatype>\n\n```python\n# because this is not allowed:\n# data1: !TAG !!float ${ENV_TAG2:27017}\n# use tag:yaml.org,2002:datatype to convert value:\ntest_data = '''\n data0: !TAG ${ENV_TAG1}\n data1: !TAG tag:yaml.org,2002:float ${ENV_TAG2:27017}\n data2: !!float 1024\n data3: !TAG ${ENV_TAG2:some_value}\n data4: !TAG tag:yaml.org,2002:bool ${ENV_TAG2:false}\n '''\n```\nWill become:\n```python\nos.environ['ENV_TAG1'] = \"1024\"\nconfig = parse_config(data=test_data, tag='!TAG')\nprint(config)\n{\n 'data0': '1024', \n 'data1': 27017.0, \n 'data2': 1024.0, \n 'data3': 'some_value', \n 'data4': False\n}\n```\n\n[reference in yaml code](https://github.com/yaml/pyyaml/blob/master/lib/yaml/parser.py#L78)\n\n---\n#### If nothing matches: `N/A` as `default_value`:\n\nIf no defaults are found and no environment variables, the `default_value` (**which is `N/A` by default**) is used:\n```python\n{\n 'database': {\n 'name': 'test_db',\n 'username': 'N/A',\n 'password': 'N/A',\n 'url': 'http://N/A:N/A',\n }\n}\n```\nWhich, of course, means something went wrong and we need to set the correct environment variables.\nIf you want this process to fail if a *default value* is not found, you can set the `raise_if_na` flag to `True`.\nFor example:\n\n```yaml\ntest1:\n data0: !TEST ${ENV_TAG1:has_default}/somethingelse/${ENV_TAG2:also_has_default}\n data1: !TEST ${ENV_TAG2}\n```\nwill raise a `ValueError` because `data1: !TEST ${ENV_TAG2}` there is no default value for `ENV_TAG2` in this line.\n\n--- \n\n\n#### Using a different loader:\n\nThe default yaml loader is `yaml.SafeLoader`. If you need to work with serialized Python objects, \nyou can specify a different loader.\n\nSo given a class:\n```python\nclass OtherLoadTest:\n def __init__(self):\n self.data0 = 'it works!'\n self.data1 = 'this works too!'\n\n```\n\nWhich has become a yaml output like the following using `yaml.dump(OtherLoadTest())`:\n```yaml\n!!python/object:__main__.OtherLoadTest\ndata0: it works!\ndata1: this works too!\n```\n\nYou can use `parse_config` to load the object like this:\n```python\nimport yaml\nfrom pyaml_env import parse_config\n\nother_load_test = parse_config(path='path/to/config.yaml', loader=yaml.UnsafeLoader)\nprint(other_load_test)\n<__main__.OtherLoadTest object at 0x7fc38ccd5470>\n```\n---\n\n## Long story: Load a YAML configuration file and resolve any environment variables\n\n\n\nIf you\u2019ve worked with Python projects, you\u2019ve probably have stumbled across the many ways to provide configuration. I am not going to go through all the ways here, but a few of them are:\n\n* using .ini files\n\n* using a python class\n\n* using .env files\n\n* using JSON or XML files\n\n* using a yaml file\n\nAnd so on. I\u2019ve put some useful links about the different ways below, in case you are interested in digging deeper.\n\nMy preference is working with yaml configuration because I usually find very handy and easy to use and I really like that yaml files are also used in e.g. docker-compose configuration so it is something most are familiar with.\n\nFor yaml parsing I use the [PyYAML](https://pyyaml.org/wiki/PyYAMLDocumentation) Python library.\n\nIn this article we\u2019ll talk about the yaml file case and more specifically what you can do to **avoid keeping your secrets, e.g. passwords, hosts, usernames etc, directly on it**.\n\nLet\u2019s say we have a very simple example of a yaml file configuration:\n\n database:\n name: database_name\n user: me\n password: very_secret_and_complex\n host: localhost\n port: 5432\n\n ws:\n user: username\n password: very_secret_and_complex_too\n host: localhost\n\nWhen you come to a point where you need to deploy your project, it is not really safe to have passwords and sensitive data in a plain text configuration file lying around on your production server. That\u2019s where [**environment variables](https://medium.com/dataseries/hiding-secret-info-in-python-using-environment-variables-a2bab182eea) **come in handy. So the goal here is to be able to easily replace the very_secret_and_complex password with input from an environment variable, e.g. DB_PASS, so that this variable only exists when you set it and run your program instead of it being hardcoded somewhere.\n\nFor PyYAML to be able to resolve environment variables, we need three main things:\n\n* A regex pattern for the environment variable identification e.g. pattern = re.compile(\u2018.*?\\${(\\w+)}.*?\u2019)\n\n* A tag that will signify that there\u2019s an environment variable (or more) to be parsed, e.g. !ENV.\n\n* And a function that the loader will use to resolve the environment variables\n\n```python\ndef constructor_env_variables(loader, node):\n \"\"\"\n Extracts the environment variable from the node's value\n :param yaml.Loader loader: the yaml loader\n :param node: the current node in the yaml\n :return: the parsed string that contains the value of the environment\n variable\n \"\"\"\n value = loader.construct_scalar(node)\n match = pattern.findall(value)\n if match:\n full_value = value\n for g in match:\n full_value = full_value.replace(\n f'${{{g}}}', os.environ.get(g, g)\n )\n return full_value\n return value\n```\n\nExample of a YAML configuration with environment variables:\n\n database:\n name: database_name\n user: !ENV ${DB_USER}\n password: !ENV ${DB_PASS}\n host: !ENV ${DB_HOST}\n port: 5432\n\n ws:\n user: !ENV ${WS_USER}\n password: !ENV ${WS_PASS}\n host: !ENV \u2018[https://${CURR_ENV}.ws.com.local'](https://${CURR_ENV}.ws.com.local')\n\nThis can also work **with more than one environment variables** declared in the same line for the same configuration parameter like this:\n\n ws:\n user: !ENV ${WS_USER}\n password: !ENV ${WS_PASS}\n host: !ENV '[https://${CURR_ENV}.ws.com.](https://${CURR_ENV}.ws.com.local')[${MODE}](https://${CURR_ENV}.ws.com.local')' # multiple env var\n\nAnd how to use this:\n\nFirst set the environment variables. For example, for the DB_PASS :\n\n export DB_PASS=very_secret_and_complex\n\nOr even better, so that the password is not echoed in the terminal:\n\n read -s \u2018Database password: \u2018 db_pass\n export DB_PASS=$db_pass\n\n```python\n\n# To run this:\n# export DB_PASS=very_secret_and_complex \n# python use_env_variables_in_config_example.py -c /path/to/yaml\n# do stuff with conf, e.g. access the database password like this: conf['database']['DB_PASS']\n\nif __name__ == '__main__':\n parser = argparse.ArgumentParser(description='My awesome script')\n parser.add_argument(\n \"-c\", \"--conf\", action=\"store\", dest=\"conf_file\",\n help=\"Path to config file\"\n )\n args = parser.parse_args()\n conf = parse_config(path=args.conf_file)\n```\n\n\nThen you can run the above script:\n```bash\npython use_env_variables_in_config_example.py -c /path/to/yaml\n```\n\nAnd in your code, do stuff with conf, e.g. access the database password like this: `conf['database']['DB_PASS']`\n\nI hope this was helpful. Any thoughts, questions, corrections and suggestions are very welcome :)\n\n## Useful links\n[**The Many Faces and Files of Python Configs**\n*As we cling harder and harder to Dockerfiles, Kubernetes, or any modern preconfigured app environment, our dependency\u2026*hackersandslackers.com](https://hackersandslackers.com/simplify-your-python-projects-configuration/)\n[**4 Ways to manage the configuration in Python**\n*I\u2019m not a native speaker. Sorry for my english. Please understand.*hackernoon.com](https://hackernoon.com/4-ways-to-manage-the-configuration-in-python-4623049e841b)\n[**Python configuration files**\n*A common need when writing an application is loading and saving configuration values in a human-readable text format\u2026*www.devdungeon.com](https://www.devdungeon.com/content/python-configuration-files)\n[**Configuration files in Python**\n*Most interesting programs need some kind of configuration: Content Management Systems like WordPress blogs, WikiMedia\u2026*martin-thoma.com](https://martin-thoma.com/configuration-files-in-python/)\n\n\n\n<a href=\"https://www.buymeacoffee.com/mkaranasou\" target=\"_blank\" style=\"background: #40DCA5;\"><img src=\"https://cdn.buymeacoffee.com/buttons/default-orange.png\" alt=\"Buy Me A Coffee\" height=\"41\" width=\"174\"></a>\n",
"bugtrack_url": null,
"license": "",
"summary": "Provides yaml file parsing with environment variable resolution",
"version": "1.2.1",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"md5": "c8ec74ab33001334b65468ade2a41065",
"sha256": "2e7da2d4bba0629711ade1a41864e5e200c84ded896a3d27e9f560fae7311c36"
},
"downloads": -1,
"filename": "pyaml_env-1.2.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c8ec74ab33001334b65468ade2a41065",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 9038,
"upload_time": "2022-12-06T09:36:52",
"upload_time_iso_8601": "2022-12-06T09:36:52.974065Z",
"url": "https://files.pythonhosted.org/packages/d7/9b/eb5d41a8ec8891ca85bac75b409d83f116b852fd46520f19c24464cba032/pyaml_env-1.2.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"md5": "cd3e8a292708cc64dc8789f0c02c8d83",
"sha256": "6d5dc98c8c82df743a132c196e79963050c9feb05b0a6f25f3ad77771d3d95b0"
},
"downloads": -1,
"filename": "pyaml_env-1.2.1.tar.gz",
"has_sig": false,
"md5_digest": "cd3e8a292708cc64dc8789f0c02c8d83",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 12759,
"upload_time": "2022-12-06T09:36:54",
"upload_time_iso_8601": "2022-12-06T09:36:54.704915Z",
"url": "https://files.pythonhosted.org/packages/28/72/7a65881cb83e8af22b36259436fb87507e9399e7251831dc9248303faaae/pyaml_env-1.2.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2022-12-06 09:36:54",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "mkaranasou",
"github_project": "pyaml_env",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "PyYAML",
"specs": [
[
">=",
"5.0"
],
[
"<=",
"7.0"
]
]
}
],
"lcname": "pyaml-env"
}
JSON
