- [Introduction](#introduction)
- [Documentation](#documentation)
- [Quick Start](#quick-start)
* [Install](#install)
* [Using It](#using-it)
- [Quick Overview](#quick-overview)
* [Places Configuration is Retrieved From](#places-configuration-is-retrieved-from)
+ [Param Store Provider Specifics](#param-store-provider-specifics)
+ [Secrets Manager Provider Specifics](#secrets-manager-provider-specifics)
* [Case Sensitivity](#case-sensitivity)
* [Add Permissions](#add-permissions)
* [Caching](#caching)
+ [Time to live](#time-to-live)
+ [Table Layout Details](#table-layout-details)
* [Unit Tests](#unit-tests)
- [Licensing](#licensing)
# Introduction
Helps retrieve configuration information from aws/boto services such as Ssm's Param Store and Secrets Manager,
with the ability the cache a flattened list into a dynamodb table.
Right now this is **pre-release software**, as the dynamo cache table and related need further documentation and testing.
Retrieving values from Param Store and Secrets Manager should work and be relatively fast, as we bulk-grab values
at the various directory-levels that are checked.
**More documentation and testing will be coming soon, for a full 1.0.0 release sometime in the next month or so.**
See **[xcon docs](https://xyngular.github.io/py-xcon/latest/)**.
# Documentation
**[📄 Detailed Documentation](https://xyngular.github.io/py-xcon/latest/)** | **[🐍 PyPi](https://pypi.org/project/xcon/)**
# Quick Start
## Install
```bash
# via pip
pip install xcon
# via poetry
poetry add xcon
```
## Using It
From the get-go and by default, environmental variables will 'just work'.
The main class is `Config` via `from xcon import Config`.
This class uses [xinject](https://pypi.org/project/xinject/) to do dependency injection.
You can easily inject a new version/configuration of the object without having to couple you code too close
together to get your configuration settings.
You get the current Config object via:
```python
from xcon import Config
current_config = Config.grab()
setting_value = current_config.get('some_setting')
```
An easier way to always use the current Config object is to use a proxy object.
```python
# Instead of importing the class, we import a proxy to the currently injected instance:
from xcon import config
setting_value = config.get('some_setting')
```
Alternatively, ou can also use `Config.proxy()` to get a proxy.
```python
# Importing proxy object to current Config injectable dependency.
# You can use it as if you did `Config.grab()`, as it does this
# for you each time you get something from it.
from xcon import config
import os
# Setting a environmental variable value to showcase retrieving it.
os.environ['SOME_CONFIG_VARIABLE'] = "my-value"
# If you had an environmental variable called `SOME_CONFIG_VARIABLE`, this would find it:
assert config.get('some_config_variable') == "my-value"
# Alternate 'dict; syntax, works just like you would expect.
# Just like dict, it will raise an exception if value not found.
assert config['some_config_variable'] == "my-value"
```
Config names are case-insensitive (although directory names are case-sensitive).
By default, [`Config`](api/xcon/config.html#xcon.config.Config){target=_blank} will look at environmental variables first,
and then other remote places second (the order and where to look is all configurable).
# Quick Overview
## Places Configuration is Retrieved From
As a side note for the below paths the `SERVICE_NAME` and `APP_ENV`variables
come from `xcon.xcon_settings.environment` and `xcon.xcon_settings.service`.
By default, these settings will use the `SERVICE_NAME` and `APP_ENV`
environmental variables. You can also set/override them expiclity
by setting a value one `xcon.xcon_settings.environment` and/or `xcon.xcon_settings.service`.
By default, Config will look in these paths (first).
- /{APP_NAME}/{APP_ENV}/{variable_name}
- /{APP_NAME}/all/{variable_name}
- /global/{APP_ENV}/{variable_name}
- /global/all/{variable_name}
- Details:
- [Standard Directory Paths](#standard-directory-paths)
- [Directory Chain](#directory-chain)
For each directory/path, we go through these providers (second):
1. Environmental Variables
2. Dynamo Config Cache
- Will be skipped if table/permissions don't exist.
3. AWS Secrets Manager
- Will be skipped if needed permissions not in place
4. AWS Param Store
- Will be skipped if needed permissions not in place
- Details:
- [Provider Chain](#provider-chain)
- [Supported Providers](#supported-providers)
** TODO In the order they are specified above (see [Standard Lookup Order](https://xyngular.github.io/py-xcon/latest/detailed_overview/#standard-lookup-order)).
### Param Store Provider Specifics
Values are exclusively retrieved via "GetParametersByPath"; which allows for bulk-retrieval of settings.
All settings in a particular directory are retrieved in one request, and then whatever value is needed is returned.
These values are cached within the provider retriever object, so when other config values are asked for
there is a good chance it can return a value without having to go back to param store to ask for another value.
### Secrets Manager Provider Specifics
Secrets manager does not allow for bulk-retrieval of values.
Instead, you can bulk-request get a list of available secret names via `ListSecretVersionIds`.
The secrets provider will grab the full list, and then use that to know what is or is not available to get.
This makes it much faster, as it can quickly determine if it should attempt to retrieve a value or not based on this list.
## Case Sensitivity
The directory/path is case-sensitive; but the `VARIABLE_NAME` part at the end is case-insensitive.
So environmental variables are entirely case-insensitive, as they only have the `VAIRABLE_NAME`
and no directory path.
So you can do `config.get('some_var_name')`, and it would still find a value for it,
even if the name in the source/provider of values is `SOME_VAR_NAME`.
## Add Permissions
If you want to receive values from remote locations, the app will need the correct permissions.
For example, AWS's Param Store service will restrict access to the param values by path/directory.
There is a serverless permission resource template yaml file you can use directory or copy
and change as needed for your purposes.
- AWS Configuration Store Permissions:
- [Param Store](https://github.com/xyngular/py-xcon/blob/main/xcon/serverless_files/ssm-permissions.yml)
- [Secrets Manager](https://github.com/xyngular/py-xcon/blob/main/xcon/serverless_files/secrets-permissions.yml)
If you want to use a dynamo table cache (see [caching](#caching) in next section), use these:
- DynamoDB Cache Table:
- [App Permissions](https://github.com/xyngular/py-xcon/blob/main/xcon/serverless_files/config_manager/cache-table.yml)
- [Table Definition](https://github.com/xyngular/py-xcon/blob/main/xcon/serverless_files/cache-permissions.yml)
For more details, see [Permissions](https://xyngular.github.io/py-xcon/latest/detailed_overview/#permissions).
## Caching
The purpose of the cache is to limit calls to the providers,
to prevent throttling from them.
For example, the AWS Param Store will throttle calls if there are too many per-second,
which could happen if several lambdas get launched and each lookup configuration simultaneously.
By default, values that are remotely looked up (ie: non-environmental variables)
are cached in a dynamo table.
Each of these lambdas can first check a DynamoDB cache table first, and if the value they need
is in there it will use that instead of attempting to retrieve values from the providers.
When something is not in the cache table, Config will look at each configured provider
and when it finds the value (or lack of a value), it will store what it found in the
dynamo cache table for later faster lookup.
The cache is a flattened list of all resolved values from all configured sources.
It will correctly cache according to the current providers, paths, and app environment + service.
Any of these variables can dynamically change, this information is added to each cached
entry so the correct value will be used in any situation.
### Time to live
The cache table is configured with a time-to-live attribute (named `ttl`).
The value is set for 12 hours, after which the item will expire.
There is an logarithm built into `Config` caching mechanism that will
pre-expire items sooner than normal randomly.
The algorith makes it more likely a particular item in the cache will expire
sooner as the expire-time approaches.
This means something that will expire in one hour will be more likely to
be pre-expired than something that has 10 hours left.
This helps ensure that if a lambda is very busy and has many concurrent
instances running that it's likely only one of the lambdas would pre-expire
the cached items and 'refresh' them by re-looking up the values from the
providers and re-caching the newly looked up values.
This is a way to coordinate cache expiring and refreshing without
having to actually have any coordinating communication happening.
This allows the configuration refreshing to automatically scale
with the lambda activity in such a way as to limit the possibility
of being throttled from param store or secrets manager.
### Table Layout Details
The dynamo table has a two-part primary key.
The first part of the primary key is a hash key made up of apps
`xcon.xcon_settings.environment` and `xcon.xcon_settings.service` values.
This is the 'partition' key in the DynamoDB table, and AWS policies can allow
or deny access based on this hash key.
This allows the table to limit access to cached items by app's environment + service.
The second part of the primary key is a range-key made up of all provider names
and directory paths in the order they are looked up in. This allows multiple
values to be stored for the same config setting, depending on which providers
and directory paths were used to lookup the config setting.
This allows all looked up values for all dynamic situations to be cached
and used correctly.
For details see [Caching Details](#caching-details), [Historical Background](#historical-background).
## Unit Tests
By default, unit tests will always start with a Config object that has caching disabled,
and only uses the environmental provider (ie: only looks at environmental variables).
This is accomplished via an autouse fixture in a pytest plugin module
(see plugin module `xcon.pytest_plugin` or fixture `xcon.pytest_plugin.xcon`).
If a project has `xcon` as a dependency, pytest will find this plugin module
and automatically use it. Nothing more to do.
As an FYI/side-note: There is a `xinject.pytest_plugin.xyn_context` that will also
automatically configure a blank context for each unit test.
This does mean you must configure Config using a fixture or at the top of your unit test method,
as any changes at the module-level will be forgotten.
The reason we do this is it guarantees that resources/config changes won't be propagaed/leak
into another unit test.
The end result is there is need to worry about these basics,
as they are taken care of for you automatically as long as the library is installed
as a dependency.
# Licensing
This library is licensed under the "The Unlicense" License. See the LICENSE file.
Raw data
{
"_id": null,
"home_page": "https://github.com/xyngular/py-xcon",
"name": "xcon",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.11,<4.0",
"maintainer_email": "",
"keywords": "settings,configuration,lazy,boto,aws,secrets manager,param store,ssm",
"author": "Josh Orr",
"author_email": "josh@orr.blue",
"download_url": "https://files.pythonhosted.org/packages/25/bc/8cf641ba3385e687a381f0a6030a6630d78103f94ed54b527f31edaf9e81/xcon-0.8.0.tar.gz",
"platform": null,
"description": "\n\n\n- [Introduction](#introduction)\n- [Documentation](#documentation)\n- [Quick Start](#quick-start)\n * [Install](#install)\n * [Using It](#using-it)\n- [Quick Overview](#quick-overview)\n * [Places Configuration is Retrieved From](#places-configuration-is-retrieved-from)\n + [Param Store Provider Specifics](#param-store-provider-specifics)\n + [Secrets Manager Provider Specifics](#secrets-manager-provider-specifics)\n * [Case Sensitivity](#case-sensitivity)\n * [Add Permissions](#add-permissions)\n * [Caching](#caching)\n + [Time to live](#time-to-live)\n + [Table Layout Details](#table-layout-details)\n * [Unit Tests](#unit-tests)\n- [Licensing](#licensing)\n\n# Introduction\n\nHelps retrieve configuration information from aws/boto services such as Ssm's Param Store and Secrets Manager,\nwith the ability the cache a flattened list into a dynamodb table.\n\nRight now this is **pre-release software**, as the dynamo cache table and related need further documentation and testing.\n\nRetrieving values from Param Store and Secrets Manager should work and be relatively fast, as we bulk-grab values\nat the various directory-levels that are checked.\n\n**More documentation and testing will be coming soon, for a full 1.0.0 release sometime in the next month or so.**\n\nSee **[xcon docs](https://xyngular.github.io/py-xcon/latest/)**.\n\n# Documentation\n\n**[\ud83d\udcc4 Detailed Documentation](https://xyngular.github.io/py-xcon/latest/)** | **[\ud83d\udc0d PyPi](https://pypi.org/project/xcon/)**\n\n\n# Quick Start\n\n## Install\n\n```bash\n# via pip\npip install xcon\n\n# via poetry\npoetry add xcon\n```\n\n\n## Using It\n\nFrom the get-go and by default, environmental variables will 'just work'.\n\nThe main class is `Config` via `from xcon import Config`.\n\nThis class uses [xinject](https://pypi.org/project/xinject/) to do dependency injection.\nYou can easily inject a new version/configuration of the object without having to couple you code too close\ntogether to get your configuration settings.\n\nYou get the current Config object via:\n\n```python\nfrom xcon import Config\n\ncurrent_config = Config.grab()\nsetting_value = current_config.get('some_setting')\n```\n\nAn easier way to always use the current Config object is to use a proxy object.\n\n```python\n# Instead of importing the class, we import a proxy to the currently injected instance:\nfrom xcon import config\n\nsetting_value = config.get('some_setting')\n```\n\nAlternatively, ou can also use `Config.proxy()` to get a proxy.\n\n```python\n# Importing proxy object to current Config injectable dependency.\n# You can use it as if you did `Config.grab()`, as it does this\n# for you each time you get something from it.\nfrom xcon import config\nimport os\n\n# Setting a environmental variable value to showcase retrieving it.\nos.environ['SOME_CONFIG_VARIABLE'] = \"my-value\"\n\n# If you had an environmental variable called `SOME_CONFIG_VARIABLE`, this would find it:\nassert config.get('some_config_variable') == \"my-value\"\n\n# Alternate 'dict; syntax, works just like you would expect.\n# Just like dict, it will raise an exception if value not found.\nassert config['some_config_variable'] == \"my-value\"\n```\n\nConfig names are case-insensitive (although directory names are case-sensitive).\n\nBy default, [`Config`](api/xcon/config.html#xcon.config.Config){target=_blank} will look at environmental variables first,\nand then other remote places second (the order and where to look is all configurable).\n\n# Quick Overview\n\n## Places Configuration is Retrieved From\n\nAs a side note for the below paths the `SERVICE_NAME` and `APP_ENV`variables\ncome from `xcon.xcon_settings.environment` and `xcon.xcon_settings.service`.\nBy default, these settings will use the `SERVICE_NAME` and `APP_ENV`\nenvironmental variables. You can also set/override them expiclity\nby setting a value one `xcon.xcon_settings.environment` and/or `xcon.xcon_settings.service`.\n\nBy default, Config will look in these paths (first).\n\n- /{APP_NAME}/{APP_ENV}/{variable_name}\n- /{APP_NAME}/all/{variable_name}\n- /global/{APP_ENV}/{variable_name}\n- /global/all/{variable_name}\n- Details:\n - [Standard Directory Paths](#standard-directory-paths) \n - [Directory Chain](#directory-chain)\n\nFor each directory/path, we go through these providers (second):\n\n1. Environmental Variables\n2. Dynamo Config Cache\n - Will be skipped if table/permissions don't exist.\n3. AWS Secrets Manager\n - Will be skipped if needed permissions not in place\n4. AWS Param Store\n - Will be skipped if needed permissions not in place\n- Details:\n - [Provider Chain](#provider-chain)\n - [Supported Providers](#supported-providers)\n\n** TODO In the order they are specified above (see [Standard Lookup Order](https://xyngular.github.io/py-xcon/latest/detailed_overview/#standard-lookup-order)).\n\n### Param Store Provider Specifics\n\nValues are exclusively retrieved via \"GetParametersByPath\"; which allows for bulk-retrieval of settings.\n\nAll settings in a particular directory are retrieved in one request, and then whatever value is needed is returned.\nThese values are cached within the provider retriever object, so when other config values are asked for\nthere is a good chance it can return a value without having to go back to param store to ask for another value.\n\n### Secrets Manager Provider Specifics\n\nSecrets manager does not allow for bulk-retrieval of values.\nInstead, you can bulk-request get a list of available secret names via `ListSecretVersionIds`.\n\nThe secrets provider will grab the full list, and then use that to know what is or is not available to get.\nThis makes it much faster, as it can quickly determine if it should attempt to retrieve a value or not based on this list.\n\n## Case Sensitivity\n\nThe directory/path is case-sensitive; but the `VARIABLE_NAME` part at the end is case-insensitive.\n\nSo environmental variables are entirely case-insensitive, as they only have the `VAIRABLE_NAME`\nand no directory path.\n\nSo you can do `config.get('some_var_name')`, and it would still find a value for it,\neven if the name in the source/provider of values is `SOME_VAR_NAME`.\n\n## Add Permissions\n\nIf you want to receive values from remote locations, the app will need the correct permissions.\n\nFor example, AWS's Param Store service will restrict access to the param values by path/directory.\n\nThere is a serverless permission resource template yaml file you can use directory or copy\nand change as needed for your purposes.\n\n- AWS Configuration Store Permissions:\n - [Param Store](https://github.com/xyngular/py-xcon/blob/main/xcon/serverless_files/ssm-permissions.yml)\n - [Secrets Manager](https://github.com/xyngular/py-xcon/blob/main/xcon/serverless_files/secrets-permissions.yml)\n\nIf you want to use a dynamo table cache (see [caching](#caching) in next section), use these:\n\n- DynamoDB Cache Table:\n - [App Permissions](https://github.com/xyngular/py-xcon/blob/main/xcon/serverless_files/config_manager/cache-table.yml)\n - [Table Definition](https://github.com/xyngular/py-xcon/blob/main/xcon/serverless_files/cache-permissions.yml)\n\nFor more details, see [Permissions](https://xyngular.github.io/py-xcon/latest/detailed_overview/#permissions).\n\n\n## Caching\n\nThe purpose of the cache is to limit calls to the providers,\nto prevent throttling from them.\n\nFor example, the AWS Param Store will throttle calls if there are too many per-second,\nwhich could happen if several lambdas get launched and each lookup configuration simultaneously.\n\nBy default, values that are remotely looked up (ie: non-environmental variables)\nare cached in a dynamo table.\nEach of these lambdas can first check a DynamoDB cache table first, and if the value they need\nis in there it will use that instead of attempting to retrieve values from the providers.\n\nWhen something is not in the cache table, Config will look at each configured provider\nand when it finds the value (or lack of a value), it will store what it found in the\ndynamo cache table for later faster lookup.\n\nThe cache is a flattened list of all resolved values from all configured sources.\nIt will correctly cache according to the current providers, paths, and app environment + service.\nAny of these variables can dynamically change, this information is added to each cached\nentry so the correct value will be used in any situation.\n\n### Time to live\n\nThe cache table is configured with a time-to-live attribute (named `ttl`).\nThe value is set for 12 hours, after which the item will expire.\n\nThere is an logarithm built into `Config` caching mechanism that will\npre-expire items sooner than normal randomly.\nThe algorith makes it more likely a particular item in the cache will expire\nsooner as the expire-time approaches.\n\nThis means something that will expire in one hour will be more likely to\nbe pre-expired than something that has 10 hours left.\n\nThis helps ensure that if a lambda is very busy and has many concurrent\ninstances running that it's likely only one of the lambdas would pre-expire\nthe cached items and 'refresh' them by re-looking up the values from the\nproviders and re-caching the newly looked up values.\n\nThis is a way to coordinate cache expiring and refreshing without\nhaving to actually have any coordinating communication happening.\n\nThis allows the configuration refreshing to automatically scale\nwith the lambda activity in such a way as to limit the possibility\nof being throttled from param store or secrets manager.\n\n### Table Layout Details\n\nThe dynamo table has a two-part primary key.\n\nThe first part of the primary key is a hash key made up of apps\n`xcon.xcon_settings.environment` and `xcon.xcon_settings.service` values.\nThis is the 'partition' key in the DynamoDB table, and AWS policies can allow\nor deny access based on this hash key.\nThis allows the table to limit access to cached items by app's environment + service.\n\nThe second part of the primary key is a range-key made up of all provider names\nand directory paths in the order they are looked up in. This allows multiple\nvalues to be stored for the same config setting, depending on which providers\nand directory paths were used to lookup the config setting.\n\nThis allows all looked up values for all dynamic situations to be cached\nand used correctly.\n\nFor details see [Caching Details](#caching-details), [Historical Background](#historical-background).\n\n## Unit Tests\n\nBy default, unit tests will always start with a Config object that has caching disabled,\nand only uses the environmental provider (ie: only looks at environmental variables).\n\nThis is accomplished via an autouse fixture in a pytest plugin module\n(see plugin module `xcon.pytest_plugin` or fixture `xcon.pytest_plugin.xcon`).\n\nIf a project has `xcon` as a dependency, pytest will find this plugin module\nand automatically use it. Nothing more to do.\n\nAs an FYI/side-note: There is a `xinject.pytest_plugin.xyn_context` that will also\nautomatically configure a blank context for each unit test.\n\nThis does mean you must configure Config using a fixture or at the top of your unit test method,\nas any changes at the module-level will be forgotten.\n\nThe reason we do this is it guarantees that resources/config changes won't be propagaed/leak\ninto another unit test.\n\nThe end result is there is need to worry about these basics,\nas they are taken care of for you automatically as long as the library is installed\nas a dependency.\n\n# Licensing\n\nThis library is licensed under the \"The Unlicense\" License. See the LICENSE file.\n",
"bugtrack_url": null,
"license": "",
"summary": "Dynamic configuration retreiver.",
"version": "0.8.0",
"project_urls": {
"Homepage": "https://github.com/xyngular/py-xcon",
"Repository": "https://github.com/xyngular/py-xcon"
},
"split_keywords": [
"settings",
"configuration",
"lazy",
"boto",
"aws",
"secrets manager",
"param store",
"ssm"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "030b9ca50f1fde0ff9d976a360fa68a14391dff6f49d0ec2539bc78d6432b215",
"md5": "d8afeaf6b6e5600be50676e1be803a4f",
"sha256": "15c44b0cc7353dbdad1c0d769cabca32cece9ea018fe7941fd88fb626dbbf202"
},
"downloads": -1,
"filename": "xcon-0.8.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d8afeaf6b6e5600be50676e1be803a4f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.11,<4.0",
"size": 69123,
"upload_time": "2024-01-03T21:14:25",
"upload_time_iso_8601": "2024-01-03T21:14:25.357854Z",
"url": "https://files.pythonhosted.org/packages/03/0b/9ca50f1fde0ff9d976a360fa68a14391dff6f49d0ec2539bc78d6432b215/xcon-0.8.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "25bc8cf641ba3385e687a381f0a6030a6630d78103f94ed54b527f31edaf9e81",
"md5": "39182db7e8d2e8049586002491d88522",
"sha256": "6a34d7c337849db01f0f90da45d47db90f541c1c825a10aea9f8d4e32d54eee8"
},
"downloads": -1,
"filename": "xcon-0.8.0.tar.gz",
"has_sig": false,
"md5_digest": "39182db7e8d2e8049586002491d88522",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.11,<4.0",
"size": 62415,
"upload_time": "2024-01-03T21:14:27",
"upload_time_iso_8601": "2024-01-03T21:14:27.049408Z",
"url": "https://files.pythonhosted.org/packages/25/bc/8cf641ba3385e687a381f0a6030a6630d78103f94ed54b527f31edaf9e81/xcon-0.8.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-03 21:14:27",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "xyngular",
"github_project": "py-xcon",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "xcon"
}
JSON
