Dynaconf AWS Systems Manager Parameter Store Loader
====================================================
When configured, this loader will permit Dynaconf to query `AWS Systems Manager Parameter Store <https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html>`_ for slash-delimited hierarchical configuration data.
Loader Configuration
--------------------
An example:
.. code-block:: python
from dynaconf import Dynaconf
settings = Dynaconf(
environments=True,
settings_file="settings.toml",
LOADERS_FOR_DYNACONF=[
"dynaconf_aws_loader.loader",
"dynaconf.loaders.env_loader"
],
)
Note that for the basic functioning of this loader, the `environments <https://www.dynaconf.com/configuration/#environments>`_ option for ``Dynaconf`` must be set, and an environment must be used.
Configuration Variables
-----------------------
Both of the following configuration values should be set in the *environment* to avoid a chicken/egg scenario for initializing this custom loader:
- ``SSM_PARAMETER_PROJECT_PREFIX_FOR_DYNACONF``: Required.
The ``project`` prefix in the parameter store path. For example, if the parameter hierarchy looks something like ``/baldur/development/database_uri``, then in this case ``SSM_PARAMETER_PROJECT_PREFIX_FOR_DYNACONF=baldur``.
- ``SSM_PARAMETER_NAMESPACE_FOR_DYNACONF``: Optional.
This provides an additional level of grouping once the project and environment have been determined. For example, if the parameter hierarchy looks something like ``/baldur/pr-123/development/database_uri``, then ``SSM_PARAMETER_NAMESPACE_FOR_DYNACONF=pr-123``.
.. note::
If a namespace is utilized, be aware that namespaced settings will be *merged* with non-namespaced settings. This merge is a naive one, where namespaced settings will completely overwrite non-namespaced settings with the same key.
The following optional variables should be set in your ``settings.toml`` (or equivalent format), if desired:
- ``SSM_ENDPOINT_URL_FOR_DYNACONF``: If your AWS SSM uses a different endpoint than the AWS default. This can be useful for local development when you are running something like `LocalStack <https://localstack.cloud/>`_.
- ``SSM_SESSION_FOR_DYNACONF``: If you require custom `boto3.session.Session <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html>`_ arguments, you can specify then as a dictionary here. Note that this will override the default ``boto3`` credential configuration.
- ``SSM_LOAD_DEFAULT_ENV_FOR_DYNACONF``: Boolean, defaults to ``True``. If you want the SSM loader to load keys under the ``default`` environment name. The key name itself can be set via the Dynaconf setting of ``DEFAULT_ENV_FOR_DYNACONF`` if you want it to be something other than ``default``.
Parameter Store Details
~~~~~~~~~~~~~~~~~~~~~~~
The structure that this loader expects from the path-based organization in SSM is:
.. code-block::
/<project-name>/<environment>/<parameter-name>
An optional ``namespace`` can be specified as a sub-project grouping for parameters:
.. code-block::
/<project-name>/<environment>/<namespace>/<parameter-name>
Note that if you choose to use a ``namespace`` identifier, it must not conflict with existing ``environment`` identifiers.
If ``SSM_LOAD_DEFAULT_ENV_FOR_DYNACONF`` is set to ``True`` (which is the default value), the loader will add whatever the value of ``DEFAULT_ENV_FOR_DYNACONF`` as an ``environment`` key to load from SSM. The typical use case here is to have a default value for all environments that can be overriden on a per-environment basis as necessary.
Security Considerations
~~~~~~~~~~~~~~~~~~~~~~~
For this loader to function correctly and securely, the use of IAM policies to restrict which parameters can be read/mutated is highly encouraged.
Policies can be enacted on a glob-path basis, which will ensure that the only parameters that can be fetched/hydrated into the local object instance are the ones that the current environment is permitted to load.
The following policy for a fictional account allows a user to call the ``DescribeParameters`` and ``GetParameters`` API operations for parameters that begin with the path ``/testapp/production``:
.. code-block:: json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ssm:DescribeParameters"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ssm:GetParameters"
],
"Resource": "arn:aws:ssm:us-east-1:000000000000:parameter/testapp/production*"
}
]
}
.. warning::
If a user has access to a path, then the user can access all levels of that path. For example, if a user has permission to access path ``/testapp``, then the user can also access ``testapp/production``. Even if a user has explicitly been denied access in IAM for parameter ``/testapp/production``, they can still call the ``GetParametersByPath`` API operation recursively for ``/testapp`` and view ``/testapp/production``.
Setting Parameters via Boto3
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Parameters may be set via the AWS Web Console UI, or one of their many client libraries. The `boto3 <https://boto3.amazonaws.com/v1/documentation/api/latest/index.html>`_ library is perhaps the most well-known, and the process is relatively straightforward:
.. code-block:: python
import boto3
ssm_client = boto3.client("ssm")
ssm_client.put_parameter(
Name="/testapp/development/database/host",
Value="localhost",
Type="String",
)
ssm_client.put_parameter(
Name="/testapp/production/database/password",
Value="sekrit",
Type="SecureString",
)
ssm_client.put_parameter(
Name="/testapp/production/database/host",
Value="db.example.com",
Type="String",
)
ssm_client.put_parameter(
Name="/testapp/production/admin_email",
Value="help@example.com",
Type="String",
)
This creates a parameter hierarchy with the following structure:
.. code-block:: json
{
"testapp": {
"development": {"database": {"host": "localhost"}},
"production": {
"database": {"host": "db.example.com", "password": "sekrit"},
"admin_email": "help@example.com",
},
},
}
Parameter Name Limitations
--------------------------
AWS SSM has the following key (and thus path) limitations:
- Parameter names are case sensitive
- A parameter name must be unique within an Amazon Web Services Region
- A parameter name can't be prefixed with "aws" or "ssm" (case-insensitive)
- Parameter names can include only the following symbols and letters: a-zA-Z0-9\_.-
- The slash character ( ``/`` ) is used to delineate hierarchies in parameter names
- A parameter name can't include spaces
- Parameter hierarchies are limited to a maximum depth of fifteen levels
Testing
~~~~~~~
0. Have Docker installed and running
1. Clone this repository
2. Ensure you have `poetry` available on your system
3. `poetry run pytest`
The test suite will spin up an ephemeral Docker container; it may take a few seconds for it to load. The relevant test fixtures will handle setting parameters and their values in the Localstack SSM service.
TODO
~~~~
- [ ] CI configuration for matrix-based python/dynaconf version testing
- [ ] Handle `Parameter Store references to AWS Secrets Manager <https://docs.aws.amazon.com/systems-manager/latest/userguide/integration-ps-secretsmanager.html>`_
- [ ] Make ``tests/docker-compose.yml`` more configurable, e.g. ports, in case a different Localstack container is already running for the user
Raw data
{
"_id": null,
"home_page": "https://github.com/fictivekin/dynaconf-aws-loader",
"name": "dynaconf-aws-loader",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8,<4.0",
"maintainer_email": "",
"keywords": "dynaconf,AWS,SSM",
"author": "Jo\u00ebl Perras",
"author_email": "joel@fictivekin.com",
"download_url": "https://files.pythonhosted.org/packages/9c/3f/4c309449230a40b851db368241c21f8602c2773ec6414e06211d6be196a5/dynaconf_aws_loader-0.4.0.tar.gz",
"platform": null,
"description": "Dynaconf AWS Systems Manager Parameter Store Loader\n====================================================\n\nWhen configured, this loader will permit Dynaconf to query `AWS Systems Manager Parameter Store <https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html>`_ for slash-delimited hierarchical configuration data.\n\nLoader Configuration\n--------------------\n\nAn example:\n\n.. code-block:: python\n\n from dynaconf import Dynaconf\n\n settings = Dynaconf(\n environments=True,\n settings_file=\"settings.toml\",\n LOADERS_FOR_DYNACONF=[\n \"dynaconf_aws_loader.loader\",\n \"dynaconf.loaders.env_loader\"\n ],\n )\n\n\nNote that for the basic functioning of this loader, the `environments <https://www.dynaconf.com/configuration/#environments>`_ option for ``Dynaconf`` must be set, and an environment must be used.\n\nConfiguration Variables\n-----------------------\n\nBoth of the following configuration values should be set in the *environment* to avoid a chicken/egg scenario for initializing this custom loader:\n\n- ``SSM_PARAMETER_PROJECT_PREFIX_FOR_DYNACONF``: Required.\n The ``project`` prefix in the parameter store path. For example, if the parameter hierarchy looks something like ``/baldur/development/database_uri``, then in this case ``SSM_PARAMETER_PROJECT_PREFIX_FOR_DYNACONF=baldur``.\n\n- ``SSM_PARAMETER_NAMESPACE_FOR_DYNACONF``: Optional.\n This provides an additional level of grouping once the project and environment have been determined. For example, if the parameter hierarchy looks something like ``/baldur/pr-123/development/database_uri``, then ``SSM_PARAMETER_NAMESPACE_FOR_DYNACONF=pr-123``.\n\n.. note::\n If a namespace is utilized, be aware that namespaced settings will be *merged* with non-namespaced settings. This merge is a naive one, where namespaced settings will completely overwrite non-namespaced settings with the same key.\n\nThe following optional variables should be set in your ``settings.toml`` (or equivalent format), if desired:\n\n- ``SSM_ENDPOINT_URL_FOR_DYNACONF``: If your AWS SSM uses a different endpoint than the AWS default. This can be useful for local development when you are running something like `LocalStack <https://localstack.cloud/>`_.\n- ``SSM_SESSION_FOR_DYNACONF``: If you require custom `boto3.session.Session <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html>`_ arguments, you can specify then as a dictionary here. Note that this will override the default ``boto3`` credential configuration.\n- ``SSM_LOAD_DEFAULT_ENV_FOR_DYNACONF``: Boolean, defaults to ``True``. If you want the SSM loader to load keys under the ``default`` environment name. The key name itself can be set via the Dynaconf setting of ``DEFAULT_ENV_FOR_DYNACONF`` if you want it to be something other than ``default``.\n\n\nParameter Store Details\n~~~~~~~~~~~~~~~~~~~~~~~\n\nThe structure that this loader expects from the path-based organization in SSM is:\n\n.. code-block::\n\n /<project-name>/<environment>/<parameter-name>\n\n\nAn optional ``namespace`` can be specified as a sub-project grouping for parameters:\n\n.. code-block::\n\n /<project-name>/<environment>/<namespace>/<parameter-name>\n\n\nNote that if you choose to use a ``namespace`` identifier, it must not conflict with existing ``environment`` identifiers.\n\nIf ``SSM_LOAD_DEFAULT_ENV_FOR_DYNACONF`` is set to ``True`` (which is the default value), the loader will add whatever the value of ``DEFAULT_ENV_FOR_DYNACONF`` as an ``environment`` key to load from SSM. The typical use case here is to have a default value for all environments that can be overriden on a per-environment basis as necessary.\n\n\nSecurity Considerations\n~~~~~~~~~~~~~~~~~~~~~~~\n\nFor this loader to function correctly and securely, the use of IAM policies to restrict which parameters can be read/mutated is highly encouraged.\n\nPolicies can be enacted on a glob-path basis, which will ensure that the only parameters that can be fetched/hydrated into the local object instance are the ones that the current environment is permitted to load.\n\nThe following policy for a fictional account allows a user to call the ``DescribeParameters`` and ``GetParameters`` API operations for parameters that begin with the path ``/testapp/production``:\n\n.. code-block:: json\n\n {\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"ssm:DescribeParameters\"\n ],\n \"Resource\": \"*\"\n },\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"ssm:GetParameters\"\n ],\n \"Resource\": \"arn:aws:ssm:us-east-1:000000000000:parameter/testapp/production*\"\n }\n ]\n }\n\n\n.. warning::\n\n If a user has access to a path, then the user can access all levels of that path. For example, if a user has permission to access path ``/testapp``, then the user can also access ``testapp/production``. Even if a user has explicitly been denied access in IAM for parameter ``/testapp/production``, they can still call the ``GetParametersByPath`` API operation recursively for ``/testapp`` and view ``/testapp/production``.\n\n\nSetting Parameters via Boto3\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nParameters may be set via the AWS Web Console UI, or one of their many client libraries. The `boto3 <https://boto3.amazonaws.com/v1/documentation/api/latest/index.html>`_ library is perhaps the most well-known, and the process is relatively straightforward:\n\n.. code-block:: python\n\n import boto3\n ssm_client = boto3.client(\"ssm\")\n\n ssm_client.put_parameter(\n Name=\"/testapp/development/database/host\",\n Value=\"localhost\",\n Type=\"String\",\n )\n\n ssm_client.put_parameter(\n Name=\"/testapp/production/database/password\",\n Value=\"sekrit\",\n Type=\"SecureString\",\n )\n\n ssm_client.put_parameter(\n Name=\"/testapp/production/database/host\",\n Value=\"db.example.com\",\n Type=\"String\",\n )\n\n ssm_client.put_parameter(\n Name=\"/testapp/production/admin_email\",\n Value=\"help@example.com\",\n Type=\"String\",\n )\n\n\nThis creates a parameter hierarchy with the following structure:\n\n.. code-block:: json\n\n {\n \"testapp\": {\n \"development\": {\"database\": {\"host\": \"localhost\"}},\n \"production\": {\n \"database\": {\"host\": \"db.example.com\", \"password\": \"sekrit\"},\n \"admin_email\": \"help@example.com\",\n },\n },\n }\n\n\nParameter Name Limitations\n--------------------------\n\nAWS SSM has the following key (and thus path) limitations:\n\n- Parameter names are case sensitive\n- A parameter name must be unique within an Amazon Web Services Region\n- A parameter name can't be prefixed with \"aws\" or \"ssm\" (case-insensitive)\n- Parameter names can include only the following symbols and letters: a-zA-Z0-9\\_.-\n- The slash character ( ``/`` ) is used to delineate hierarchies in parameter names\n- A parameter name can't include spaces\n- Parameter hierarchies are limited to a maximum depth of fifteen levels\n\n\nTesting\n~~~~~~~\n\n0. Have Docker installed and running\n1. Clone this repository\n2. Ensure you have `poetry` available on your system\n3. `poetry run pytest`\n\nThe test suite will spin up an ephemeral Docker container; it may take a few seconds for it to load. The relevant test fixtures will handle setting parameters and their values in the Localstack SSM service.\n\n\nTODO\n~~~~\n\n- [ ] CI configuration for matrix-based python/dynaconf version testing\n- [ ] Handle `Parameter Store references to AWS Secrets Manager <https://docs.aws.amazon.com/systems-manager/latest/userguide/integration-ps-secretsmanager.html>`_\n- [ ] Make ``tests/docker-compose.yml`` more configurable, e.g. ports, in case a different Localstack container is already running for the user\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A custom loader for Dynaconf that uses AWS Systems Manager Parameter Store as a source of truth.",
"version": "0.4.0",
"project_urls": {
"Homepage": "https://github.com/fictivekin/dynaconf-aws-loader",
"Repository": "https://github.com/fictivekin/dynaconf-aws-loader"
},
"split_keywords": [
"dynaconf",
"aws",
"ssm"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "b009990ef0d63ef70689905d393ffd533f99cf919296866adebccdac95d584c1",
"md5": "4a1721b70a3d56b4e97fabca25edcd6c",
"sha256": "62230679738ef8bc5ab84404c665f1ca09f4759c444e7400d4640e0ddec8a46a"
},
"downloads": -1,
"filename": "dynaconf_aws_loader-0.4.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "4a1721b70a3d56b4e97fabca25edcd6c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8,<4.0",
"size": 10799,
"upload_time": "2023-06-22T21:50:52",
"upload_time_iso_8601": "2023-06-22T21:50:52.734968Z",
"url": "https://files.pythonhosted.org/packages/b0/09/990ef0d63ef70689905d393ffd533f99cf919296866adebccdac95d584c1/dynaconf_aws_loader-0.4.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9c3f4c309449230a40b851db368241c21f8602c2773ec6414e06211d6be196a5",
"md5": "a263ce352a2e96cf84d3fc5afb152cd4",
"sha256": "8f2581ef07d9d9923e838e8daf53c204f272379d238c147f70cd47f4fee87a4a"
},
"downloads": -1,
"filename": "dynaconf_aws_loader-0.4.0.tar.gz",
"has_sig": false,
"md5_digest": "a263ce352a2e96cf84d3fc5afb152cd4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8,<4.0",
"size": 8336,
"upload_time": "2023-06-22T21:50:54",
"upload_time_iso_8601": "2023-06-22T21:50:54.284657Z",
"url": "https://files.pythonhosted.org/packages/9c/3f/4c309449230a40b851db368241c21f8602c2773ec6414e06211d6be196a5/dynaconf_aws_loader-0.4.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-06-22 21:50:54",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "fictivekin",
"github_project": "dynaconf-aws-loader",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "dynaconf-aws-loader"
}
JSON
