========
idem-aws
========
.. image:: https://img.shields.io/badge/made%20with-pop-teal
:alt: Made with pop, a Python implementation of Plugin Oriented Programming
:target: https://pop.readthedocs.io/
.. image:: https://img.shields.io/badge/made%20with-idem-teal
:alt: Made with idem, a Python implementation of Plugin Oriented Programming
:target: https://www.idemproject.io/
.. image:: https://img.shields.io/badge/docs%20on-docs.idemproject.io-blue
:alt: Documentation is published with Sphinx on docs.idemproject.io
:target: https://docs.idemproject.io/idem-aws/en/latest/index.html
.. image:: https://img.shields.io/badge/made%20with-python-yellow
:alt: Made with Python
:target: https://www.python.org/
AWS Cloud Provider for Idem.
About
=====
``idem-aws`` helps manage AWS with ``idem``.
* `idem-aws source code <https://gitlab.com/vmware/idem/idem-aws>`__
* `idem-aws documentation <https://docs.idemproject.io/idem-aws/en/latest/index.html>`__
What is POP?
------------
This project is built with `pop <https://pop.readthedocs.io/>`__, a Python-based
implementation of *Plugin Oriented Programming (POP)*. POP seeks to bring
together concepts and wisdom from the history of computing in new ways to solve
modern computing problems.
For more information:
* `Intro to Plugin Oriented Programming (POP) <https://pop-book.readthedocs.io/en/latest/>`__
* `pop-awesome <https://gitlab.com/saltstack/pop/pop-awesome>`__
* `pop-create <https://gitlab.com/saltstack/pop/pop-create/>`__
What is Idem?
-------------
This project is built with `idem <https://www.idemproject.io/>`__, an idempotent,
imperatively executed, declarative programming language written in Python. This project extends
idem!
For more information:
* `Idem Project Website <https://www.idemproject.io/>`__
* `Idem Project docs portal <https://docs.idemproject.io/>`__
Getting Started
===============
Prerequisites
-------------
* Python 3.8+
* git *(if installing from source, or contributing to the project)*
* Idem
.. note::
It is recommended that you install Idem using Poetry. Poetry is a tool for virtual environment and dependency management. See the `Idem Getting Started guide <https://docs.idemproject.io/getting-started/en/latest/topics/gettingstarted/installing.html>`_ for more information.
Installation
------------
You can install ``idem-aws`` from PyPI, a source repository, or a local directory.
Before you install ``idem-aws``, ensure that you are in the same directory as your ``pyproject.toml`` file. Optionally, you can specify the directory containing your ``pyproject.toml`` file by using the ``--directory=DIRECTORY (-C)`` option.
Install from PyPI
+++++++++++++++++
To install ``idem-aws`` from PyPI, run the following command:
.. code-block:: bash
poetry add idem-aws
Install from source
+++++++++++++++++++
You can also install ``idem-aws`` directly from the source repository:
.. code-block:: bash
poetry add git+https://gitlab.com/vmware/idem/idem-aws.git
If you don't specify a branch, Poetry uses the latest commit on the ``master`` branch.
Install from a local directory
++++++++++++++++++++++++++++++
Clone the ``idem-aws`` repository. Then run the following command to install from the cloned directory:
.. code-block:: bash
poetry add ~/path/to/idem-aws
Setup
=====
After installation the AWS Idem Provider execution and state modules will be accessible to the pop ``hub``.
In order to use them we need to set up our credentials.
Create a new file called ``credentials.yaml`` and populate it with credentials.
If you are using localstack, then the ``id`` and ``key`` can be bogus values.
The ``default`` profile will be picked up automatically by ``idem``.
There are many ways aws providers/profiles can be stored. See `acct backends <https://gitlab.com/Akm0d/acct-backends>`_
for more information.
There are multiple authentication backends for ``idem-aws`` which each have their own unique set of parameters.
The following examples show some of the parameters that can be used in these backends to define profiles.
All backends end up creating a boto3 session under the hood and storing it in the ``ctx`` variable that gets passed
to all idem ``exec`` and ``state`` functions.
All authentication backends support two optional parameters, ``endpoint_url`` and ``provider_tag_key``. The ``endpoint url``
is used to specify an alternate destination for boto3 calls, such as a localstack server or custom dynamodb server.
The ``provider_tag_key`` is used when creating new resources. ``idem-aws`` will only interact with resources that are tagged
with the the customizable ``provider_tag_key`` key.
credentials.yaml:
.. code:: sls
aws:
default:
endpoint_url: http://localhost:4566
use_ssl: False
aws_access_key_id: localstack
aws_secret_access_key: _
region_name: us-west-1
You can also use `aws_session_token` with Idem for temporary security credentials
.. code:: sls
aws:
default:
endpoint_url: http://localhost:4566
use_ssl: False
aws_access_key_id: localstack
aws_secret_access_key: _
region_name: us-west-1
aws_session_token: my_token
Additionally, you can use AWS AssumeRole with Idem
.. code:: sls
aws:
default:
endpoint_url: http://localhost:4566
use_ssl: False
aws_access_key_id: localstack
aws_secret_access_key: _
region_name: us-west-1
assume_role:
role_arn: arn:aws:iam::999999999999999:role/xacct/developer
role_session_name: IdemSessionName
If ``region_name`` is unspecified in the acct profile, it can come from ``acct.extras`` in the idem config file:
.. code:: sls
# idem.cfg
acct:
extras:
aws:
region_name: us-west-1
You can also authenticate with ``aws-google-auth`` if it is installed.
.. code:: sls
aws.gsuite:
my-staging-env:
username: user@gmail.com
password: this_is_available_but_avoid_it
role_arn: arn:aws:iam::999999999999999:role/xacct/developer
idp_id: 9999999
sp_id: 999999999999
region: us-east-1
duration: 36000
account: developer
The google profile example is not named ``default``. To use it, it will need to be specified explicitly in an idem state.
.. code:: sls
ensure_resource_exists:
aws.ec2.vpc.present:
- acct_profile: my-staging-env
- name: idem_aws_vpc
- cidr_block: 10.0.0.0/24
It can also be specified from the command line when executing states.
.. code:: bash
idem state --acct-profile my-staging-env my_state.sls
It can also be specified from the command line when calling an exec module directly.
.. code:: bash
idem exec --acct-profile my-staging-env boto3.client.ec2.describe_vpcs
The last step to get up and running is to encrypt the credentials file and add the encryption key and encrypted file
path to the ENVIRONMENT.
The ``acct`` command should be available as ``acct`` is a requisite of ``idem`` and ``idem-aws``.
Encrypt the the credential file.
.. code:: bash
acct encrypt credentials.yaml
output::
-A9ZkiCSOjWYG_lbGmmkVh4jKLFDyOFH4e4S1HNtNwI=
Add these to your environment:
.. code:: bash
export ACCT_KEY="-A9ZkiCSOjWYG_lbGmmkVh4jKLFDyOFH4e4S1HNtNwI="
export ACCT_FILE=$PWD/credentials.yaml.fernet
If no acct_file is supplied, then the default awscli credentials that are picked-up by botocore will be used.
Missing cli options will be filled in by botocore from awscli config.
The order of config priority is:
#. acct profile
#. idem config file
#. awscli config
You are ready to use idem-aws!
Execution Modules
=================
Once everything has been set up properly, execution modules can be called directly by ``idem``.
Execution modules mirror the namespacing of the boto3.client and boto3.resource modules and have the same parameters.
For example, this is how you could list Vpcs from the command line with idem:
.. code:: bash
idem exec boto3.client.ec2.describe_vpcs
You can specify parameters as well.
In the case of boto3 resources, args will be passed to the resource constructor and kwargs will be passed to the operation like so:
.. code:: bash
idem exec boto3.resource.ec2.Vpc.create_subnet vpc-71d00419 CidrBlock="10.0.0.0/24"
States
======
States are also accessed by their relative location in ``idem-aws/idem_aws/states``.
For example, ``idem-aws/idem_aws/states/aws/ec2/vpc.py`` contains a function ``absent()``.
In my state file I can create a state that uses the ``absent`` function like so.
my_state.sls:
.. code:: sls
idem_aws_test_vpc:
aws.ec2.vpc.absent:
- name: "idem_aws_test_vpc"
This state can be executed with:
.. code:: bash
idem state my_state.sls
``idem state`` also has some flags that can significantly boost the scalability and performance of the run.
Let's use this new state which verifies that 100 vpcs are absent:
.. code:: sls
{% for i in range(100) %}
idem_aws_test_vpc_{{i}}:
aws.ec2.vpc.absent:
- name: "idem_aws_test_vpc_{{i}}"
{% endfor -%}
State can be executed with ``--runtime parallel`` to make full use of idem's async execution calls:
.. code:: bash
idem state --runtime parallel my_state.sls
Remote storage for enforced state management
--------------------------------------------
Idem-aws supports remote storage for Idem's enforced state management feature. That is, Idem can
store esm data on AWS S3 bucket. DynamoDB will be used as a file lock to prevent multiple users/processes
access the same storage file concurrently. To use remote storage, the esm profile need to be added to
the credential profile like the following:
.. code:: sls
aws:
default:
use_ssl: True
aws_access_key_id: AAAAAAAAA5CDFSDER3UQ
aws_secret_access_key: eHjPASFWERSFwVXKlsdfS4afD
region_name: eu-west-2
esm:
bucket: "idem-state-storage-bucket"
dynamodb_table: "idem-state-storage-table"
key: "/idem-state/demo-storage.json"
This esm file means that Idem will use AWS S3 bucket "idem-state-storage-bucket" and DynamoDB table
"idem-state-storage-table" in region eu-west-2. The "key" is the file path to which the esm data
will be read and stored. Both S3 bucket and DynamoDB table need to be created before using the feature.
The DynamoDB table should have the primary key as string "LockID" and nothing else.
Raw data
{
"_id": null,
"home_page": "https://docs.idemproject.io/idem-aws/en/latest/index.html",
"name": "idem-aws",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "",
"author": "VMware, Inc.",
"author_email": "idemproject@vmware.com",
"download_url": "https://files.pythonhosted.org/packages/9e/ec/0e1a3b1641f5c1a0c7efc87beb782861798ccd2ef670abac39cb3fc9232b/idem-aws-6.2.1.tar.gz",
"platform": null,
"description": "========\nidem-aws\n========\n\n.. image:: https://img.shields.io/badge/made%20with-pop-teal\n :alt: Made with pop, a Python implementation of Plugin Oriented Programming\n :target: https://pop.readthedocs.io/\n\n.. image:: https://img.shields.io/badge/made%20with-idem-teal\n :alt: Made with idem, a Python implementation of Plugin Oriented Programming\n :target: https://www.idemproject.io/\n\n.. image:: https://img.shields.io/badge/docs%20on-docs.idemproject.io-blue\n :alt: Documentation is published with Sphinx on docs.idemproject.io\n :target: https://docs.idemproject.io/idem-aws/en/latest/index.html\n\n.. image:: https://img.shields.io/badge/made%20with-python-yellow\n :alt: Made with Python\n :target: https://www.python.org/\n\nAWS Cloud Provider for Idem.\n\nAbout\n=====\n\n``idem-aws`` helps manage AWS with ``idem``.\n\n* `idem-aws source code <https://gitlab.com/vmware/idem/idem-aws>`__\n* `idem-aws documentation <https://docs.idemproject.io/idem-aws/en/latest/index.html>`__\n\nWhat is POP?\n------------\n\nThis project is built with `pop <https://pop.readthedocs.io/>`__, a Python-based\nimplementation of *Plugin Oriented Programming (POP)*. POP seeks to bring\ntogether concepts and wisdom from the history of computing in new ways to solve\nmodern computing problems.\n\nFor more information:\n\n* `Intro to Plugin Oriented Programming (POP) <https://pop-book.readthedocs.io/en/latest/>`__\n* `pop-awesome <https://gitlab.com/saltstack/pop/pop-awesome>`__\n* `pop-create <https://gitlab.com/saltstack/pop/pop-create/>`__\n\nWhat is Idem?\n-------------\n\nThis project is built with `idem <https://www.idemproject.io/>`__, an idempotent,\nimperatively executed, declarative programming language written in Python. This project extends\nidem!\n\nFor more information:\n\n* `Idem Project Website <https://www.idemproject.io/>`__\n* `Idem Project docs portal <https://docs.idemproject.io/>`__\n\nGetting Started\n===============\n\nPrerequisites\n-------------\n\n* Python 3.8+\n* git *(if installing from source, or contributing to the project)*\n* Idem\n\n.. note::\n It is recommended that you install Idem using Poetry. Poetry is a tool for virtual environment and dependency management. See the `Idem Getting Started guide <https://docs.idemproject.io/getting-started/en/latest/topics/gettingstarted/installing.html>`_ for more information.\n\nInstallation\n------------\n\nYou can install ``idem-aws`` from PyPI, a source repository, or a local directory.\n\nBefore you install ``idem-aws``, ensure that you are in the same directory as your ``pyproject.toml`` file. Optionally, you can specify the directory containing your ``pyproject.toml`` file by using the ``--directory=DIRECTORY (-C)`` option.\n\nInstall from PyPI\n+++++++++++++++++\n\nTo install ``idem-aws`` from PyPI, run the following command:\n\n.. code-block:: bash\n\n poetry add idem-aws\n\nInstall from source\n+++++++++++++++++++\n\nYou can also install ``idem-aws`` directly from the source repository:\n\n.. code-block:: bash\n\n poetry add git+https://gitlab.com/vmware/idem/idem-aws.git\n\nIf you don't specify a branch, Poetry uses the latest commit on the ``master`` branch.\n\nInstall from a local directory\n++++++++++++++++++++++++++++++\n\nClone the ``idem-aws`` repository. Then run the following command to install from the cloned directory:\n\n.. code-block:: bash\n\n poetry add ~/path/to/idem-aws\n\nSetup\n=====\n\nAfter installation the AWS Idem Provider execution and state modules will be accessible to the pop ``hub``.\nIn order to use them we need to set up our credentials.\n\nCreate a new file called ``credentials.yaml`` and populate it with credentials.\nIf you are using localstack, then the ``id`` and ``key`` can be bogus values.\nThe ``default`` profile will be picked up automatically by ``idem``.\n\nThere are many ways aws providers/profiles can be stored. See `acct backends <https://gitlab.com/Akm0d/acct-backends>`_\nfor more information.\n\nThere are multiple authentication backends for ``idem-aws`` which each have their own unique set of parameters.\nThe following examples show some of the parameters that can be used in these backends to define profiles.\nAll backends end up creating a boto3 session under the hood and storing it in the ``ctx`` variable that gets passed\nto all idem ``exec`` and ``state`` functions.\n\nAll authentication backends support two optional parameters, ``endpoint_url`` and ``provider_tag_key``. The ``endpoint url``\nis used to specify an alternate destination for boto3 calls, such as a localstack server or custom dynamodb server.\nThe ``provider_tag_key`` is used when creating new resources. ``idem-aws`` will only interact with resources that are tagged\nwith the the customizable ``provider_tag_key`` key.\n\ncredentials.yaml:\n\n.. code:: sls\n\n aws:\n default:\n endpoint_url: http://localhost:4566\n use_ssl: False\n aws_access_key_id: localstack\n aws_secret_access_key: _\n region_name: us-west-1\n\n\nYou can also use `aws_session_token` with Idem for temporary security credentials\n\n.. code:: sls\n\n aws:\n default:\n endpoint_url: http://localhost:4566\n use_ssl: False\n aws_access_key_id: localstack\n aws_secret_access_key: _\n region_name: us-west-1\n aws_session_token: my_token\n\n\nAdditionally, you can use AWS AssumeRole with Idem\n\n.. code:: sls\n\n aws:\n default:\n endpoint_url: http://localhost:4566\n use_ssl: False\n aws_access_key_id: localstack\n aws_secret_access_key: _\n region_name: us-west-1\n assume_role:\n role_arn: arn:aws:iam::999999999999999:role/xacct/developer\n role_session_name: IdemSessionName\n\nIf ``region_name`` is unspecified in the acct profile, it can come from ``acct.extras`` in the idem config file:\n\n.. code:: sls\n\n # idem.cfg\n acct:\n extras:\n aws:\n region_name: us-west-1\n\nYou can also authenticate with ``aws-google-auth`` if it is installed.\n\n.. code:: sls\n\n aws.gsuite:\n my-staging-env:\n username: user@gmail.com\n password: this_is_available_but_avoid_it\n role_arn: arn:aws:iam::999999999999999:role/xacct/developer\n idp_id: 9999999\n sp_id: 999999999999\n region: us-east-1\n duration: 36000\n account: developer\n\nThe google profile example is not named ``default``. To use it, it will need to be specified explicitly in an idem state.\n\n.. code:: sls\n\n ensure_resource_exists:\n aws.ec2.vpc.present:\n - acct_profile: my-staging-env\n - name: idem_aws_vpc\n - cidr_block: 10.0.0.0/24\n\nIt can also be specified from the command line when executing states.\n\n.. code:: bash\n\n idem state --acct-profile my-staging-env my_state.sls\n\nIt can also be specified from the command line when calling an exec module directly.\n\n.. code:: bash\n\n idem exec --acct-profile my-staging-env boto3.client.ec2.describe_vpcs\n\nThe last step to get up and running is to encrypt the credentials file and add the encryption key and encrypted file\npath to the ENVIRONMENT.\n\nThe ``acct`` command should be available as ``acct`` is a requisite of ``idem`` and ``idem-aws``.\nEncrypt the the credential file.\n\n.. code:: bash\n\n acct encrypt credentials.yaml\n\noutput::\n\n -A9ZkiCSOjWYG_lbGmmkVh4jKLFDyOFH4e4S1HNtNwI=\n\nAdd these to your environment:\n\n.. code:: bash\n\n export ACCT_KEY=\"-A9ZkiCSOjWYG_lbGmmkVh4jKLFDyOFH4e4S1HNtNwI=\"\n export ACCT_FILE=$PWD/credentials.yaml.fernet\n\n\nIf no acct_file is supplied, then the default awscli credentials that are picked-up by botocore will be used.\nMissing cli options will be filled in by botocore from awscli config.\nThe order of config priority is:\n\n#. acct profile\n#. idem config file\n#. awscli config\n\nYou are ready to use idem-aws!\n\nExecution Modules\n=================\n\nOnce everything has been set up properly, execution modules can be called directly by ``idem``.\nExecution modules mirror the namespacing of the boto3.client and boto3.resource modules and have the same parameters.\n\nFor example, this is how you could list Vpcs from the command line with idem:\n\n.. code:: bash\n\n idem exec boto3.client.ec2.describe_vpcs\n\nYou can specify parameters as well.\nIn the case of boto3 resources, args will be passed to the resource constructor and kwargs will be passed to the operation like so:\n\n.. code:: bash\n\n idem exec boto3.resource.ec2.Vpc.create_subnet vpc-71d00419 CidrBlock=\"10.0.0.0/24\"\n\nStates\n======\n\nStates are also accessed by their relative location in ``idem-aws/idem_aws/states``.\nFor example, ``idem-aws/idem_aws/states/aws/ec2/vpc.py`` contains a function ``absent()``.\nIn my state file I can create a state that uses the ``absent`` function like so.\n\nmy_state.sls:\n\n.. code:: sls\n\n idem_aws_test_vpc:\n aws.ec2.vpc.absent:\n - name: \"idem_aws_test_vpc\"\n\nThis state can be executed with:\n\n.. code:: bash\n\n idem state my_state.sls\n\n``idem state`` also has some flags that can significantly boost the scalability and performance of the run.\nLet's use this new state which verifies that 100 vpcs are absent:\n\n.. code:: sls\n\n {% for i in range(100) %}\n idem_aws_test_vpc_{{i}}:\n aws.ec2.vpc.absent:\n - name: \"idem_aws_test_vpc_{{i}}\"\n {% endfor -%}\n\nState can be executed with ``--runtime parallel`` to make full use of idem's async execution calls:\n\n.. code:: bash\n\n idem state --runtime parallel my_state.sls\n\nRemote storage for enforced state management\n--------------------------------------------\n\nIdem-aws supports remote storage for Idem's enforced state management feature. That is, Idem can\nstore esm data on AWS S3 bucket. DynamoDB will be used as a file lock to prevent multiple users/processes\naccess the same storage file concurrently. To use remote storage, the esm profile need to be added to\nthe credential profile like the following:\n\n.. code:: sls\n\n aws:\n default:\n use_ssl: True\n aws_access_key_id: AAAAAAAAA5CDFSDER3UQ\n aws_secret_access_key: eHjPASFWERSFwVXKlsdfS4afD\n region_name: eu-west-2\n esm:\n bucket: \"idem-state-storage-bucket\"\n dynamodb_table: \"idem-state-storage-table\"\n key: \"/idem-state/demo-storage.json\"\n\nThis esm file means that Idem will use AWS S3 bucket \"idem-state-storage-bucket\" and DynamoDB table\n\"idem-state-storage-table\" in region eu-west-2. The \"key\" is the file path to which the esm data\nwill be read and stored. Both S3 bucket and DynamoDB table need to be created before using the feature.\nThe DynamoDB table should have the primary key as string \"LockID\" and nothing else.\n",
"bugtrack_url": null,
"license": "Apache Software License 2.0",
"summary": "AWS Cloud Provider for Idem",
"version": "6.2.1",
"project_urls": {
"Code": "https://gitlab.com/vmware/idem/idem-aws",
"Homepage": "https://docs.idemproject.io/idem-aws/en/latest/index.html",
"Issue tracker": "https://gitlab.com/vmware/idem/idem-aws/-/issues"
},
"split_keywords": [],
"urls": [
{
"comment_text": "pypi",
"digests": {
"blake2b_256": "4302b69ce838e2118970d6089f7f22c6a88e761ea787d442343d9b8b4609547d",
"md5": "fab38ede21734b7f1a323830b0384896",
"sha256": "4965ed3f7195b937fd8782c52c83075b2bec7d090c51550a536065bf02c0eed5"
},
"downloads": -1,
"filename": "idem_aws-6.2.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "fab38ede21734b7f1a323830b0384896",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 1245640,
"upload_time": "2024-01-22T16:47:06",
"upload_time_iso_8601": "2024-01-22T16:47:06.943045Z",
"url": "https://files.pythonhosted.org/packages/43/02/b69ce838e2118970d6089f7f22c6a88e761ea787d442343d9b8b4609547d/idem_aws-6.2.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "pypi",
"digests": {
"blake2b_256": "9eec0e1a3b1641f5c1a0c7efc87beb782861798ccd2ef670abac39cb3fc9232b",
"md5": "153c47fa649a5bdd319e0f8575c71ee0",
"sha256": "93dd08ecd0d48f5c16a8dfc5939e980a3f2aec1f9b747a548f7171a604493ef4"
},
"downloads": -1,
"filename": "idem-aws-6.2.1.tar.gz",
"has_sig": false,
"md5_digest": "153c47fa649a5bdd319e0f8575c71ee0",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 818642,
"upload_time": "2024-01-22T16:47:12",
"upload_time_iso_8601": "2024-01-22T16:47:12.185756Z",
"url": "https://files.pythonhosted.org/packages/9e/ec/0e1a3b1641f5c1a0c7efc87beb782861798ccd2ef670abac39cb3fc9232b/idem-aws-6.2.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-22 16:47:12",
"github": false,
"gitlab": true,
"bitbucket": false,
"codeberg": false,
"gitlab_user": "vmware",
"gitlab_project": "idem",
"lcname": "idem-aws"
}
JSON