# Botocove
Run a function against a selection of AWS accounts, Organizational Units (OUs)
or all AWS accounts in an organization, concurrently with thread safety.
Run in one or multiple regions.
Opinionated by default to work with the standard AWS Organization master/member
configuration from an organization master account but customisable for any
context.
- Fast
- Easy
- Dolphin Themed 🐬
Botocove is a simple decorator for functions to remove time and complexity
burden. Uses a `ThreadPoolExecutor` to run boto3 sessions against AWS accounts
concurrently.
Decorating a function in `@cove` provides a boto3 session to the decorated
Python function and runs it in every account requested, gathering all results
into a dictionary.
**Warning**: this tool gives you the potential to make dangerous changes
at scale. **Test carefully and make idempotent changes**! Please read available
arguments to understand safe experimentation with this package.
## Pre-requisites and Info
An AWS session with `sts:assumerole` and `sts:get-caller-identity` access,
and accounts that contain a IAM role with trust relationship to the Botocove
calling account.
By default, the session is expected to be in an AWS Organization Master or
a delegated Organization admin account. You can alter nearly all behaviour of
Cove with appropriate [arguments](#arguments)
Cove will not execute a function call in the account it's called from.
In the Botocove calling account the minimum IAM requirements are:
- `sts:AssumeRole` on all of the roles in the target accounts.
- `organizations:ListAccounts` to run against an AWS organization and capture
account metadata.
- `organizations:ListChildren` to run against specific OUs.
In the organization member accounts:
- A target role that trusts the calling account - for example `AWSControlTowerExecution`
or
[`OrganizationAccountAccessRole` role](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_accounts_access.html)
See the [arguments](#arguments) section for how to change these defaults to work
with any account configuration, including running without an AWS Organization.
## Quickstart
A function written to interact with one
[boto3 session](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html)
can now be called with a `session` from every account and region required by
assuming a role in for each account - except the host account you're running
from.
For example:
A standard approach: this function takes a boto3 `session` and gets all IAM
users from a single AWS account. You would then manually run it in each account.
```python
import boto3
def get_iam_users(session):
iam = session.client("iam", region_name="eu-west-1")
all_users = iam.get_paginator("list_users").paginate().build_full_result()
return all_users
def main():
session = boto3.session.Session(profile_name="my_dev_account")
users = get_iam_users(session)
print(users) # A single account's IAM users
```
Now with `@cove`: a session for every account in the organization is injected
by the decorator. A safe example to run as a test!
```python
from pprint import pprint
import boto3
from botocove import cove
@cove()
def get_iam_users(session):
iam = session.client("iam", region_name="eu-west-1")
all_users = iam.get_paginator("list_users").paginate().build_full_result()
return all_users
def main():
# No session passed as the decorator injects it
all_results = get_iam_users()
# Now returns a Dict with keys Results, Exceptions and FailedAssumeRole
# A list of dictionaries for each account, with account details included.
# Each account's get_iam_users return is in a "Result" key.
pprint(all_results["Results"])
# A list of dictionaries for each account that raised an exception
pprint(all_results["Exceptions"])
# A list of dictionaries for each account that could not be assumed into
pprint(all_results["FailedAssumeRole"])
if __name__ == "__main__":
main()
```
Here's an example of a more customised Cove decorator:
```python
@cove(
target_ids=["123456789101", "234567891011"], # also accepts OU ids!
rolename="AWSControlTowerExecution",
raise_exception=True,
regions=["eu-west-1", "eu-west-2", "us-east-1"],
)
def do_things(session):
# Cove will return six results of True, 2 accounts * 3 regions
return True
```
## Arguments
### Cove
`@cove()`:
Uses the standard boto3 credential chain to start with, assuming roles in every
account required. Defaults to assuming the `OrganizationAccountAccessRole` in
every account in an AWS organization.
Equivalent to:
```python
@cove(
target_ids=None, ignore_ids=None, rolename=None, role_session_name=None,
policy=None, policy_arns=None, assuming_session=None, raise_exception=False,
thread_workers=20, regions=None, partition=None
)
```
`target_ids`: List[str]
A list of AWS account IDs and/or AWS Organization Units IDs to attempt to assume
role in to. When unset, attempts to use every available account ID in an AWS
organization. When specifying target OU's, all child OUs and accounts belonging
to that OU will be collected.
`ignore_ids`: List[str]
A list of AWS account ID's and OU's to prevent functions being run by Cove.
Ignored IDs takes precedence over `target_ids`. Providing an OU ID will collect
all child OUs and accounts to ignore.
The calling account that is running the Cove-wrapped function at runtime is
always ignored.
`rolename`: str
An IAM role name that will be attempted to assume in all target accounts.
Defaults to the AWS Organization default, `OrganizationAccountAccessRole`.
`role_session_name`: str
An IAM role session name that will be passed to each Cove session's
`sts.assume_role()` call. Defaults to the name of the role being used if unset.
`policy`: str
A policy document that will be used as a session policy. A non-None value is
passed through via the Policy parameter in each Cove session's
`sts.assume_role()` call.
`policy_arns`: List[[PolicyDescriptorTypeTypeDef](https://pypi.org/project/mypy-boto3-sts/)]
A list of managed policy ARNs that will be used as a session policy. A non-None
value is passed through via the PolicyArns parameter in each Cove session's
`sts.assume_role()` call.
`assuming_session`: Session
A Boto3 `boto3.session.Session()` object that will be used to call
`sts:assumerole`. If not provided, cove will instantiate one which will use the
standard boto3 credential chain.
`raise_exception`: bool
Defaults to False. Default behaviour is that exceptions are not raised from
decorated function. This is due to `cove` running asynchronously and preferring
to resolve all tasks and report their results instead of exiting early.
`raise_exception=True` will allow a full stack trace to escape on the first
exception seen; but will not gracefully or consistently interrupt running tasks.
It is vital to run interruptible, idempotent code with this argument as `True`.
`thread_workers`: int
Defaults to 20. Cove utilises a ThreadPoolWorker under the hood, which can be
tuned with this argument. Number of thread workers directly correlates to memory
usage: see [here](#is-botocove-thread-safe)
`regions`: List[str]
If not provided, Cove will respect your profile's default region via the boto
credential chain. If provided, Cove will run the decorated function in every
region named.
You can get all regions with:
```python
regions = [
r['RegionName'] for r in boto3.client('ec2').describe_regions()['Regions']
]
```
`partition`: str
If not provided, Cove will use the [AWS partition](https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html)
of your profile in constructing the ARN for the role to assume in all target
accounts.
`external_id`: str
Defaults to None. An external id that will be passed to each Cove session's
`sts.assume_role()` call.
### CoveSession
Cove supplies an enriched Boto3 session to each function called. Account details
are available with the `session_information` attribute.
Otherwise, it functions exactly as calling `boto3` would.
```python
@cove()
def do_nothing(session: CoveSession):
print(session.session_information) # Outputs a dict of known information
# This function runs no boto3-specific API, but does test that a role
# can be assumed
```
## Return values
Wrapped functions return a dictionary. Each value contains List[Dict[str, Any]]:
```python
{
"Results": results:
"Exceptions": exceptions,
"FailedAssumeRole": invalid_sessions,
}
```
An example of cove_output["Results"]:
```python
[ # A list of dictionaries per account called
{
'Id': '123456789010',
'Email': 'email@address.com',
'Name': 'account-name',
'Status': 'ACTIVE',
'AssumeRoleSuccess': True,
'Result': wrapped_function_return_value # Result of wrapped func
}
]
```
### Is botocove thread safe?
botocove is thread safe, but number of threaded executions will be bound by
memory, network IO and AWS api rate limiting. Defaulting to 20 thread workers is
a reasonable starting point, but can be further optimised for runtime with
experimentation.
botocove has no constraint or understanding of the function it's wrapping: it is
recommended to avoid shared state for botocove wrapped functions, and to write
simple functions that are written to be idempotent and independent.
[Boto3 Session objects are not natively thread safe](https://boto3.amazonaws.com/v1/documentation/api/1.14.31/guide/session.html#multithreading-or-multiprocessing-with-sessions)
and should not be shared across threads. However, botocove is instantiating a
new Session object per thread/account and running decorated functions inside
their own closure. A shared client is used from the host account that botocove
is run from (eg, an organization master account) -
[clients are threadsafe](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/clients.html#multithreading-or-multiprocessing-with-clients)
and allow this.
boto3 sessions have a significant memory footprint:
Version 1.5.0 of botocove was re-written to ensure that boto3 sessions are
released after completion which resolved memory starvation issues. This was
discussed here: <https://github.com/connelldave/botocove/issues/20> and a
relevant boto3 issue is here: <https://github.com/boto/boto3/issues/1670>
### botocove?
It turns out that the Amazon's Boto dolphins are solitary or small-group
animals, unlike the large pods of dolphins in the oceans. This killed my "large
group of boto" idea, so the next best idea was where might they all shelter
together... a cove!
Raw data
{
"_id": null,
"home_page": "https://github.com/connelldave/botocove",
"name": "botocove",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8,<4.0",
"maintainer_email": "",
"keywords": "AWS,organizations,AWS organizations,boto3",
"author": "Dave Connell",
"author_email": "daveconn41@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/40/8d/318b1de0aecb0f8f555aa4ec7f4ea9ecee3f7f7f4f6c36f41c9e7175085c/botocove-1.7.4.tar.gz",
"platform": null,
"description": "# Botocove\n\nRun a function against a selection of AWS accounts, Organizational Units (OUs)\nor all AWS accounts in an organization, concurrently with thread safety.\nRun in one or multiple regions.\n\nOpinionated by default to work with the standard AWS Organization master/member\nconfiguration from an organization master account but customisable for any\ncontext.\n\n- Fast\n- Easy\n- Dolphin Themed \ud83d\udc2c\n\nBotocove is a simple decorator for functions to remove time and complexity\nburden. Uses a `ThreadPoolExecutor` to run boto3 sessions against AWS accounts\nconcurrently.\n\nDecorating a function in `@cove` provides a boto3 session to the decorated\nPython function and runs it in every account requested, gathering all results\ninto a dictionary.\n\n**Warning**: this tool gives you the potential to make dangerous changes\nat scale. **Test carefully and make idempotent changes**! Please read available\narguments to understand safe experimentation with this package.\n\n## Pre-requisites and Info\n\nAn AWS session with `sts:assumerole` and `sts:get-caller-identity` access,\nand accounts that contain a IAM role with trust relationship to the Botocove\ncalling account.\n\nBy default, the session is expected to be in an AWS Organization Master or\na delegated Organization admin account. You can alter nearly all behaviour of\nCove with appropriate [arguments](#arguments)\n\nCove will not execute a function call in the account it's called from.\n\nIn the Botocove calling account the minimum IAM requirements are:\n\n- `sts:AssumeRole` on all of the roles in the target accounts.\n- `organizations:ListAccounts` to run against an AWS organization and capture\n account metadata.\n- `organizations:ListChildren` to run against specific OUs.\n\nIn the organization member accounts:\n\n- A target role that trusts the calling account - for example `AWSControlTowerExecution`\nor\n[`OrganizationAccountAccessRole` role](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_accounts_access.html)\n\nSee the [arguments](#arguments) section for how to change these defaults to work\nwith any account configuration, including running without an AWS Organization.\n\n## Quickstart\n\nA function written to interact with one\n[boto3 session](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html)\ncan now be called with a `session` from every account and region required by\nassuming a role in for each account - except the host account you're running\nfrom.\n\nFor example:\n\nA standard approach: this function takes a boto3 `session` and gets all IAM\nusers from a single AWS account. You would then manually run it in each account.\n\n```python\nimport boto3\n\n\ndef get_iam_users(session):\n iam = session.client(\"iam\", region_name=\"eu-west-1\")\n all_users = iam.get_paginator(\"list_users\").paginate().build_full_result()\n return all_users\n\ndef main():\n session = boto3.session.Session(profile_name=\"my_dev_account\")\n users = get_iam_users(session)\n print(users) # A single account's IAM users\n```\n\nNow with `@cove`: a session for every account in the organization is injected\nby the decorator. A safe example to run as a test!\n\n```python\nfrom pprint import pprint\nimport boto3\nfrom botocove import cove\n\n@cove()\ndef get_iam_users(session):\n iam = session.client(\"iam\", region_name=\"eu-west-1\")\n all_users = iam.get_paginator(\"list_users\").paginate().build_full_result()\n return all_users\n\ndef main():\n # No session passed as the decorator injects it\n all_results = get_iam_users()\n # Now returns a Dict with keys Results, Exceptions and FailedAssumeRole\n\n # A list of dictionaries for each account, with account details included.\n # Each account's get_iam_users return is in a \"Result\" key.\n pprint(all_results[\"Results\"])\n\n # A list of dictionaries for each account that raised an exception\n pprint(all_results[\"Exceptions\"])\n\n # A list of dictionaries for each account that could not be assumed into\n pprint(all_results[\"FailedAssumeRole\"])\n\n\nif __name__ == \"__main__\":\n main()\n```\n\nHere's an example of a more customised Cove decorator:\n\n```python\n@cove(\n target_ids=[\"123456789101\", \"234567891011\"], # also accepts OU ids!\n rolename=\"AWSControlTowerExecution\",\n raise_exception=True,\n regions=[\"eu-west-1\", \"eu-west-2\", \"us-east-1\"],\n)\ndef do_things(session):\n # Cove will return six results of True, 2 accounts * 3 regions\n return True\n```\n\n## Arguments\n\n### Cove\n\n`@cove()`:\n\nUses the standard boto3 credential chain to start with, assuming roles in every\naccount required. Defaults to assuming the `OrganizationAccountAccessRole` in\nevery account in an AWS organization.\n\nEquivalent to:\n\n```python\n@cove(\n target_ids=None, ignore_ids=None, rolename=None, role_session_name=None,\n policy=None, policy_arns=None, assuming_session=None, raise_exception=False,\n thread_workers=20, regions=None, partition=None\n )\n```\n\n`target_ids`: List[str]\n\nA list of AWS account IDs and/or AWS Organization Units IDs to attempt to assume\nrole in to. When unset, attempts to use every available account ID in an AWS\norganization. When specifying target OU's, all child OUs and accounts belonging\nto that OU will be collected.\n\n`ignore_ids`: List[str]\n\nA list of AWS account ID's and OU's to prevent functions being run by Cove.\nIgnored IDs takes precedence over `target_ids`. Providing an OU ID will collect\nall child OUs and accounts to ignore.\n\nThe calling account that is running the Cove-wrapped function at runtime is\nalways ignored.\n\n`rolename`: str\n\nAn IAM role name that will be attempted to assume in all target accounts.\nDefaults to the AWS Organization default, `OrganizationAccountAccessRole`.\n\n`role_session_name`: str\n\nAn IAM role session name that will be passed to each Cove session's\n`sts.assume_role()` call. Defaults to the name of the role being used if unset.\n\n`policy`: str\n\nA policy document that will be used as a session policy. A non-None value is\npassed through via the Policy parameter in each Cove session's\n`sts.assume_role()` call.\n\n`policy_arns`: List[[PolicyDescriptorTypeTypeDef](https://pypi.org/project/mypy-boto3-sts/)]\n\nA list of managed policy ARNs that will be used as a session policy. A non-None\nvalue is passed through via the PolicyArns parameter in each Cove session's\n`sts.assume_role()` call.\n\n`assuming_session`: Session\n\nA Boto3 `boto3.session.Session()` object that will be used to call\n`sts:assumerole`. If not provided, cove will instantiate one which will use the\nstandard boto3 credential chain.\n\n`raise_exception`: bool\n\nDefaults to False. Default behaviour is that exceptions are not raised from\ndecorated function. This is due to `cove` running asynchronously and preferring\nto resolve all tasks and report their results instead of exiting early.\n\n`raise_exception=True` will allow a full stack trace to escape on the first\nexception seen; but will not gracefully or consistently interrupt running tasks.\nIt is vital to run interruptible, idempotent code with this argument as `True`.\n\n`thread_workers`: int\n\nDefaults to 20. Cove utilises a ThreadPoolWorker under the hood, which can be\ntuned with this argument. Number of thread workers directly correlates to memory\nusage: see [here](#is-botocove-thread-safe)\n\n`regions`: List[str]\n\nIf not provided, Cove will respect your profile's default region via the boto\ncredential chain. If provided, Cove will run the decorated function in every\nregion named.\n\nYou can get all regions with:\n\n```python\nregions = [\n r['RegionName'] for r in boto3.client('ec2').describe_regions()['Regions']\n ]\n```\n\n`partition`: str\n\nIf not provided, Cove will use the [AWS partition](https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html)\nof your profile in constructing the ARN for the role to assume in all target\naccounts.\n\n`external_id`: str\n\nDefaults to None. An external id that will be passed to each Cove session's\n`sts.assume_role()` call.\n\n### CoveSession\n\nCove supplies an enriched Boto3 session to each function called. Account details\nare available with the `session_information` attribute.\n\nOtherwise, it functions exactly as calling `boto3` would.\n\n```python\n@cove()\ndef do_nothing(session: CoveSession):\n print(session.session_information) # Outputs a dict of known information\n # This function runs no boto3-specific API, but does test that a role\n # can be assumed\n```\n\n## Return values\n\nWrapped functions return a dictionary. Each value contains List[Dict[str, Any]]:\n\n```python\n{\n \"Results\": results:\n \"Exceptions\": exceptions,\n \"FailedAssumeRole\": invalid_sessions,\n}\n```\n\nAn example of cove_output[\"Results\"]:\n\n```python\n[ # A list of dictionaries per account called\n {\n 'Id': '123456789010',\n 'Email': 'email@address.com',\n 'Name': 'account-name',\n 'Status': 'ACTIVE',\n 'AssumeRoleSuccess': True,\n 'Result': wrapped_function_return_value # Result of wrapped func\n }\n]\n```\n\n### Is botocove thread safe?\n\nbotocove is thread safe, but number of threaded executions will be bound by\nmemory, network IO and AWS api rate limiting. Defaulting to 20 thread workers is\na reasonable starting point, but can be further optimised for runtime with\nexperimentation.\n\nbotocove has no constraint or understanding of the function it's wrapping: it is\nrecommended to avoid shared state for botocove wrapped functions, and to write\nsimple functions that are written to be idempotent and independent.\n\n[Boto3 Session objects are not natively thread safe](https://boto3.amazonaws.com/v1/documentation/api/1.14.31/guide/session.html#multithreading-or-multiprocessing-with-sessions)\n\nand should not be shared across threads. However, botocove is instantiating a\nnew Session object per thread/account and running decorated functions inside\ntheir own closure. A shared client is used from the host account that botocove\nis run from (eg, an organization master account) -\n[clients are threadsafe](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/clients.html#multithreading-or-multiprocessing-with-clients)\nand allow this.\n\nboto3 sessions have a significant memory footprint:\nVersion 1.5.0 of botocove was re-written to ensure that boto3 sessions are\nreleased after completion which resolved memory starvation issues. This was\ndiscussed here: <https://github.com/connelldave/botocove/issues/20> and a\nrelevant boto3 issue is here: <https://github.com/boto/boto3/issues/1670>\n\n### botocove?\n\nIt turns out that the Amazon's Boto dolphins are solitary or small-group\nanimals, unlike the large pods of dolphins in the oceans. This killed my \"large\ngroup of boto\" idea, so the next best idea was where might they all shelter\ntogether... a cove!\n",
"bugtrack_url": null,
"license": "LGPL-3.0-or-later",
"summary": "A decorator to allow running a function against all AWS accounts in an organization",
"version": "1.7.4",
"project_urls": {
"Homepage": "https://github.com/connelldave/botocove",
"Repository": "https://github.com/connelldave/botocove"
},
"split_keywords": [
"aws",
"organizations",
"aws organizations",
"boto3"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "60b683ebeeab5d796db8982d961ee31b41875f9b76802f4fc96bb82eab8c5f3f",
"md5": "0583f3dc4c5786d8bd9bddeddaca5285",
"sha256": "33c95faf09bb5bc068524e4471361d625ae6188ea6df94d2562922416dbb9966"
},
"downloads": -1,
"filename": "botocove-1.7.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0583f3dc4c5786d8bd9bddeddaca5285",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8,<4.0",
"size": 15747,
"upload_time": "2023-11-26T16:35:06",
"upload_time_iso_8601": "2023-11-26T16:35:06.007487Z",
"url": "https://files.pythonhosted.org/packages/60/b6/83ebeeab5d796db8982d961ee31b41875f9b76802f4fc96bb82eab8c5f3f/botocove-1.7.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "408d318b1de0aecb0f8f555aa4ec7f4ea9ecee3f7f7f4f6c36f41c9e7175085c",
"md5": "c11e5aeab9a77142e3a1cae345eea87c",
"sha256": "60f1ad287008e388b80bc26961ae799efeec48eb7dc969c9e4806e3f281f89af"
},
"downloads": -1,
"filename": "botocove-1.7.4.tar.gz",
"has_sig": false,
"md5_digest": "c11e5aeab9a77142e3a1cae345eea87c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8,<4.0",
"size": 17726,
"upload_time": "2023-11-26T16:35:07",
"upload_time_iso_8601": "2023-11-26T16:35:07.281595Z",
"url": "https://files.pythonhosted.org/packages/40/8d/318b1de0aecb0f8f555aa4ec7f4ea9ecee3f7f7f4f6c36f41c9e7175085c/botocove-1.7.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-26 16:35:07",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "connelldave",
"github_project": "botocove",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "botocove"
}
JSON
