# `aboto3`
`aboto3` is an async boto3 client generator!
There are other boto3-like libraries that offer asyncio but the interface can be quite different from normal `boto3` clients.
The goal of `aboto3` is to closely replicate the `boto3` client interface with acceptable performance from the python [`ThreadPoolExecutor`](https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor)!
> **API NOTE** - `aboto3` was created with boto3 API compatibility in mind. Because of this it does not support [`boto3` "Resources"](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html), and there is no plan to support them. New "Resources" are no longer being added to `boto3`.
> **Performance NOTE** - Because `aboto3` provides an async wrapper around `boto3`, it is not truly async to it's core. It sends a boto3 call to a thread and the thread runs asynchronously. `asyncio` tasks are much lighter weight than threads so if you want to run hundreds of concurrent calls with the best performance, pure async boto3 equivalents are a better fit.
## Tutorial
To create an async client simply pass in the normal `boto3` client to create an `AIOClient`.
Use the async client, in a coroutine, like you would if the boto3 client's API calls were all async! See [`boto3` docs](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) for details.
```python
import asyncio
import aboto3
import boto3
# create a normal boto3 client
ec2_client = boto3.client("ec2")
# create the asyncio version from the client
aio_ec2_client = aboto3.AIOClient(ec2_client)
# you can still use the other client as usual
instances = ec2_client.describe_instances()
# the async client must be used in a coroutine
# but acts exactly the same as the boto3 client except method calls are async
async def aio_tester():
aio_instances = await aio_ec2_client.describe_instances()
return aio_instances
aio_instances = asyncio.run(aio_tester())
```
Pass in parameters to the coroutine like a normal client.
```python
import asyncio
import aboto3
import boto3
ec2_client = boto3.client("ec2")
aio_ec2_client = aboto3.AIOClient(ec2_client)
instances = ec2_client.describe_instances(InstanceIds=["i-123412341234"])
async def aio_tester():
aio_instances = await aio_ec2_client.describe_instances(InstanceIds=["i-123412341234"])
return aio_instances
aio_instances = asyncio.run(aio_tester())
```
Get an async paginator from the `aboto3` client.
```python
import asyncio
import aboto3
import boto3
ec2_client = boto3.client("ec2")
aio_ec2_client = aboto3.AIOClient(ec2_client)
filters = [
{
"Name": "instance-type",
"Values": [
"t2.micro"
]
}
]
pages = []
pager = ec2_client.get_paginator("describe_instances")
for page in pager.paginate(Filters=filters):
pages.append(page)
# note the use of an "async for" loop so calls for a page are non-blocking.
async def aio_tester():
aio_pages = []
aio_pager = aio_ec2_client.get_paginator("describe_instances")
async for page in aio_pager.paginate(Filters=filters):
aio_pages.append(page)
return aio_pages
aio_pages = asyncio.run(aio_tester())
```
Client exceptions can be caught on the `AIOClient` just like a normal `boto3` client.
`botocore` exceptions are caught as normal.
```python
import asyncio
import aboto3
import boto3
ssm_client = boto3.client("ssm")
aio_ssm_client = aboto3.AIOClient(ssm_client)
try:
ssm_client.get_parameter(Name="/unknown/param")
except ssm_client.exceptions.ParameterNotFound as error:
print("found an error here: {}".format(error))
async def aio_tester():
try:
aio_ssm_client.get_parameter(Name="/unknown/param")
except aio_ssm_client.exceptions.ParameterNotFound as error:
print("found an error here: {}".format(error))
aio_pages = asyncio.run(aio_tester())
```
You can also use `boto3` augmenting libraries since `aboto3` is only a wrapper.
## Optimization
When an `AIOClient` is created it will automatically create a `ThreadPoolExecutor` to run the boto3 calls asynchronously. The size of max workers of the pool is determined by the boto3 client's config for `max_pool_connections`. By default this is 10.
See [botocore Config Reference](https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html#botocore-config) for more details.
The thread pool adds a small amount of overhead for each `AIOClient` that is created (though this is far less than the overhead of creating a boto3 client). To save some initialization time or have more control over total number of threads you can provide your own `ThreadPoolExecutor` and share this between clients.
```python
import asyncio
from concurrent.futures import ThreadPoolExecutor
import aboto3
import boto3
boto3_thread_pool = ThreadPoolExecutor(max_workers=16)
ec2_client = boto3.client("ec2")
aio_ec2_client = AIOClient(
boto3_client=ec2_client,
thread_pool_executor=boto3_thread_pool
)
rds_client = boto3.client("rds")
aio_rds_client = AIOClient(
boto3_client=rds_client,
thread_pool_executor=boto3_thread_pool
)
```
In general, for applications, you will want to cache the clients if possible. Try not to create a new one in every function. For applications, a shared thread pool can be useful in limiting the total number of threads, when necessary.
If you are making large numbers of concurrent calls with the same `AIOClient` you may want to pass in a custom `botocore.config.Config` to the boto3 client with a higher `max_pool_connections`. If you are using a shared thread pool you may also need to increase the max workers in that as well.
The example below will allow up to 32 concurrent calls to be in flight for the EC2 and RDS `AIOClient`'s.
```python
import asyncio
from concurrent.futures import ThreadPoolExecutor
import aboto3
import boto3
from botocore.config import Config
boto_config = Config(
max_pool_connections=32
)
boto3_thread_pool = ThreadPoolExecutor(max_workers=64)
ec2_client = boto3.client("ec2", config=boto_config)
aio_ec2_client = AIOClient(
boto3_client=ec2_client,
thread_pool_executor=boto3_thread_pool
)
rds_client = boto3.client("rds", config=boto_config)
aio_rds_client = AIOClient(
boto3_client=rds_client,
thread_pool_executor=boto3_thread_pool
)
```
Or if you don't care about sharing the thread pool just pass in the config
and each `AIOClient` will have it's own pool of 32 threads.
```python
import asyncio
import aboto3
import boto3
from botocore.config import Config
boto_config = Config(
max_pool_connections=32
)
ec2_client = boto3.client("ec2", config=boto_config)
aio_ec2_client = AIOClient(boto3_client=ec2_client)
rds_client = boto3.client("rds", config=boto_config)
aio_rds_client = AIOClient(boto3_client=rds_client)
```
## Development
Install the package in editable mode with dev dependencies.
```text
(venv) $ pip install -e .[dev]
```
[`nox`](https://nox.thea.codes/en/stable/) is used to manage various dev functions.
Start with
```text
(venv) $ nox --help
```
[`pyenv`](https://github.com/pyenv/pyenv) is used to manage python versions.
To run the nox tests for applicable python version you will first need to install them.
In the root project dir run:
```text
(venv) $ pyenv install
```
# Changelog
Changelog for `aboto3`.
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
<!--
## [Unreleased] - YYYY-MM-DD
### Added
### Changed
### Deprecated
### Removed
### Fixed
### Security
-->
## [0.1.1] - 2024-01-28
### Added
Tests for python 3.12
### Changed
Updated README to reflect recommendations.
### Removed
Support for python 3.7
## [0.1.0] - 2023-06-22
Initial Release.
Raw data
{
"_id": null,
"home_page": "https://github.com/btemplep/aboto3",
"name": "aboto3",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "aio,async,asyncio,aws,boto3,botocore,sdk",
"author": "Brandon Temple Paul",
"author_email": "btemplepgit@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/e2/6a/b713370dc51efd11ab80b77f2baa293b63201d11f9e350bddbc8780eb9ed/aboto3-0.1.1.tar.gz",
"platform": null,
"description": "# `aboto3`\n\n`aboto3` is an async boto3 client generator!\n\nThere are other boto3-like libraries that offer asyncio but the interface can be quite different from normal `boto3` clients. \nThe goal of `aboto3` is to closely replicate the `boto3` client interface with acceptable performance from the python [`ThreadPoolExecutor`](https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor)! \n\n> **API NOTE** - `aboto3` was created with boto3 API compatibility in mind. Because of this it does not support [`boto3` \"Resources\"](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html), and there is no plan to support them. New \"Resources\" are no longer being added to `boto3`.\n\n> **Performance NOTE** - Because `aboto3` provides an async wrapper around `boto3`, it is not truly async to it's core. It sends a boto3 call to a thread and the thread runs asynchronously. `asyncio` tasks are much lighter weight than threads so if you want to run hundreds of concurrent calls with the best performance, pure async boto3 equivalents are a better fit.\n\n\n## Tutorial\n\nTo create an async client simply pass in the normal `boto3` client to create an `AIOClient`. \n\nUse the async client, in a coroutine, like you would if the boto3 client's API calls were all async! See [`boto3` docs](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) for details.\n\n```python\nimport asyncio\n\nimport aboto3\nimport boto3\n\n# create a normal boto3 client\nec2_client = boto3.client(\"ec2\")\n# create the asyncio version from the client\naio_ec2_client = aboto3.AIOClient(ec2_client)\n\n# you can still use the other client as usual\ninstances = ec2_client.describe_instances()\n\n# the async client must be used in a coroutine\n# but acts exactly the same as the boto3 client except method calls are async\nasync def aio_tester():\n aio_instances = await aio_ec2_client.describe_instances()\n\n return aio_instances\n\n\naio_instances = asyncio.run(aio_tester())\n```\n\nPass in parameters to the coroutine like a normal client.\n\n```python\nimport asyncio\n\nimport aboto3\nimport boto3\n\nec2_client = boto3.client(\"ec2\")\naio_ec2_client = aboto3.AIOClient(ec2_client)\n\n\ninstances = ec2_client.describe_instances(InstanceIds=[\"i-123412341234\"])\n\n\nasync def aio_tester():\n aio_instances = await aio_ec2_client.describe_instances(InstanceIds=[\"i-123412341234\"])\n\n return aio_instances\n\n\naio_instances = asyncio.run(aio_tester())\n```\n\nGet an async paginator from the `aboto3` client.\n\n```python\nimport asyncio\n\nimport aboto3\nimport boto3\n\nec2_client = boto3.client(\"ec2\")\naio_ec2_client = aboto3.AIOClient(ec2_client)\nfilters = [\n {\n \"Name\": \"instance-type\",\n \"Values\": [\n \"t2.micro\"\n ]\n }\n]\n\npages = []\npager = ec2_client.get_paginator(\"describe_instances\")\nfor page in pager.paginate(Filters=filters):\n pages.append(page)\n\n# note the use of an \"async for\" loop so calls for a page are non-blocking.\nasync def aio_tester():\n aio_pages = []\n aio_pager = aio_ec2_client.get_paginator(\"describe_instances\")\n async for page in aio_pager.paginate(Filters=filters):\n aio_pages.append(page)\n\n return aio_pages\n\n\naio_pages = asyncio.run(aio_tester())\n```\n\nClient exceptions can be caught on the `AIOClient` just like a normal `boto3` client.\n`botocore` exceptions are caught as normal.\n\n```python\nimport asyncio\n\nimport aboto3\nimport boto3\n\nssm_client = boto3.client(\"ssm\")\naio_ssm_client = aboto3.AIOClient(ssm_client)\n\ntry:\n ssm_client.get_parameter(Name=\"/unknown/param\")\nexcept ssm_client.exceptions.ParameterNotFound as error:\n print(\"found an error here: {}\".format(error))\n\n\nasync def aio_tester():\n try:\n aio_ssm_client.get_parameter(Name=\"/unknown/param\")\n except aio_ssm_client.exceptions.ParameterNotFound as error:\n print(\"found an error here: {}\".format(error))\n\n\naio_pages = asyncio.run(aio_tester())\n```\n\nYou can also use `boto3` augmenting libraries since `aboto3` is only a wrapper. \n\n\n## Optimization \n\nWhen an `AIOClient` is created it will automatically create a `ThreadPoolExecutor` to run the boto3 calls asynchronously. The size of max workers of the pool is determined by the boto3 client's config for `max_pool_connections`. By default this is 10.\nSee [botocore Config Reference](https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html#botocore-config) for more details.\n\nThe thread pool adds a small amount of overhead for each `AIOClient` that is created (though this is far less than the overhead of creating a boto3 client). To save some initialization time or have more control over total number of threads you can provide your own `ThreadPoolExecutor` and share this between clients.\n\n\n```python\nimport asyncio\nfrom concurrent.futures import ThreadPoolExecutor\n\nimport aboto3\nimport boto3\n\nboto3_thread_pool = ThreadPoolExecutor(max_workers=16)\n\nec2_client = boto3.client(\"ec2\")\naio_ec2_client = AIOClient(\n boto3_client=ec2_client, \n thread_pool_executor=boto3_thread_pool\n)\n\nrds_client = boto3.client(\"rds\")\naio_rds_client = AIOClient(\n boto3_client=rds_client, \n thread_pool_executor=boto3_thread_pool\n)\n\n```\n\nIn general, for applications, you will want to cache the clients if possible. Try not to create a new one in every function. For applications, a shared thread pool can be useful in limiting the total number of threads, when necessary. \n\nIf you are making large numbers of concurrent calls with the same `AIOClient` you may want to pass in a custom `botocore.config.Config` to the boto3 client with a higher `max_pool_connections`. If you are using a shared thread pool you may also need to increase the max workers in that as well. \n\nThe example below will allow up to 32 concurrent calls to be in flight for the EC2 and RDS `AIOClient`'s.\n\n```python\nimport asyncio\nfrom concurrent.futures import ThreadPoolExecutor\n\nimport aboto3\nimport boto3\nfrom botocore.config import Config\n\nboto_config = Config(\n max_pool_connections=32\n)\nboto3_thread_pool = ThreadPoolExecutor(max_workers=64)\n\nec2_client = boto3.client(\"ec2\", config=boto_config)\naio_ec2_client = AIOClient(\n boto3_client=ec2_client, \n thread_pool_executor=boto3_thread_pool\n)\nrds_client = boto3.client(\"rds\", config=boto_config)\naio_rds_client = AIOClient(\n boto3_client=rds_client, \n thread_pool_executor=boto3_thread_pool\n)\n\n```\n\nOr if you don't care about sharing the thread pool just pass in the config \nand each `AIOClient` will have it's own pool of 32 threads. \n\n```python\nimport asyncio\n\nimport aboto3\nimport boto3\nfrom botocore.config import Config\n\nboto_config = Config(\n max_pool_connections=32\n)\n\nec2_client = boto3.client(\"ec2\", config=boto_config)\naio_ec2_client = AIOClient(boto3_client=ec2_client)\n\nrds_client = boto3.client(\"rds\", config=boto_config)\naio_rds_client = AIOClient(boto3_client=rds_client)\n```\n\n\n## Development\n\nInstall the package in editable mode with dev dependencies.\n\n```text\n(venv) $ pip install -e .[dev]\n```\n\n[`nox`](https://nox.thea.codes/en/stable/) is used to manage various dev functions.\nStart with\n\n```text\n(venv) $ nox --help\n```\n\n[`pyenv`](https://github.com/pyenv/pyenv) is used to manage python versions. \nTo run the nox tests for applicable python version you will first need to install them. \nIn the root project dir run:\n\n```text\n(venv) $ pyenv install\n```\n\n# Changelog\n\nChangelog for `aboto3`.\nAll notable changes to this project will be documented in this file.\n\nThe format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),\nand this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).\n\n<!-- \n## [Unreleased] - YYYY-MM-DD\n\n### Added\n\n### Changed\n\n### Deprecated\n\n### Removed\n\n### Fixed\n\n### Security \n-->\n\n\n## [0.1.1] - 2024-01-28\n\n### Added\n\nTests for python 3.12\n\n### Changed\n\nUpdated README to reflect recommendations.\n\n### Removed\n\nSupport for python 3.7\n\n\n## [0.1.0] - 2023-06-22\n\nInitial Release.\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Async boto3 client generator.",
"version": "0.1.1",
"project_urls": {
"Homepage": "https://github.com/btemplep/aboto3"
},
"split_keywords": [
"aio",
"async",
"asyncio",
"aws",
"boto3",
"botocore",
"sdk"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "3ccbc63440c0a49286e5529dedbd80c6122989a0f77067932e3a013dccf20b3c",
"md5": "d4759c845db509642efe42387733016f",
"sha256": "74f90377b4ec75950f5713a21d1f451f104b5fc1c6de0d0fd60850dac597e626"
},
"downloads": -1,
"filename": "aboto3-0.1.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d4759c845db509642efe42387733016f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 6666,
"upload_time": "2024-01-29T03:13:21",
"upload_time_iso_8601": "2024-01-29T03:13:21.425812Z",
"url": "https://files.pythonhosted.org/packages/3c/cb/c63440c0a49286e5529dedbd80c6122989a0f77067932e3a013dccf20b3c/aboto3-0.1.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "e26ab713370dc51efd11ab80b77f2baa293b63201d11f9e350bddbc8780eb9ed",
"md5": "9d9c2931a5170359db403240f01ae8be",
"sha256": "376a10715aa1151bf5274dd330aa24126c5f57d575a19200a42a370dc0994eea"
},
"downloads": -1,
"filename": "aboto3-0.1.1.tar.gz",
"has_sig": false,
"md5_digest": "9d9c2931a5170359db403240f01ae8be",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 6808,
"upload_time": "2024-01-29T03:13:22",
"upload_time_iso_8601": "2024-01-29T03:13:22.589055Z",
"url": "https://files.pythonhosted.org/packages/e2/6a/b713370dc51efd11ab80b77f2baa293b63201d11f9e350bddbc8780eb9ed/aboto3-0.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-29 03:13:22",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "btemplep",
"github_project": "aboto3",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "aboto3"
}
JSON
