| Name | botocache JSON |
| Version |
0.0.6
JSON |
| download |
| home_page | |
| Summary | Caching for Boto and Boto3 SDK |
| upload_time | 2024-03-13 18:05:37 |
| maintainer | |
| docs_url | None |
| author | rams3sh |
| requires_python | >=3.8 |
| license | |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Botocache
Caching layer for Boto / Botocore libraries.
## Background
---
This project was started to solve the issue raised [here](https://github.com/boto/boto3/issues/2723).
My day job requires me to write and use multiple tools and standalone scripts to audit AWS environments.
Most of these tools are hacky and written in python which uses boto3 / botocore in the backend to interact with AWS API.
Sometimes, I would have to write a wrapper to combine these scripts to get a custom consolidated report.
Since these scripts are standalone, they repeat the same API calls that a previous script would have already
called before, leading to redundant API calls, throttling and unnecessary IO wait.
These wrapped tools are sometimes even used as Lambda for automation of certain things.
The IO wait times becomes a bottleneck when used in environment such as Lambda where execution is a
time-bound activity.
Hence I was looking for a caching layer over boto that can solve reducing these direct redundant calls and
the wait times. Thus the birth of botocache.
## About
---
Botocache caches the response of API calls initiated through boto3 / botocore.
Any subsequent redundant call will end up getting the previously cached response from Botocache as long as the call is
within the expiry timeout of the cached response.
Botocache can work with any cache library that is based on [cachetools](https://github.com/tkem/cachetools/)
It uses the unittest module's patch as the magic component to achieve this. :wink:
This project is little hacky given the nature how it achieves caching, but it gets the job done.
## Installation
---
Stable release installation (PyPI) :-
```bash
pip3 install botocache
```
Test release installation (Test PyPI)
```bash
pip install -i https://test.pypi.org/pypi/ --extra-index-url https://pypi.org/simple botocache
```
Installation directly from this Repository:-
```bash
pip3 install git+https://github.com/rams3sh/botocache.git
```
## Usage
---
Below snippet demonstrates usage of botocache.
```python
from boto3.session import Session
from cachetools_ext.fs import FSLRUCache
from botocache.botocache import botocache_context
cache = FSLRUCache(ttl=900, path=".cache", maxsize=1000)
# action_regex_to_cache parameter consists list of regex to be matched against a given action for considering the call to be cached
with botocache_context(cache=cache,
action_regex_to_cache=["List.*", "Get.*", "Describe.*"],
call_log=True, # This helps in logging all calls made to AWS. Useful while debugging. Default value is False.
supress_warning_message=False # This supresses warning messages encountered while caching. Default value is False.
):
cached_session = Session()
cached_client = cached_session.client('iam')
paginator = cached_client.get_paginator('list_users')
for page in paginator.paginate():
print(page)
"""
Don't do this, if you want to have a new session without caching.
The below paginator object was initialised under the botocache context which means it's subsequent
attributes was initialised with patched Botocache class leading it to still use the backend cache.
"""
for page in paginator.paginate():
print(page)
"""
Always use a fresh initialised session client and subsequent new objects outside the context of botocache to use
boto3 without caching layer. Below is an example.
"""
non_cached_session = Session()
non_cached_client = non_cached_session.client('iam')
paginator = non_cached_client.get_paginator('list_users')
for page in paginator.paginate():
print(page)
```
**Note:**
Botocache has been tested with cachetools_ext's FSLRU cache, but it should logically work with any cache that is
compatible with cachetools.
## Disclaimer(s)
---
* This project was created mainly to support my specific internal use cases.
Hence, there is a good scope of it having bugs and functional issues. Feel free to raise a PR / Issue in those cases.
* Botocache does not understand [HTTP related caching specification](https://tools.ietf.org/html/rfc7234).
It works based on function caching.Its a very simple dumb caching layer that checks for specific attributes in an API call, converts into a key
and stores the response against the key.
Any subsequent call having matching attributes will be returned with the value stored against the same key.
Raw data
{
"_id": null,
"home_page": "",
"name": "botocache",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "",
"author": "rams3sh",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/17/ac/d1093531a52a6c227b132ab3923dc6351cf5de3117e98720c2261a01043e/botocache-0.0.6.tar.gz",
"platform": null,
"description": "# Botocache\n\nCaching layer for Boto / Botocore libraries.\n\n\n## Background\n\n---\n\nThis project was started to solve the issue raised [here](https://github.com/boto/boto3/issues/2723).\n\nMy day job requires me to write and use multiple tools and standalone scripts to audit AWS environments. \nMost of these tools are hacky and written in python which uses boto3 / botocore in the backend to interact with AWS API.\n\nSometimes, I would have to write a wrapper to combine these scripts to get a custom consolidated report. \nSince these scripts are standalone, they repeat the same API calls that a previous script would have already \ncalled before, leading to redundant API calls, throttling and unnecessary IO wait. \n\nThese wrapped tools are sometimes even used as Lambda for automation of certain things. \nThe IO wait times becomes a bottleneck when used in environment such as Lambda where execution is a \ntime-bound activity. \n\nHence I was looking for a caching layer over boto that can solve reducing these direct redundant calls and \nthe wait times. Thus the birth of botocache.\n\n\n## About\n\n---\n\nBotocache caches the response of API calls initiated through boto3 / botocore.\nAny subsequent redundant call will end up getting the previously cached response from Botocache as long as the call is\nwithin the expiry timeout of the cached response. \n\nBotocache can work with any cache library that is based on [cachetools](https://github.com/tkem/cachetools/)\n\nIt uses the unittest module's patch as the magic component to achieve this. :wink:\n\nThis project is little hacky given the nature how it achieves caching, but it gets the job done. \n\n\n## Installation\n\n---\nStable release installation (PyPI) :-\n```bash\npip3 install botocache\n```\n\nTest release installation (Test PyPI)\n```bash\npip install -i https://test.pypi.org/pypi/ --extra-index-url https://pypi.org/simple botocache\n```\n\nInstallation directly from this Repository:-\n```bash\npip3 install git+https://github.com/rams3sh/botocache.git\n```\n\n\n## Usage\n\n---\n\nBelow snippet demonstrates usage of botocache.\n\n```python\nfrom boto3.session import Session\nfrom cachetools_ext.fs import FSLRUCache\n\nfrom botocache.botocache import botocache_context\n\ncache = FSLRUCache(ttl=900, path=\".cache\", maxsize=1000)\n\n# action_regex_to_cache parameter consists list of regex to be matched against a given action for considering the call to be cached\nwith botocache_context(cache=cache,\n action_regex_to_cache=[\"List.*\", \"Get.*\", \"Describe.*\"], \n call_log=True, # This helps in logging all calls made to AWS. Useful while debugging. Default value is False.\n supress_warning_message=False # This supresses warning messages encountered while caching. Default value is False. \n ):\n cached_session = Session()\n cached_client = cached_session.client('iam')\n paginator = cached_client.get_paginator('list_users')\n for page in paginator.paginate():\n print(page)\n\n\"\"\"\nDon't do this, if you want to have a new session without caching. \nThe below paginator object was initialised under the botocache context which means it's subsequent \nattributes was initialised with patched Botocache class leading it to still use the backend cache. \n\"\"\"\nfor page in paginator.paginate():\n print(page)\n\n\"\"\"\nAlways use a fresh initialised session client and subsequent new objects outside the context of botocache to use \nboto3 without caching layer. Below is an example. \n\"\"\"\n\nnon_cached_session = Session()\nnon_cached_client = non_cached_session.client('iam')\npaginator = non_cached_client.get_paginator('list_users')\nfor page in paginator.paginate():\n print(page)\n\n```\n**Note:** \n\nBotocache has been tested with cachetools_ext's FSLRU cache, but it should logically work with any cache that is \ncompatible with cachetools.\n\n\n## Disclaimer(s)\n\n---\n\n* This project was created mainly to support my specific internal use cases. \nHence, there is a good scope of it having bugs and functional issues. Feel free to raise a PR / Issue in those cases.\n\n\n* Botocache does not understand [HTTP related caching specification](https://tools.ietf.org/html/rfc7234).\nIt works based on function caching.Its a very simple dumb caching layer that checks for specific attributes in an API call, converts into a key \nand stores the response against the key. \nAny subsequent call having matching attributes will be returned with the value stored against the same key.\n",
"bugtrack_url": null,
"license": "",
"summary": "Caching for Boto and Boto3 SDK",
"version": "0.0.6",
"project_urls": null,
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f8c4e140fd6dac246d81861695b46be056ee09747fdec65da23b81ac77c5a25b",
"md5": "5156a245533b05869f3b090e3c85ddd7",
"sha256": "a8ca51dca7bb5e2319cdc9095ef0f49dbb30a028d1c335471ca696c161f0e0e4"
},
"downloads": -1,
"filename": "botocache-0.0.6-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5156a245533b05869f3b090e3c85ddd7",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 3997,
"upload_time": "2024-03-13T18:05:34",
"upload_time_iso_8601": "2024-03-13T18:05:34.448107Z",
"url": "https://files.pythonhosted.org/packages/f8/c4/e140fd6dac246d81861695b46be056ee09747fdec65da23b81ac77c5a25b/botocache-0.0.6-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "17acd1093531a52a6c227b132ab3923dc6351cf5de3117e98720c2261a01043e",
"md5": "89cf23d6468a5b4dc8892631babf3dc0",
"sha256": "657752e86d36cb1bd4b9197a25800bb716029623ec1c778dea542443de5b6758"
},
"downloads": -1,
"filename": "botocache-0.0.6.tar.gz",
"has_sig": false,
"md5_digest": "89cf23d6468a5b4dc8892631babf3dc0",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 3543,
"upload_time": "2024-03-13T18:05:37",
"upload_time_iso_8601": "2024-03-13T18:05:37.563079Z",
"url": "https://files.pythonhosted.org/packages/17/ac/d1093531a52a6c227b132ab3923dc6351cf5de3117e98720c2261a01043e/botocache-0.0.6.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-13 18:05:37",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "botocache"
}
