| Name | leaky-bucket-py JSON |
| Version |
0.1.3
JSON |
| download |
| home_page | None |
| Summary | Leaky bucket implementation in python with different persistence options |
| upload_time | 2025-01-01 23:59:25 |
| maintainer | None |
| docs_url | None |
| author | DuneRaccoon |
| requires_python | >=3.7 |
| license | MIT License Copyright (c) 2024 Benjamin Herro Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
rate-limiter
asyncio
leaky-bucket
redis
sqlite3
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Py Leaky Bucket
An implementation of the leaky bucket algorithm in python, with different persistence options for use in high throughput, multi process/worker applications. This is a useful package when managing integrations with various API's that have different rate limits.
## What's in it?
The package includes:
- **Leaky Bucket Algorithm**: A flexible rate-limiting algorithm that supports various persistence backends.
- **Persistence Backends**:
- **In-Memory**: Fast, lightweight, and great for single-process applications.
- **Redis**: Suitable for distributed, multi-worker environments.
- **SQLite**: A file-based backend that supports high concurrency in single-node setups (honestly not the best option for high-throughput due to the nature of sqlite and the need to deadlock the database - better to use the redis option if possible)
- **Hourly Limit Support**: Control total operations over a rolling hourly window in addition to per-second rate limits.
- **Thread-Safe and Process-Safe**: Implements proper locking mechanisms to ensure safe concurrent usage.
- **Asynchronous and Synchronous**: Works in both `asyncio`-based and synchronous applications.
- **Decorators and Context Managers**: Simplify integration with your existing functions and methods.
### Installation
Easy to install:
```
pip install leaky-bucket-py
```
## Usage:
#### Redis backend with async bucket:
```python
import asyncio
import redis
from leakybucket.bucket import AsyncLeakyBucket
from leakybucket.persistence.redis import RedisLeakyBucketStorage
# Connect to Redis
redis_conn = redis.Redis(host='localhost', port=6379, db=0)
# Create a new Redis storage backend
storage = RedisLeakyBucketStorage(
redis_conn,
redis_key="api_bucket",
max_rate=5,
time_period=1
)
# Create a new LeakyBucket instance
bucket = AsyncLeakyBucket(storage)
# Make requests using the bucket as a context manager
async def make_requests():
async def make_request():
async with bucket: # block if the rate limit is exceeded
print("Making request")
await asyncio.sleep(1)
await asyncio.gather(*[make_request() for i in range(10)])
# or use a decorator to rate limit a coroutine
@bucket.throttle()
async def make_request(index):
print(f"Making request {index}")
await asyncio.sleep(1)
async def main():
await make_requests()
await asyncio.gather(*[make_request(i) for i in range(10)])
asyncio.run(main())
```
#### Memory backend:
###### Synchronous:
```python
import httpx
from leakybucket.bucket import LeakyBucket
from leakybucket.persistence.memory import InMemoryLeakyBucketStorage
# Create a new Memory storage backend (3 requests per second)
storage = InMemoryLeakyBucketStorage(max_rate=3, time_period=1)
# Create a new LeakyBucket instance
throttler = LeakyBucket(storage)
@throttler.throttle()
def fetch_data(api_url: str):
response = httpx.get(api_url)
data = response.json()
print(data)
return data
def main():
# make multiple requests
api_url = "https://jsonplaceholder.typicode.com/posts/1"
results = []
for _ in range(10):
results.append(fetch_data(api_url))
print(results)
main()
```
###### Asynchronous:
```python
import asyncio
import httpx
from leakybucket.bucket import AsyncLeakyBucket
from leakybucket.persistence.memory import InMemoryLeakyBucketStorage
# Create a new Memory storage backend (3 requests per second)
storage = InMemoryLeakyBucketStorage(max_rate=3, time_period=1)
# Create a new LeakyBucket instance
async_throttler = AsyncLeakyBucket(storage)
@async_throttler.throttle()
async def async_fetch_data(api_url):
async with httpx.AsyncClient() as client:
response = await client.get(api_url)
data = response.json()
print(data)
return data
async def main():
# make multiple requests
api_url = "https://jsonplaceholder.typicode.com/posts/1"
tasks = [async_fetch_data(api_url) for _ in range(10)]
results = await asyncio.gather(*tasks)
print(results)
asyncio.run(main())
```
#### Sqlite backend:
```python
import time
from leakybucket.bucket import LeakyBucket
from leakybucket.persistence.sqlite import SqliteLeakyBucketStorage
# Create a shared SQLite bucket
bucket = LeakyBucket(
SqliteLeakyBucketStorage(
db_path="leakybucket.db",
max_rate=10,
time_period=10
)
)
# Decorate the function
@bucket.throttle()
def make_request(index):
print(f"Making request {index}")
def main():
for i in range(35):
make_request(i)
main()
```
Raw data
{
"_id": null,
"home_page": null,
"name": "leaky-bucket-py",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": null,
"keywords": "rate-limiter, asyncio, leaky-bucket, redis, sqlite3",
"author": "DuneRaccoon",
"author_email": "DuneRaccoon <benjamincsherro@hotmail.com>",
"download_url": "https://files.pythonhosted.org/packages/85/4f/9111cf73bb3be09314f7c3df889bfc1e51bb204a1f67a53c6c2ae061b5f3/leaky_bucket_py-0.1.3.tar.gz",
"platform": null,
"description": "# Py Leaky Bucket\n\nAn implementation of the leaky bucket algorithm in python, with different persistence options for use in high throughput, multi process/worker applications. This is a useful package when managing integrations with various API's that have different rate limits.\n\n\n\n## What's in it?\n\nThe package includes:\n\n- **Leaky Bucket Algorithm**: A flexible rate-limiting algorithm that supports various persistence backends.\n- **Persistence Backends**:\n - **In-Memory**: Fast, lightweight, and great for single-process applications.\n - **Redis**: Suitable for distributed, multi-worker environments.\n - **SQLite**: A file-based backend that supports high concurrency in single-node setups (honestly not the best option for high-throughput due to the nature of sqlite and the need to deadlock the database - better to use the redis option if possible)\n- **Hourly Limit Support**: Control total operations over a rolling hourly window in addition to per-second rate limits.\n- **Thread-Safe and Process-Safe**: Implements proper locking mechanisms to ensure safe concurrent usage.\n- **Asynchronous and Synchronous**: Works in both `asyncio`-based and synchronous applications.\n- **Decorators and Context Managers**: Simplify integration with your existing functions and methods.\n\n\n\n### Installation\n\nEasy to install:\n\n```\npip install leaky-bucket-py\n```\n\n\n\n## Usage:\n\n\n\n#### Redis backend with async bucket:\n\n```python\nimport asyncio\nimport redis\nfrom leakybucket.bucket import AsyncLeakyBucket\nfrom leakybucket.persistence.redis import RedisLeakyBucketStorage\n\n# Connect to Redis\nredis_conn = redis.Redis(host='localhost', port=6379, db=0)\n\n# Create a new Redis storage backend\nstorage = RedisLeakyBucketStorage(\n redis_conn,\n redis_key=\"api_bucket\",\n max_rate=5,\n time_period=1\n)\n\n# Create a new LeakyBucket instance\nbucket = AsyncLeakyBucket(storage)\n\n# Make requests using the bucket as a context manager\nasync def make_requests():\n async def make_request():\n async with bucket: # block if the rate limit is exceeded\n print(\"Making request\")\n await asyncio.sleep(1)\n await asyncio.gather(*[make_request() for i in range(10)])\n\n\n# or use a decorator to rate limit a coroutine\n@bucket.throttle()\nasync def make_request(index):\n print(f\"Making request {index}\")\n await asyncio.sleep(1)\n\n\nasync def main():\n await make_requests()\n await asyncio.gather(*[make_request(i) for i in range(10)])\n\n\nasyncio.run(main())\n```\n\n\n\n#### Memory backend:\n\n###### Synchronous:\n\n```python\nimport httpx\nfrom leakybucket.bucket import LeakyBucket\nfrom leakybucket.persistence.memory import InMemoryLeakyBucketStorage\n\n# Create a new Memory storage backend (3 requests per second)\nstorage = InMemoryLeakyBucketStorage(max_rate=3, time_period=1)\n\n# Create a new LeakyBucket instance\nthrottler = LeakyBucket(storage)\n\n@throttler.throttle()\ndef fetch_data(api_url: str):\n response = httpx.get(api_url)\n data = response.json()\n print(data)\n return data\n\ndef main():\n # make multiple requests\n api_url = \"https://jsonplaceholder.typicode.com/posts/1\"\n results = []\n for _ in range(10):\n results.append(fetch_data(api_url))\n print(results)\n\nmain()\n\n```\n\n###### Asynchronous:\n\n```python\nimport asyncio\nimport httpx\nfrom leakybucket.bucket import AsyncLeakyBucket\nfrom leakybucket.persistence.memory import InMemoryLeakyBucketStorage\n\n# Create a new Memory storage backend (3 requests per second)\nstorage = InMemoryLeakyBucketStorage(max_rate=3, time_period=1)\n\n# Create a new LeakyBucket instance\nasync_throttler = AsyncLeakyBucket(storage)\n\n@async_throttler.throttle()\nasync def async_fetch_data(api_url):\n async with httpx.AsyncClient() as client:\n response = await client.get(api_url)\n data = response.json()\n print(data)\n return data\n\nasync def main():\n # make multiple requests\n api_url = \"https://jsonplaceholder.typicode.com/posts/1\"\n tasks = [async_fetch_data(api_url) for _ in range(10)]\n results = await asyncio.gather(*tasks)\n print(results)\n\nasyncio.run(main())\n\n```\n\n\n\n#### Sqlite backend:\n\n```python\nimport time\nfrom leakybucket.bucket import LeakyBucket\nfrom leakybucket.persistence.sqlite import SqliteLeakyBucketStorage\n\n# Create a shared SQLite bucket\nbucket = LeakyBucket(\n SqliteLeakyBucketStorage(\n db_path=\"leakybucket.db\", \n max_rate=10, \n time_period=10\n )\n)\n\n# Decorate the function\n@bucket.throttle()\ndef make_request(index):\n print(f\"Making request {index}\")\n\ndef main():\n for i in range(35):\n make_request(i)\n\nmain()\n\n```\n\n\n\n\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) 2024 Benjamin Herro Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
"summary": "Leaky bucket implementation in python with different persistence options",
"version": "0.1.3",
"project_urls": null,
"split_keywords": [
"rate-limiter",
" asyncio",
" leaky-bucket",
" redis",
" sqlite3"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2003494f9430bf30064d39e356100c9595953e8b449394531bfe1e15f5a87ea6",
"md5": "6ff3bae63f84210b523435b5a2fb81e8",
"sha256": "1c591a0efb3566785ffe9fa7113edf9f917d37dbf843517fb48adeb1508b3a3d"
},
"downloads": -1,
"filename": "leaky_bucket_py-0.1.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "6ff3bae63f84210b523435b5a2fb81e8",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 13005,
"upload_time": "2025-01-01T23:59:23",
"upload_time_iso_8601": "2025-01-01T23:59:23.470681Z",
"url": "https://files.pythonhosted.org/packages/20/03/494f9430bf30064d39e356100c9595953e8b449394531bfe1e15f5a87ea6/leaky_bucket_py-0.1.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "854f9111cf73bb3be09314f7c3df889bfc1e51bb204a1f67a53c6c2ae061b5f3",
"md5": "9084374647e4c72342df8835f691fe1e",
"sha256": "ca88b95e506a8f70437d0694fb0a7f91d5f0f8c7354bce9b5ae2e309f89b508f"
},
"downloads": -1,
"filename": "leaky_bucket_py-0.1.3.tar.gz",
"has_sig": false,
"md5_digest": "9084374647e4c72342df8835f691fe1e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 11266,
"upload_time": "2025-01-01T23:59:25",
"upload_time_iso_8601": "2025-01-01T23:59:25.854476Z",
"url": "https://files.pythonhosted.org/packages/85/4f/9111cf73bb3be09314f7c3df889bfc1e51bb204a1f67a53c6c2ae061b5f3/leaky_bucket_py-0.1.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-01 23:59:25",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "leaky-bucket-py"
}
