# Django celery token bucket
A dynamic [token bucket](https://medium.com/analytics-vidhya/celery-throttling-setting-rate-limit-for-queues-5b5bf16c73ce) implementation using the database scheduler [django celery beat](https://github.com/celery/django-celery-beat).
## How it works
The bucket is represented by a celery queue that will not be processed by a worker but just hold our tokens (messages).
Whenever a rate limited task should be run, the decorator tries to consume a message from that queue. If the queue is empty, the task gets retried after the defined timeout.
A periodic task will then refill the bucket with tokens whenever they should be available again.
## Define a token bucket
Buckets are defined in the Django config.
Following example allows one thousand tokens per hour to throttle access to a rate limited third party API.
Add to `settings.py` of your project.
```python
from typing import Dict
from celery import schedules
from django_celery_token_bucket import TokenBucket
INSTALLED_APPS = [
...,
'django_celery_token_bucket'
]
CELERY_TOKEN_BUCKETS: Dict[str, TokenBucket] = {
"my_api_client": TokenBucket(
name="my_api_client",
schedule=schedules.crontab(minute=0), # once every hour
amount=1000,
maximum=1000,
)
}
```
### name
The name must only consist of letters, numbers and the underscore character as it's used in the name of the celery
queue. It should also be the same as the key in the CELERY_TOKEN_BUCKETS dictionary.
### schedule
A `celery.schedules.crontab` that defines when the tokens should be refilled.
### amount
The amount of tokens to add whenever the scheduled refill is run.
### maximum
The maximum amount of tokens our bucket can hold.
### token_refill_queue
Override this setting if you want token refill tasks for this bucket to be placed on a specific queue.
This field is by default None. If no value is provided, the `CELERY_DEFAULT_QUEUE` setting is used or `celery`
## Sync period tasks to refill the buckets
A management command `token_bucket_register_periodic_tasks` is provided that should be run during deployment of your
application to sync the period tasks and make sure that buckets get properly refilled.
## Use the rate_limit decorator
The decorator will make sure that the task that gets decorated will not exceed the limit of available tokens.
Decorated tasks must always be [bound](https://docs.celeryq.dev/en/latest/userguide/tasks.html#bound-tasks) to allow access to the task instance.
```python
from my_app.celery import celery_app
from django_celery_token_bucket.decorators import rate_limit
@celery_app.task(bind=True)
@rate_limit(token_bucket="my_api_client", countdown=300)
def my_tasK(self, *args, **kwargs):
return
```
### token_bucket
Name of the token bucket to consume from. Must be defined in the settings (see above) or will fail with an Exception.
### countdown
Time to wait in seconds before the next try when no token is available.
### affect_task_retries
Defaults to `False`
By default a failed token retrieval will not impact the retry count of your task. To change this behavior, set `affect_task_retries` to `True`.
```python
@celery_app.task(bind=True, max_retries=3, countdown=60)
@rate_limit(token_bucket="my_api_client", countdown=300, affect_task_retries=True)
def my_tasK(self, *args, **kwargs):
return
```
In this scenario, a failed token retrieval will increase the retry count of the task decorator.
If we cannot get a token on the first try, we will start over again with the 2nd try.
## Run the tests locally
A docker compose environment is provided to easily run the tests:
```bash
docker compose run --rm app test
```
## Making a new release
[bumpversion](https://github.com/peritus/bumpversion) is used to manage releases.
Add your changes to the [CHANGELOG](./CHANGELOG.md), run
```bash
bumpversion <major|minor|patch>
```
and push (including tags).
Raw data
{
"_id": null,
"home_page": "https://github.com/RegioHelden/django-celery-token-bucket",
"name": "django-celery-token-bucket",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "django,celery,token,bucket,rate limiting",
"author": "Jens Nistler <opensource@jensnistler.de>, Richard Ackon <richard.ackon@stroeer-online-marketing.de>",
"author_email": "opensource@regiohelden.de",
"download_url": "https://files.pythonhosted.org/packages/54/0d/932ee507c97b4c9a86f26232cd441ab1fb8ef1cc5ac1590028d167d0be64/django-celery-token-bucket-2.1.0.tar.gz",
"platform": null,
"description": "# Django celery token bucket\n\nA dynamic [token bucket](https://medium.com/analytics-vidhya/celery-throttling-setting-rate-limit-for-queues-5b5bf16c73ce) implementation using the database scheduler [django celery beat](https://github.com/celery/django-celery-beat).\n\n## How it works\n\nThe bucket is represented by a celery queue that will not be processed by a worker but just hold our tokens (messages).\nWhenever a rate limited task should be run, the decorator tries to consume a message from that queue. If the queue is empty, the task gets retried after the defined timeout. \nA periodic task will then refill the bucket with tokens whenever they should be available again.\n\n## Define a token bucket\n\nBuckets are defined in the Django config.\n\nFollowing example allows one thousand tokens per hour to throttle access to a rate limited third party API.\n\nAdd to `settings.py` of your project.\n\n```python\nfrom typing import Dict\nfrom celery import schedules\nfrom django_celery_token_bucket import TokenBucket\n\nINSTALLED_APPS = [\n ...,\n 'django_celery_token_bucket'\n]\n\nCELERY_TOKEN_BUCKETS: Dict[str, TokenBucket] = {\n \"my_api_client\": TokenBucket(\n name=\"my_api_client\",\n schedule=schedules.crontab(minute=0), # once every hour\n amount=1000,\n maximum=1000,\n )\n}\n```\n\n### name\n\nThe name must only consist of letters, numbers and the underscore character as it's used in the name of the celery\nqueue. It should also be the same as the key in the CELERY_TOKEN_BUCKETS dictionary.\n\n### schedule\n\nA `celery.schedules.crontab` that defines when the tokens should be refilled.\n\n### amount\n\nThe amount of tokens to add whenever the scheduled refill is run.\n\n### maximum\n\nThe maximum amount of tokens our bucket can hold.\n\n### token_refill_queue\n\nOverride this setting if you want token refill tasks for this bucket to be placed on a specific queue.\nThis field is by default None. If no value is provided, the `CELERY_DEFAULT_QUEUE` setting is used or `celery`\n\n\n## Sync period tasks to refill the buckets\n\nA management command `token_bucket_register_periodic_tasks` is provided that should be run during deployment of your\napplication to sync the period tasks and make sure that buckets get properly refilled.\n\n## Use the rate_limit decorator\n\nThe decorator will make sure that the task that gets decorated will not exceed the limit of available tokens. \nDecorated tasks must always be [bound](https://docs.celeryq.dev/en/latest/userguide/tasks.html#bound-tasks) to allow access to the task instance.\n\n```python\nfrom my_app.celery import celery_app\nfrom django_celery_token_bucket.decorators import rate_limit\n\n\n@celery_app.task(bind=True)\n@rate_limit(token_bucket=\"my_api_client\", countdown=300)\ndef my_tasK(self, *args, **kwargs):\n return\n```\n\n### token_bucket\n\nName of the token bucket to consume from. Must be defined in the settings (see above) or will fail with an Exception.\n\n### countdown\n\nTime to wait in seconds before the next try when no token is available.\n\n### affect_task_retries\n\nDefaults to `False` \nBy default a failed token retrieval will not impact the retry count of your task. To change this behavior, set `affect_task_retries` to `True`.\n\n```python\n@celery_app.task(bind=True, max_retries=3, countdown=60)\n@rate_limit(token_bucket=\"my_api_client\", countdown=300, affect_task_retries=True)\ndef my_tasK(self, *args, **kwargs):\n return\n```\n\nIn this scenario, a failed token retrieval will increase the retry count of the task decorator.\nIf we cannot get a token on the first try, we will start over again with the 2nd try.\n\n## Run the tests locally\n\nA docker compose environment is provided to easily run the tests:\n\n```bash\ndocker compose run --rm app test\n```\n\n## Making a new release\n\n[bumpversion](https://github.com/peritus/bumpversion) is used to manage releases.\n\nAdd your changes to the [CHANGELOG](./CHANGELOG.md), run\n\n```bash\nbumpversion <major|minor|patch>\n```\n\nand push (including tags).\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A token bucket implementation for celery rate limiting in Django",
"version": "2.1.0",
"split_keywords": [
"django",
"celery",
"token",
"bucket",
"rate limiting"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "0e8ce3a5cfe1e78750a871f0cf357b576d392864eb7d3fa720404237bc319c7a",
"md5": "cab8824a14428b876ae4a81d9e0ad778",
"sha256": "65f909edb565abd0fbec20ec75933dbe0c8e04528f44fc5203e646abe9ed3f38"
},
"downloads": -1,
"filename": "django_celery_token_bucket-2.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "cab8824a14428b876ae4a81d9e0ad778",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 11622,
"upload_time": "2023-01-03T16:59:55",
"upload_time_iso_8601": "2023-01-03T16:59:55.489284Z",
"url": "https://files.pythonhosted.org/packages/0e/8c/e3a5cfe1e78750a871f0cf357b576d392864eb7d3fa720404237bc319c7a/django_celery_token_bucket-2.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "540d932ee507c97b4c9a86f26232cd441ab1fb8ef1cc5ac1590028d167d0be64",
"md5": "dadca719cb1af09b84dbe5a85365c5b6",
"sha256": "21a9471e64b808b591e48376b614053e8312c1f8e63a65f33d526b66123adf73"
},
"downloads": -1,
"filename": "django-celery-token-bucket-2.1.0.tar.gz",
"has_sig": false,
"md5_digest": "dadca719cb1af09b84dbe5a85365c5b6",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 10665,
"upload_time": "2023-01-03T16:59:56",
"upload_time_iso_8601": "2023-01-03T16:59:56.608987Z",
"url": "https://files.pythonhosted.org/packages/54/0d/932ee507c97b4c9a86f26232cd441ab1fb8ef1cc5ac1590028d167d0be64/django-celery-token-bucket-2.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-01-03 16:59:56",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "RegioHelden",
"github_project": "django-celery-token-bucket",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "django-celery-token-bucket"
}
JSON
