# Requests-Ratelimiter
[](https://github.com/JWCook/requests-ratelimiter/actions)
[](https://codecov.io/gh/JWCook/requests-ratelimiter)
[](https://requests-ratelimiter.readthedocs.io)
[](https://pypi.org/project/requests-ratelimiter)
[](https://anaconda.org/conda-forge/requests-ratelimiter)
[](https://pypi.org/project/requests-ratelimiter)
[](https://pypi.org/project/requests-ratelimiter)
This package is a simple wrapper around [pyrate-limiter v2](https://github.com/vutran1710/PyrateLimiter/tree/v2.10.0)
that adds convenient integration with the [requests](https://requests.readthedocs.io) library.
Full project documentation can be found at [requests-ratelimiter.readthedocs.io](https://requests-ratelimiter.readthedocs.io).
# Features
* `pyrate-limiter` is a general-purpose rate-limiting library that implements the leaky bucket
algorithm, supports multiple rate limits, and has optional persistence with SQLite and Redis
backends
* `requests-ratelimiter` adds some conveniences for sending rate-limited HTTP requests with the
`requests` library
* It can be used as either a
[session](https://requests.readthedocs.io/en/latest/user/advanced/#session-objects) or a
[transport adapter](https://requests.readthedocs.io/en/latest/user/advanced/#transport-adapters)
* It can also be used as a mixin, for compatibility with other `requests`-based libraries
* Rate limits are tracked separately per host
* Different rate limits can optionally be applied to different hosts
# Installation
```
pip install requests-ratelimiter
```
# Usage
## Usage Options
There are three ways to use `requests-ratelimiter`:
### Session
The simplest option is
[`LimiterSession`](https://requests-ratelimiter.readthedocs.io/en/stable/reference.html#requests_ratelimiter.LimiterSession),
which can be used as a drop-in replacement for
[`requests.Session`](https://requests.readthedocs.io/en/latest/api/#requests.Session).
Note: By default, each session will perform rate limiting independently. If you are using a multi-threaded environment
or multiple processes, you should use a persistent backend like SQLite or Redis which can persist the rate limit across
threads, processes, and/or application restarts. When using `requests-ratelimiter` as part of a web application, it is
recommended to use a persistent backend to ensure that the rate limit is shared across all requests.
Example:
```python
from requests_ratelimiter import LimiterSession
from time import time
# Apply a rate limit of 5 requests per second to all requests
session = LimiterSession(per_second=5)
start = time()
# Send requests that stay within the defined rate limit
for i in range(20):
response = session.get('https://httpbin.org/get')
print(f'[t+{time()-start:.2f}] Sent request {i+1}')
```
Example output:
```bash
[t+0.22] Sent request 1
[t+0.26] Sent request 2
[t+0.30] Sent request 3
[t+0.34] Sent request 4
[t+0.39] Sent request 5
[t+1.24] Sent request 6
[t+1.28] Sent request 7
[t+1.32] Sent request 8
[t+1.37] Sent request 9
[t+1.41] Sent request 10
[t+2.04] Sent request 11
...
```
### Adapter
For more advanced usage,
[`LimiterAdapter`](https://requests-ratelimiter.readthedocs.io/en/stable/reference.html#requests_ratelimiter.LimiterAdapter)
is available to be used as a
[transport adapter](https://requests.readthedocs.io/en/latest/user/advanced/#transport-adapters).
Example:
```python
from requests import Session
from requests_ratelimiter import LimiterAdapter
session = Session()
# Apply a rate-limit (5 requests per second) to all requests
adapter = LimiterAdapter(per_second=5)
session.mount('http://', adapter)
session.mount('https://', adapter)
# Send rate-limited requests
for user_id in range(100):
response = session.get(f'https://api.some_site.com/v1/users/{user_id}')
print(response.json())
```
### Mixin
Finally,
[`LimiterMixin`](https://requests-ratelimiter.readthedocs.io/en/stable/reference.html#requests_ratelimiter.LimiterMixin)
is available for advanced use cases in which you want add rate-limiting features to a custom session
or adapter class. See
[Custom Session Example](#custom-session-example-requests-cache) below for an example.
## Rate Limit Settings
### Basic Settings
The following parameters are available for the most common rate limit intervals:
* `per_second`: Max requests per second
* `per_minute`: Max requests per minute
* `per_hour`: Max requests per hour
* `per_day`: Max requests per day
* `per_month`: Max requests per month
* `burst`: Max number of consecutive requests allowed before applying per-second rate-limiting
<!-- TODO: Section explaining burst rate limit -->
### Advanced Settings
If you need to define more complex rate limits, you can create a `Limiter` object instead:
```python
from pyrate_limiter import Duration, RequestRate, Limiter
from requests_ratelimiter import LimiterSession
nanocentury_rate = RequestRate(10, Duration.SECOND * 3.156)
fortnight_rate = RequestRate(1000, Duration.DAY * 14)
trimonthly_rate = RequestRate(10000, Duration.MONTH * 3)
limiter = Limiter(nanocentury_rate, fortnight_rate, trimonthly_rate)
session = LimiterSession(limiter=limiter)
```
See [pyrate-limiter docs](https://pyratelimiter.readthedocs.io/en/latest/#basic-usage) for more `Limiter` usage details.
## Backends
By default, rate limits are tracked in memory and are not persistent. You can optionally use either
SQLite or Redis to persist rate limits across threads, processes, and/or application restarts.
You can specify which backend to use with the `bucket_class` argument. For example, to use SQLite:
```python
from pyrate_limiter import SQLiteBucket
from requests_ratelimiter import LimiterSession
session = LimiterSession(per_second=5, bucket_class=SQLiteBucket)
```
See [pyrate-limiter docs](https://pyratelimiter.readthedocs.io/en/latest/#backends) for more details.
## Other Features
### Per-Host Rate Limit Tracking
With either `LimiterSession` or `LimiterAdapter`, rate limits are tracked separately for each host.
In other words, requests sent to one host will not count against the rate limit for any other hosts:
```python
session = LimiterSession(per_second=5)
# Make requests for two different hosts
for _ in range(10):
response = session.get(f'https://httpbin.org/get')
print(response.json())
session.get(f'https://httpbingo.org/get')
print(response.json())
```
If you have a case where multiple hosts share the same rate limit, you can disable this behavior
with the `per_host` option:
```python
session = LimiterSession(per_second=5, per_host=False)
```
### Per-Host Rate Limit Definitions
With `LimiterAdapter`, you can apply different rate limits to different hosts or URLs:
```python
# Apply a different set of rate limits (2/second and 100/minute) to a specific host
adapter_2 = LimiterAdapter(per_second=2, per_minute=100)
session.mount('https://api.some_site.com', adapter_2)
```
Behavior for matching requests is the same as other transport adapters: `requests` will use the
adapter with the most specific (i.e., longest) URL prefix that matches a given request. For example:
```python
session.mount('https://api.some_site.com/v1', adapter_3)
session.mount('https://api.some_site.com/v1/users', adapter_4)
# This request will use adapter_3
session.get('https://api.some_site.com/v1/')
# This request will use adapter_4
session.get('https://api.some_site.com/v1/users/1234')
```
### Custom Tracking
For advanced use cases, you can define your own custom tracking behavior with the `bucket` option.
For example, an API that enforces rate limits based on a tenant ID, this feature can be used to track
rate limits per tenant. If `bucket` is specified, host tracking is disabled.
Note: It is advisable to use SQLite or Redis backends when using custom tracking because using the default backend
each session will track rate limits independently, even if both sessions call the same URL.
```python
sessionA = LimiterSession(per_second=5, bucket='tenant1')
sessionB = LimiterSession(per_second=5, bucket='tenant2')
```
### Rate Limit Error Handling
Sometimes, server-side rate limiting may not behave exactly as documented (or may not be documented
at all). Or you might encounter other scenarios where your client-side limit gets out of sync with
the server-side limit. Typically, a server will send a `429: Too Many Requests` response for an
exceeded rate limit.
When this happens, `requests-ratelimiter` will adjust its request log in an attempt to catch up to
the server-side limit. If a server sends a different status code other than 429 to indicate an
exceeded limit, you can set this with `limit_statuses`:
```python
session = LimiterSession(per_second=5, limit_statuses=[429, 500])
```
Or if you would prefer to disable this behavior and handle it yourself:
```python
session = LimiterSession(per_second=5, limit_statuses=[])
```
# Compatibility
There are many other useful libraries out there that add features to `requests`, most commonly by
extending or modifying
[requests.Session](https://requests.readthedocs.io/en/latest/api/#requests.Session) or
[requests.HTTPAdapter](https://requests.readthedocs.io/en/latest/api/#requests.adapters.HTTPAdapter).
To use `requests-ratelimiter` with one of these libraries, you have a few different options:
1. If the library provides a custom `Session` class, mount a `LimiterAdapter` on it
2. Or use `LimiterMixin` to create a custom `Session` class with features from both libraries
3. If the library provides a custom `Adapter` class, use `LimiterMixin` to create a custom `Adapter`
class with features from both libraries
## Custom Session Example: Requests-Cache
For example, to combine with [requests-cache](https://github.com/requests-cache/requests-cache), which
also includes a separate mixin class:
```python
from requests import Session
from requests_cache import CacheMixin
from requests_ratelimiter import LimiterMixin, SQLiteBucket
class CachedLimiterSession(CacheMixin, LimiterMixin, Session):
"""
Session class with caching and rate-limiting behavior. Accepts arguments for both
LimiterSession and CachedSession.
"""
# Optionally use SQLite as both the bucket backend and the cache backend
session = CachedLimiterSession(
per_second=5,
cache_name='cache.db',
bucket_class=SQLiteBucket,
bucket_kwargs={
"path": "cache.db",
'isolation_level': "EXCLUSIVE",
'check_same_thread': False,
},
)
```
This example has an extra benefit: cache hits won't count against your rate limit!
Raw data
{
"_id": null,
"home_page": "https://github.com/JWCook/requests-ratelimiter",
"name": "requests-ratelimiter",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.7",
"maintainer_email": null,
"keywords": "requests, rate-limiting, leaky-bucket",
"author": "Jordan Cook",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/0a/9b/06ef6289f3b774593354286b2d032ca7df30e28a5aeb9423c97bcd13c676/requests_ratelimiter-0.7.0.tar.gz",
"platform": null,
"description": "# Requests-Ratelimiter\n[](https://github.com/JWCook/requests-ratelimiter/actions)\n[](https://codecov.io/gh/JWCook/requests-ratelimiter)\n[](https://requests-ratelimiter.readthedocs.io)\n[](https://pypi.org/project/requests-ratelimiter)\n[](https://anaconda.org/conda-forge/requests-ratelimiter)\n[](https://pypi.org/project/requests-ratelimiter)\n[](https://pypi.org/project/requests-ratelimiter)\n\nThis package is a simple wrapper around [pyrate-limiter v2](https://github.com/vutran1710/PyrateLimiter/tree/v2.10.0)\nthat adds convenient integration with the [requests](https://requests.readthedocs.io) library.\n\nFull project documentation can be found at [requests-ratelimiter.readthedocs.io](https://requests-ratelimiter.readthedocs.io).\n\n\n# Features\n* `pyrate-limiter` is a general-purpose rate-limiting library that implements the leaky bucket\n algorithm, supports multiple rate limits, and has optional persistence with SQLite and Redis\n backends\n* `requests-ratelimiter` adds some conveniences for sending rate-limited HTTP requests with the\n `requests` library\n* It can be used as either a\n [session](https://requests.readthedocs.io/en/latest/user/advanced/#session-objects) or a\n [transport adapter](https://requests.readthedocs.io/en/latest/user/advanced/#transport-adapters)\n* It can also be used as a mixin, for compatibility with other `requests`-based libraries\n* Rate limits are tracked separately per host\n* Different rate limits can optionally be applied to different hosts\n\n# Installation\n```\npip install requests-ratelimiter\n```\n\n# Usage\n\n## Usage Options\nThere are three ways to use `requests-ratelimiter`:\n\n### Session\nThe simplest option is\n[`LimiterSession`](https://requests-ratelimiter.readthedocs.io/en/stable/reference.html#requests_ratelimiter.LimiterSession),\nwhich can be used as a drop-in replacement for\n[`requests.Session`](https://requests.readthedocs.io/en/latest/api/#requests.Session).\n\nNote: By default, each session will perform rate limiting independently. If you are using a multi-threaded environment\nor multiple processes, you should use a persistent backend like SQLite or Redis which can persist the rate limit across\nthreads, processes, and/or application restarts. When using `requests-ratelimiter` as part of a web application, it is\nrecommended to use a persistent backend to ensure that the rate limit is shared across all requests.\n\nExample:\n```python\nfrom requests_ratelimiter import LimiterSession\nfrom time import time\n\n# Apply a rate limit of 5 requests per second to all requests\nsession = LimiterSession(per_second=5)\nstart = time()\n\n# Send requests that stay within the defined rate limit\nfor i in range(20):\n response = session.get('https://httpbin.org/get')\n print(f'[t+{time()-start:.2f}] Sent request {i+1}')\n```\n\nExample output:\n```bash\n[t+0.22] Sent request 1\n[t+0.26] Sent request 2\n[t+0.30] Sent request 3\n[t+0.34] Sent request 4\n[t+0.39] Sent request 5\n[t+1.24] Sent request 6\n[t+1.28] Sent request 7\n[t+1.32] Sent request 8\n[t+1.37] Sent request 9\n[t+1.41] Sent request 10\n[t+2.04] Sent request 11\n...\n```\n\n### Adapter\nFor more advanced usage,\n[`LimiterAdapter`](https://requests-ratelimiter.readthedocs.io/en/stable/reference.html#requests_ratelimiter.LimiterAdapter)\nis available to be used as a\n[transport adapter](https://requests.readthedocs.io/en/latest/user/advanced/#transport-adapters).\n\nExample:\n```python\nfrom requests import Session\nfrom requests_ratelimiter import LimiterAdapter\n\nsession = Session()\n\n# Apply a rate-limit (5 requests per second) to all requests\nadapter = LimiterAdapter(per_second=5)\nsession.mount('http://', adapter)\nsession.mount('https://', adapter)\n\n# Send rate-limited requests\nfor user_id in range(100):\n response = session.get(f'https://api.some_site.com/v1/users/{user_id}')\n print(response.json())\n```\n\n### Mixin\nFinally,\n[`LimiterMixin`](https://requests-ratelimiter.readthedocs.io/en/stable/reference.html#requests_ratelimiter.LimiterMixin)\nis available for advanced use cases in which you want add rate-limiting features to a custom session\nor adapter class. See\n[Custom Session Example](#custom-session-example-requests-cache) below for an example.\n\n## Rate Limit Settings\n### Basic Settings\nThe following parameters are available for the most common rate limit intervals:\n* `per_second`: Max requests per second\n* `per_minute`: Max requests per minute\n* `per_hour`: Max requests per hour\n* `per_day`: Max requests per day\n* `per_month`: Max requests per month\n* `burst`: Max number of consecutive requests allowed before applying per-second rate-limiting\n\n<!-- TODO: Section explaining burst rate limit -->\n\n### Advanced Settings\nIf you need to define more complex rate limits, you can create a `Limiter` object instead:\n```python\nfrom pyrate_limiter import Duration, RequestRate, Limiter\nfrom requests_ratelimiter import LimiterSession\n\nnanocentury_rate = RequestRate(10, Duration.SECOND * 3.156)\nfortnight_rate = RequestRate(1000, Duration.DAY * 14)\ntrimonthly_rate = RequestRate(10000, Duration.MONTH * 3)\nlimiter = Limiter(nanocentury_rate, fortnight_rate, trimonthly_rate)\n\nsession = LimiterSession(limiter=limiter)\n```\n\nSee [pyrate-limiter docs](https://pyratelimiter.readthedocs.io/en/latest/#basic-usage) for more `Limiter` usage details.\n\n## Backends\nBy default, rate limits are tracked in memory and are not persistent. You can optionally use either\nSQLite or Redis to persist rate limits across threads, processes, and/or application restarts.\nYou can specify which backend to use with the `bucket_class` argument. For example, to use SQLite:\n```python\nfrom pyrate_limiter import SQLiteBucket\nfrom requests_ratelimiter import LimiterSession\n\nsession = LimiterSession(per_second=5, bucket_class=SQLiteBucket)\n```\n\nSee [pyrate-limiter docs](https://pyratelimiter.readthedocs.io/en/latest/#backends) for more details.\n\n## Other Features\n### Per-Host Rate Limit Tracking\nWith either `LimiterSession` or `LimiterAdapter`, rate limits are tracked separately for each host.\nIn other words, requests sent to one host will not count against the rate limit for any other hosts:\n\n```python\nsession = LimiterSession(per_second=5)\n\n# Make requests for two different hosts\nfor _ in range(10):\n response = session.get(f'https://httpbin.org/get')\n print(response.json())\n session.get(f'https://httpbingo.org/get')\n print(response.json())\n```\n\nIf you have a case where multiple hosts share the same rate limit, you can disable this behavior\nwith the `per_host` option:\n```python\nsession = LimiterSession(per_second=5, per_host=False)\n```\n\n### Per-Host Rate Limit Definitions\nWith `LimiterAdapter`, you can apply different rate limits to different hosts or URLs:\n```python\n# Apply a different set of rate limits (2/second and 100/minute) to a specific host\nadapter_2 = LimiterAdapter(per_second=2, per_minute=100)\nsession.mount('https://api.some_site.com', adapter_2)\n```\n\nBehavior for matching requests is the same as other transport adapters: `requests` will use the\nadapter with the most specific (i.e., longest) URL prefix that matches a given request. For example:\n```python\nsession.mount('https://api.some_site.com/v1', adapter_3)\nsession.mount('https://api.some_site.com/v1/users', adapter_4)\n\n# This request will use adapter_3\nsession.get('https://api.some_site.com/v1/')\n\n# This request will use adapter_4\nsession.get('https://api.some_site.com/v1/users/1234')\n```\n\n### Custom Tracking\nFor advanced use cases, you can define your own custom tracking behavior with the `bucket` option.\nFor example, an API that enforces rate limits based on a tenant ID, this feature can be used to track\nrate limits per tenant. If `bucket` is specified, host tracking is disabled.\n\nNote: It is advisable to use SQLite or Redis backends when using custom tracking because using the default backend\neach session will track rate limits independently, even if both sessions call the same URL.\n```python\nsessionA = LimiterSession(per_second=5, bucket='tenant1')\nsessionB = LimiterSession(per_second=5, bucket='tenant2')\n```\n\n### Rate Limit Error Handling\nSometimes, server-side rate limiting may not behave exactly as documented (or may not be documented\nat all). Or you might encounter other scenarios where your client-side limit gets out of sync with\nthe server-side limit. Typically, a server will send a `429: Too Many Requests` response for an\nexceeded rate limit.\n\nWhen this happens, `requests-ratelimiter` will adjust its request log in an attempt to catch up to\nthe server-side limit. If a server sends a different status code other than 429 to indicate an\nexceeded limit, you can set this with `limit_statuses`:\n```python\nsession = LimiterSession(per_second=5, limit_statuses=[429, 500])\n```\n\nOr if you would prefer to disable this behavior and handle it yourself:\n```python\nsession = LimiterSession(per_second=5, limit_statuses=[])\n```\n\n# Compatibility\nThere are many other useful libraries out there that add features to `requests`, most commonly by\nextending or modifying\n[requests.Session](https://requests.readthedocs.io/en/latest/api/#requests.Session) or\n[requests.HTTPAdapter](https://requests.readthedocs.io/en/latest/api/#requests.adapters.HTTPAdapter).\n\nTo use `requests-ratelimiter` with one of these libraries, you have a few different options:\n1. If the library provides a custom `Session` class, mount a `LimiterAdapter` on it\n2. Or use `LimiterMixin` to create a custom `Session` class with features from both libraries\n3. If the library provides a custom `Adapter` class, use `LimiterMixin` to create a custom `Adapter`\n class with features from both libraries\n\n## Custom Session Example: Requests-Cache\nFor example, to combine with [requests-cache](https://github.com/requests-cache/requests-cache), which\nalso includes a separate mixin class:\n```python\nfrom requests import Session\nfrom requests_cache import CacheMixin\nfrom requests_ratelimiter import LimiterMixin, SQLiteBucket\n\n\nclass CachedLimiterSession(CacheMixin, LimiterMixin, Session):\n \"\"\"\n Session class with caching and rate-limiting behavior. Accepts arguments for both\n LimiterSession and CachedSession.\n \"\"\"\n\n\n# Optionally use SQLite as both the bucket backend and the cache backend\nsession = CachedLimiterSession(\n per_second=5,\n cache_name='cache.db',\n bucket_class=SQLiteBucket,\n bucket_kwargs={\n \"path\": \"cache.db\",\n 'isolation_level': \"EXCLUSIVE\",\n 'check_same_thread': False,\n },\n)\n```\n\nThis example has an extra benefit: cache hits won't count against your rate limit!\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Rate-limiting for the requests library",
"version": "0.7.0",
"project_urls": {
"Documentation": "https://requests-ratelimiter.readthedocs.io",
"Homepage": "https://github.com/JWCook/requests-ratelimiter",
"Repository": "https://github.com/JWCook/requests-ratelimiter"
},
"split_keywords": [
"requests",
" rate-limiting",
" leaky-bucket"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ae812d9e20bfc065fe083494e656c1c396a33303d10d1ccfc11efe886ba51458",
"md5": "14ef647552553e8308fd65c9d32cfb15",
"sha256": "1a7ef2faaa790272722db8539728690046237766fcc479f85b9591e5356a8185"
},
"downloads": -1,
"filename": "requests_ratelimiter-0.7.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "14ef647552553e8308fd65c9d32cfb15",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.7",
"size": 9314,
"upload_time": "2024-07-02T21:58:07",
"upload_time_iso_8601": "2024-07-02T21:58:07.613082Z",
"url": "https://files.pythonhosted.org/packages/ae/81/2d9e20bfc065fe083494e656c1c396a33303d10d1ccfc11efe886ba51458/requests_ratelimiter-0.7.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "0a9b06ef6289f3b774593354286b2d032ca7df30e28a5aeb9423c97bcd13c676",
"md5": "c105b9547cecf78d3cba55c6faef3897",
"sha256": "a070c8a359a6f3a001b0ccb08f17228b7ae0a6e21d8df5b6f6bd58389cddde45"
},
"downloads": -1,
"filename": "requests_ratelimiter-0.7.0.tar.gz",
"has_sig": false,
"md5_digest": "c105b9547cecf78d3cba55c6faef3897",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.7",
"size": 14200,
"upload_time": "2024-07-02T21:58:09",
"upload_time_iso_8601": "2024-07-02T21:58:09.085200Z",
"url": "https://files.pythonhosted.org/packages/0a/9b/06ef6289f3b774593354286b2d032ca7df30e28a5aeb9423c97bcd13c676/requests_ratelimiter-0.7.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-07-02 21:58:09",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "JWCook",
"github_project": "requests-ratelimiter",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "requests-ratelimiter"
}
JSON
