# log-rate-limit - limit excessive log output
[](https://pypi.org/project/log-rate-limit/)
[](https://github.com/samuller/log-rate-limit/actions/workflows/tests.yml)
[](https://github.com/samuller/pgmerge/actions)
[](http://mypy-lang.org/)
[](https://black.readthedocs.io/en/stable/)
[](https://flake8.pycqa.org/en/6.1.0/user/options.html#cmdoption-flake8-max-complexity)
A [logging filter](https://docs.python.org/3/library/logging.html#filter-objects) using Python's standard logging mechanisms to rate-limit logs - i.e. suppress logs when they are being output too fast.
Log commands are grouped into separate **streams** that will each have their own rate limitation applied without affecting the logs in other streams. By default every log message is assigned a unique stream so that only repeated log messages will be suppressed.
However, logs can also be assigned streams manually to achieve various outcomes:
- A log can be assigned to an undefined/`None` stream so that rate-limiting doesn't apply to it.
- Logs in different parts of the code can be grouped into the same stream so that they share a rate-limit, e.g. when they all trigger due to the same issue and only some are needed to indicate it.
The default can also be changed so that rate-limiting is disabled by default and only applies to logs for which a `stream_id` has been manually set.
## Quick usage
Import the filter:
```python
from log_rate_limit import StreamRateLimitFilter, RateLimit
```
Use the filter with your `logger` - with default parameters it will rate-limit all repetitive log messages:
```python
logger.addFilter(StreamRateLimitFilter(period_sec=30))
```
All logs on `logger` will now be rate-limited, but this can be disabled per-log by setting the `stream_id` to `None`:
```python
logger.warning("Something went wrong!", extra=RateLimit(stream_id=None))
```
If you have a log message that's continually changing (e.g. contains a timestamp), you can manually define how those messages are defined as similar enough to be rate-limited together by setting a custom `stream_id`:
```python
logger.warning("Something went wrong with %s at %s!", device_id, timestamp, extra=RateLimit(stream_id=f"went_wrong_{device_id}"))
```
Note that if the timestamps are added by a `logging.Formatter` then they won't affect our rate-limiting filter as these filters run before formatting is applied (see [full workflow](https://docs.python.org/3/howto/logging.html#logging-flow)).
## Usage examples
### Rate-limiting by default
Example of rate-limiting with default options where each log message is assigned to its own stream:
```python
import time
import logging
from log_rate_limit import StreamRateLimitFilter, RateLimit
# Setup logging
logging.basicConfig()
logger = logging.getLogger(__name__)
# Add our filter
logger.addFilter(StreamRateLimitFilter(period_sec=1))
# Log many warnings
for _ in range(100):
logger.warning("Wolf!")
for i in range(100):
logger.warning("No really, a wolf!")
if i == 98:
time.sleep(1)
# Prevent rate-limited by setting/overriding the stream to be undefined (None)
for _ in range(3):
logger.warning("Sheep!", extra=RateLimit(stream_id=None))
```
Which only outputs the following:
```log
WARNING:__main__:Wolf!
WARNING:__main__:No really, a wolf!
WARNING:__main__:No really, a wolf!
+ skipped 98 logs due to rate-limiting
WARNING:__main__:Sheep!
WARNING:__main__:Sheep!
WARNING:__main__:Sheep!
```
Note that (unless overridden) logs were only repeated after the `sleep()` call, and the repeated log also included an extra summary message added afterwards.
When we override rate-limiting above, you'll see our filter reads dynamic configs from logging's `extra` parameter.
> Be very careful not to forget the `extra=` name part of the argument, as then the logging framework will assume you're passing arguments meant for formatting in the logging message and your options will silently be ignored!
### Rate-limit only when specified
If you want most of your logs to be unaffected and you only have some you want to specifically rate-limit, then you can do the following:
```python
import logging
from log_rate_limit import StreamRateLimitFilter, RateLimit
# Setup logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
# Add our filter, but don't assign unique streams to logs by default
logger.addFilter(StreamRateLimitFilter(period_sec=1, default_stream_id=None))
# Normal logs are now not rate-limited
for i in range(3):
logger.info(f"Status update: {i}")
# Only those we manually assign a stream will be rate-limited
for _ in range(3):
logger.warning("Issue!", extra=RateLimit(stream_id="issue"))
```
Which only outputs the following:
```log
INFO:__main__:Status update: 0
INFO:__main__:Status update: 1
INFO:__main__:Status update: 2
WARNING:__main__:Issue!
```
### Dynamically override configuration options
Some options set during creation of the initial filter can be overridden for individual log calls. This is done by adding the `extra` parameter to any specific log call, e.g.:
```python
# Override the rate-limit period for this specific log call
logger.warning("Test1", extra=RateLimit(stream_id="stream1", period_sec=30))
# Override the allow_next_n value for a set of logs in the same stream so that this group of logs don't restrict one
# another from occuring consecutively
logger.warning("Test", extra=RateLimit(stream_id="stream2", allow_next_n=2))
logger.info("Extra", extra=RateLimit(stream_id="stream2"))
logger.debug("Info", extra=RateLimit(stream_id="stream2"))
```
If you want to set custom options for a large group of log calls without repeatedly adding the `extra` parameter, it's possible to use a [LoggerAdapter](https://docs.python.org/3/library/logging.html#loggeradapter-objects):
```python
import logging
from log_rate_limit import StreamRateLimitFilter, RateLimit
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
# Add our filter
logger.addFilter(StreamRateLimitFilter(period_sec=1))
# Use LoggerAdapter to assign additional "extra" parameters to all calls using this logger
global_extra = RateLimit(stream_id="custom_stream", period_sec=20)
logger = logging.LoggerAdapter(logger, global_extra)
# Log many warnings
for _ in range(100):
logger.warning("Wolf!")
for i in range(100):
logger.warning("No really, a wolf!")
```
Which merely outputs:
```log
WARNING:__main__:Wolf!
```
Since both log calls are in the same stream.
Alternatively (to a LoggerAdapter), custom options can also be added by writing your own [logging.Filter](https://docs.python.org/3.8/howto/logging-cookbook.html#using-filters-to-impart-contextual-information).
### Dynamic stream ID
Dynamic stream IDs can be assigned based on any criteria you want, e.g.:
```python
logger.warning(f"Error occured on device {device_id}!", extra=RateLimit(stream_id=f"error_on_{device_id}"))
```
## Installation
### Install from PyPI
With `Python 3` installed on your system, you can run:
pip install log-rate-limit
To test that installation worked, you can run:
python -c "import log_rate_limit"
and you can uninstall at any time with:
pip uninstall log-rate-limit
To install with poetry:
poetry add log-rate-limit
### Install from Github
To install the newest code directly from Github:
pip install git+https://github.com/samuller/log-rate-limit
Raw data
{
"_id": null,
"home_page": "https://github.com/samuller/log-rate-limit",
"name": "log-rate-limit",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8.1,<4.0.0",
"maintainer_email": "",
"keywords": "logging,log,limit,rate",
"author": "Simon Muller",
"author_email": "samullers@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/91/73/9c015029fc6759ecc9196f756224302a13d2bbc18a12f48cedabfb13ca8f/log_rate_limit-1.4.1.tar.gz",
"platform": null,
"description": "# log-rate-limit - limit excessive log output\n\n[](https://pypi.org/project/log-rate-limit/)\n[](https://github.com/samuller/log-rate-limit/actions/workflows/tests.yml)\n[](https://github.com/samuller/pgmerge/actions)\n[](http://mypy-lang.org/)\n[](https://black.readthedocs.io/en/stable/)\n[](https://flake8.pycqa.org/en/6.1.0/user/options.html#cmdoption-flake8-max-complexity)\n\nA [logging filter](https://docs.python.org/3/library/logging.html#filter-objects) using Python's standard logging mechanisms to rate-limit logs - i.e. suppress logs when they are being output too fast.\n\nLog commands are grouped into separate **streams** that will each have their own rate limitation applied without affecting the logs in other streams. By default every log message is assigned a unique stream so that only repeated log messages will be suppressed.\n\nHowever, logs can also be assigned streams manually to achieve various outcomes:\n- A log can be assigned to an undefined/`None` stream so that rate-limiting doesn't apply to it.\n- Logs in different parts of the code can be grouped into the same stream so that they share a rate-limit, e.g. when they all trigger due to the same issue and only some are needed to indicate it.\n\nThe default can also be changed so that rate-limiting is disabled by default and only applies to logs for which a `stream_id` has been manually set.\n\n## Quick usage\n\nImport the filter:\n\n```python\nfrom log_rate_limit import StreamRateLimitFilter, RateLimit\n```\n\nUse the filter with your `logger` - with default parameters it will rate-limit all repetitive log messages:\n\n```python\nlogger.addFilter(StreamRateLimitFilter(period_sec=30))\n```\n\nAll logs on `logger` will now be rate-limited, but this can be disabled per-log by setting the `stream_id` to `None`:\n\n```python\nlogger.warning(\"Something went wrong!\", extra=RateLimit(stream_id=None))\n```\n\nIf you have a log message that's continually changing (e.g. contains a timestamp), you can manually define how those messages are defined as similar enough to be rate-limited together by setting a custom `stream_id`:\n\n```python\nlogger.warning(\"Something went wrong with %s at %s!\", device_id, timestamp, extra=RateLimit(stream_id=f\"went_wrong_{device_id}\"))\n```\n\nNote that if the timestamps are added by a `logging.Formatter` then they won't affect our rate-limiting filter as these filters run before formatting is applied (see [full workflow](https://docs.python.org/3/howto/logging.html#logging-flow)).\n\n## Usage examples\n\n### Rate-limiting by default\n\nExample of rate-limiting with default options where each log message is assigned to its own stream:\n```python\nimport time\nimport logging\nfrom log_rate_limit import StreamRateLimitFilter, RateLimit\n# Setup logging\nlogging.basicConfig()\nlogger = logging.getLogger(__name__)\n\n# Add our filter\nlogger.addFilter(StreamRateLimitFilter(period_sec=1))\n# Log many warnings\nfor _ in range(100):\n logger.warning(\"Wolf!\")\nfor i in range(100):\n logger.warning(\"No really, a wolf!\")\n if i == 98:\n time.sleep(1)\n# Prevent rate-limited by setting/overriding the stream to be undefined (None)\nfor _ in range(3):\n logger.warning(\"Sheep!\", extra=RateLimit(stream_id=None))\n``` \nWhich only outputs the following:\n```log\nWARNING:__main__:Wolf!\nWARNING:__main__:No really, a wolf!\nWARNING:__main__:No really, a wolf!\n+ skipped 98 logs due to rate-limiting\nWARNING:__main__:Sheep!\nWARNING:__main__:Sheep!\nWARNING:__main__:Sheep!\n```\nNote that (unless overridden) logs were only repeated after the `sleep()` call, and the repeated log also included an extra summary message added afterwards.\n\nWhen we override rate-limiting above, you'll see our filter reads dynamic configs from logging's `extra` parameter.\n\n> Be very careful not to forget the `extra=` name part of the argument, as then the logging framework will assume you're passing arguments meant for formatting in the logging message and your options will silently be ignored!\n\n### Rate-limit only when specified\n\nIf you want most of your logs to be unaffected and you only have some you want to specifically rate-limit, then you can do the following:\n```python\nimport logging\nfrom log_rate_limit import StreamRateLimitFilter, RateLimit\n# Setup logging\nlogging.basicConfig(level=logging.INFO)\nlogger = logging.getLogger(__name__)\n\n# Add our filter, but don't assign unique streams to logs by default\nlogger.addFilter(StreamRateLimitFilter(period_sec=1, default_stream_id=None))\n# Normal logs are now not rate-limited\nfor i in range(3):\n logger.info(f\"Status update: {i}\")\n# Only those we manually assign a stream will be rate-limited\nfor _ in range(3):\n logger.warning(\"Issue!\", extra=RateLimit(stream_id=\"issue\"))\n```\nWhich only outputs the following:\n```log\nINFO:__main__:Status update: 0\nINFO:__main__:Status update: 1\nINFO:__main__:Status update: 2\nWARNING:__main__:Issue!\n```\n\n### Dynamically override configuration options\n\nSome options set during creation of the initial filter can be overridden for individual log calls. This is done by adding the `extra` parameter to any specific log call, e.g.:\n```python\n# Override the rate-limit period for this specific log call\nlogger.warning(\"Test1\", extra=RateLimit(stream_id=\"stream1\", period_sec=30))\n# Override the allow_next_n value for a set of logs in the same stream so that this group of logs don't restrict one\n# another from occuring consecutively\nlogger.warning(\"Test\", extra=RateLimit(stream_id=\"stream2\", allow_next_n=2))\nlogger.info(\"Extra\", extra=RateLimit(stream_id=\"stream2\"))\nlogger.debug(\"Info\", extra=RateLimit(stream_id=\"stream2\"))\n```\n\nIf you want to set custom options for a large group of log calls without repeatedly adding the `extra` parameter, it's possible to use a [LoggerAdapter](https://docs.python.org/3/library/logging.html#loggeradapter-objects):\n```python\nimport logging\nfrom log_rate_limit import StreamRateLimitFilter, RateLimit\n\nlogging.basicConfig(level=logging.INFO)\nlogger = logging.getLogger(__name__)\n# Add our filter\nlogger.addFilter(StreamRateLimitFilter(period_sec=1))\n\n# Use LoggerAdapter to assign additional \"extra\" parameters to all calls using this logger\nglobal_extra = RateLimit(stream_id=\"custom_stream\", period_sec=20)\nlogger = logging.LoggerAdapter(logger, global_extra)\n# Log many warnings\nfor _ in range(100):\n logger.warning(\"Wolf!\")\nfor i in range(100):\n logger.warning(\"No really, a wolf!\")\n```\nWhich merely outputs:\n```log\nWARNING:__main__:Wolf!\n```\nSince both log calls are in the same stream.\n\nAlternatively (to a LoggerAdapter), custom options can also be added by writing your own [logging.Filter](https://docs.python.org/3.8/howto/logging-cookbook.html#using-filters-to-impart-contextual-information).\n\n### Dynamic stream ID\n\nDynamic stream IDs can be assigned based on any criteria you want, e.g.:\n\n```python\nlogger.warning(f\"Error occured on device {device_id}!\", extra=RateLimit(stream_id=f\"error_on_{device_id}\"))\n```\n\n## Installation\n\n### Install from PyPI\n\nWith `Python 3` installed on your system, you can run:\n\n pip install log-rate-limit\n\nTo test that installation worked, you can run:\n\n python -c \"import log_rate_limit\"\n\nand you can uninstall at any time with:\n\n pip uninstall log-rate-limit\n\nTo install with poetry:\n\n poetry add log-rate-limit\n\n### Install from Github\n\nTo install the newest code directly from Github:\n\n pip install git+https://github.com/samuller/log-rate-limit\n",
"bugtrack_url": null,
"license": "",
"summary": "Limit excessive log output with Python's standard logging framework.",
"version": "1.4.1",
"project_urls": {
"Homepage": "https://github.com/samuller/log-rate-limit",
"Repository": "https://github.com/samuller/log-rate-limit"
},
"split_keywords": [
"logging",
"log",
"limit",
"rate"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8bf287bf50e06e69c3a5f50f8b465a2e397399887a7e76844c0845989e0ff312",
"md5": "dbd6f498d783c482ffa325a29885f865",
"sha256": "7cf0dd44ebc6ef4ae3b1f749ab4f67d8cb47036acf18d59cca5aacdd1883ecb5"
},
"downloads": -1,
"filename": "log_rate_limit-1.4.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "dbd6f498d783c482ffa325a29885f865",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8.1,<4.0.0",
"size": 17780,
"upload_time": "2023-11-02T19:44:17",
"upload_time_iso_8601": "2023-11-02T19:44:17.008623Z",
"url": "https://files.pythonhosted.org/packages/8b/f2/87bf50e06e69c3a5f50f8b465a2e397399887a7e76844c0845989e0ff312/log_rate_limit-1.4.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "91739c015029fc6759ecc9196f756224302a13d2bbc18a12f48cedabfb13ca8f",
"md5": "6fb966e05f4f282f1174a5f38ffad492",
"sha256": "3718705b392736675f5eaad8f42e4f319e5c6b6cefd912ea146c77339c19af35"
},
"downloads": -1,
"filename": "log_rate_limit-1.4.1.tar.gz",
"has_sig": false,
"md5_digest": "6fb966e05f4f282f1174a5f38ffad492",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8.1,<4.0.0",
"size": 18318,
"upload_time": "2023-11-02T19:44:18",
"upload_time_iso_8601": "2023-11-02T19:44:18.580563Z",
"url": "https://files.pythonhosted.org/packages/91/73/9c015029fc6759ecc9196f756224302a13d2bbc18a12f48cedabfb13ca8f/log_rate_limit-1.4.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-02 19:44:18",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "samuller",
"github_project": "log-rate-limit",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "log-rate-limit"
}
JSON
