| Name | aa-taskmonitor JSON |
| Version |
0.23.0
JSON |
| download |
| home_page | |
| Summary | An Alliance Auth app for monitoring celery tasks. |
| upload_time | 2023-12-13 07:24:57 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.8 |
| license | |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Task Monitor
An Alliance Auth app for monitoring celery tasks.
[](https://pypi.org/project/aa-taskmonitor/)
[](https://pypi.org/project/aa-taskmonitor/)
[](https://pypi.org/project/aa-taskmonitor/)
[](https://gitlab.com/ErikKalkoken/aa-taskmonitor/-/pipelines)
[](https://codecov.io/gl/ErikKalkoken/aa-taskmonitor)
[](https://gitlab.com/ErikKalkoken/aa-taskmonitor/-/blob/master/LICENSE)
[](https://github.com/pre-commit/pre-commit)
[](https://github.com/psf/black)
[](https://discord.gg/zmh52wnfvM)
## Contents
- [Features](#features)
- [Screenshots](#screenshots)
- [Installation](#installation)
- [Command line utility](#command-line-utility)
- [Settings](#settings)
- [FAQ](#faq)
- [Change Log](CHANGELOG.md)
## Features
Task Monitor enables administrators to monitor celery tasks running on their system.
- Creates a log of all recently executed celery tasks including failed and retried tasks.
- Stores many details in task logs to support the analysis of potential celery issues, including the parameters a task with called with and complete exception messages
- Keeps the storage needs in check by automatically deleting older task logs and removing likely bloat from the collected data (but can also be turned off)
- Admins can investigate task log with search & filters
- Admins can view details for each task log incl. exceptions and trace logs
- Admins can review reports with charts providing answers to common questions, e.g:
- How many tasks have failed/retried?
- How many tasks where run by each of my apps?
- Which are the most frequent tasks?
- Which tasks have the longest runtime?
- Which tasks failed the most?
- How much backlog do I have in task queue over time?
- Admins can export all task logs to a CSV file for further analysis with 3rd party tools (e.g. Google sheets)
- Admins can see detailed statistics for all tasks
- Command line utility to manage task logs and queue directly
## Screenshots
### Full log of all recently executed tasks
### View details for each task incl. exception tracelogs
### Example chart in reports
### View detailed statistics for all tasks
## Installation
### Step 1 - Check prerequisites
Task Monitor is a plugin for Alliance Auth. If you don't have Alliance Auth running already, please install it first before proceeding. (see the official [AA installation guide](https://allianceauth.readthedocs.io/en/latest/installation/auth/allianceauth/) for details)
### Step 2 - Install app
Make sure you are in the virtual environment (venv) of your Alliance Auth installation. Then install the newest release from PyPI:
```bash
pip install aa-taskmonitor
```
### Step 3 - Configure Auth settings
Configure your Auth settings (`local.py`) as follows:
- Add `'taskmonitor'` to `INSTALLED_APPS`
- Optional: Add additional settings if you want to change any defaults. See [Settings](#settings) for the full list.
### Step 4 - Finalize App installation
Run migrations & copy static files
```bash
python manage.py migrate
python manage.py collectstatic --noinput
```
Restart your supervisor services for Auth.
## Command line utility
You can manage your logs and queue also directly via a command line utility.
The basic syntax for using the utility is:
```bash
python manage.py taskmonitorctl {command} {target}
```
For example you can inspect your logs with:
```bash
python manage.py taskmonitorctl inspect logs
```
Or for example you can purge your task queue with:
```bash
python manage.py taskmonitorctl purge queue
```
To get an overview of available commands run:
```bash
python manage.py taskmonitorctl --help
```
To get an overview of available targets for a command run:
```bash
python manage.py taskmonitorctl inspect --help
```
## Settings
Here is a list of available settings for this app. They can be configured by adding them to your AA settings file (`local.py`).
Note that all settings are optional and the app will use the documented default settings if they are not used.
Name|Description|Default
--|--|--
`TASKMONITOR_APP_NAME_MAPPING_CONFIG`|Ability to map tasks to the same app name. Map must be a dictionary with string keys and list of strings as value. All app names in the list will be replaced by it's key.|`{}`
`TASKMONITOR_DATA_MAX_AGE`|Max age of logged tasks in hours. Older logs be deleted automatically.|`24`
`TASKMONITOR_DELETE_STALE_BATCH_SIZE`|Size of task logs deleted together in one batch.|`5000`
`TASKMONITOR_ENABLED`|Global switch to enable/disable task monitor.|`True`
`TASKMONITOR_HOUSEKEEPING_FREQUENCY`|Frequency of house keeping runs in minutes.|`30`
`TASKMONITOR_QUEUED_TASKS_ADMIN_LIMIT`|The admin page will stop showing the list of queued tasks above this limit to protect against crashing caused by too high memory consumption.|`100000`
`TASKMONITOR_QUEUED_TASKS_CACHE_TIMEOUT`|Timeout for caching queued tasks in seconds. 0 will deactivate the cache.|`10`
`TASKMONITOR_REPORTS_MAX_AGE`|Max age of cached reports in minutes.|`60`
`TASKMONITOR_REPORTS_MAX_TOP`|Max items to show in the top reports. e.g. 10 will shop the top ten items.|`20`
`TASKMONITOR_TRUNCATE_NESTED_DATA`|Whether deeply nested task params and results are truncated.|`True`
## FAQ
### Is it possible to store task logs longer then for just 24 hours?
Yes, there is a setting, which you can increase according to your needs. However, please keep in mind that your storage needs will increase accordingly. The current approx. usage is 0.5 KB per entry, so e.g. you need approx. 500 MB to store 1.000.000 task logs. Note that you can use the command line utility to find out how much space your logs are currently using.
### How is this app different from celery analytics?
Celery Analytics seams to be designed mainly as data source for reports on Grafana. It apparently works great if you want to integrate analysis about your task executions into a Grafana dashboards. But it's usability without Grafana is limited.
Task Monitor on the other hand aims to be fully functional standalone by providing reports and many useful features for analyzing your task logs directly on the admin site. It also provides a more complete picture, since Celery Analytics ignores retried tasks.
## How is this app different from celery's flower?
Flower offers more detailed and technical information about task runs and might therefore be more most for developers. However, it not designed to store a larger number of task logs (default is only 10K) and is appears therefore to be less suited for Alliance Auth, where you typically have 100K+ tasks per day.
## What does data truncating do exactly?
Task Monitor has data truncating enabled by default. It is applied when storing args, kwargs and results in task logs. This helps to reduce the storage consumptions and also makes the task log better readable. But it can be turned off.
Task args are truncated by clearing all nested containers.
Example: `[1, [2, 3], 4]` becomes `[1, [], 4]`
Task kwargs are truncated by clearing all nested containers in values.
Example: `{"a": [1, 2], "b": 3}` becomes `{"a": [], "b": 3}`
Finally, task results are truncated like args and kwargs depending on their type. In addition lists of empty containers are compressed.
Example: `[ [], [], [] ]` becomes `[]`
Raw data
{
"_id": null,
"home_page": "",
"name": "aa-taskmonitor",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "",
"author": "",
"author_email": "Erik Kalkoken <kalkoken87@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/ad/cb/a8d9ef0c2d8cf22fde026a0250a4f65e6b9f40216d3e268a35230e1847cd/aa_taskmonitor-0.23.0.tar.gz",
"platform": null,
"description": "# Task Monitor\n\nAn Alliance Auth app for monitoring celery tasks.\n\n[](https://pypi.org/project/aa-taskmonitor/)\n[](https://pypi.org/project/aa-taskmonitor/)\n[](https://pypi.org/project/aa-taskmonitor/)\n[](https://gitlab.com/ErikKalkoken/aa-taskmonitor/-/pipelines)\n[](https://codecov.io/gl/ErikKalkoken/aa-taskmonitor)\n[](https://gitlab.com/ErikKalkoken/aa-taskmonitor/-/blob/master/LICENSE)\n[](https://github.com/pre-commit/pre-commit)\n[](https://github.com/psf/black)\n[](https://discord.gg/zmh52wnfvM)\n\n## Contents\n\n- [Features](#features)\n- [Screenshots](#screenshots)\n- [Installation](#installation)\n- [Command line utility](#command-line-utility)\n- [Settings](#settings)\n- [FAQ](#faq)\n- [Change Log](CHANGELOG.md)\n\n## Features\n\nTask Monitor enables administrators to monitor celery tasks running on their system.\n\n- Creates a log of all recently executed celery tasks including failed and retried tasks.\n- Stores many details in task logs to support the analysis of potential celery issues, including the parameters a task with called with and complete exception messages\n- Keeps the storage needs in check by automatically deleting older task logs and removing likely bloat from the collected data (but can also be turned off)\n- Admins can investigate task log with search & filters\n- Admins can view details for each task log incl. exceptions and trace logs\n- Admins can review reports with charts providing answers to common questions, e.g:\n - How many tasks have failed/retried?\n - How many tasks where run by each of my apps?\n - Which are the most frequent tasks?\n - Which tasks have the longest runtime?\n - Which tasks failed the most?\n - How much backlog do I have in task queue over time?\n- Admins can export all task logs to a CSV file for further analysis with 3rd party tools (e.g. Google sheets)\n- Admins can see detailed statistics for all tasks\n- Command line utility to manage task logs and queue directly\n\n## Screenshots\n\n### Full log of all recently executed tasks\n\n\n\n### View details for each task incl. exception tracelogs\n\n\n\n### Example chart in reports\n\n\n\n### View detailed statistics for all tasks\n\n\n\n## Installation\n\n### Step 1 - Check prerequisites\n\nTask Monitor is a plugin for Alliance Auth. If you don't have Alliance Auth running already, please install it first before proceeding. (see the official [AA installation guide](https://allianceauth.readthedocs.io/en/latest/installation/auth/allianceauth/) for details)\n\n### Step 2 - Install app\n\nMake sure you are in the virtual environment (venv) of your Alliance Auth installation. Then install the newest release from PyPI:\n\n```bash\npip install aa-taskmonitor\n```\n\n### Step 3 - Configure Auth settings\n\nConfigure your Auth settings (`local.py`) as follows:\n\n- Add `'taskmonitor'` to `INSTALLED_APPS`\n- Optional: Add additional settings if you want to change any defaults. See [Settings](#settings) for the full list.\n\n### Step 4 - Finalize App installation\n\nRun migrations & copy static files\n\n```bash\npython manage.py migrate\npython manage.py collectstatic --noinput\n```\n\nRestart your supervisor services for Auth.\n\n## Command line utility\n\nYou can manage your logs and queue also directly via a command line utility.\n\nThe basic syntax for using the utility is:\n\n```bash\npython manage.py taskmonitorctl {command} {target}\n```\n\nFor example you can inspect your logs with:\n\n```bash\npython manage.py taskmonitorctl inspect logs\n```\n\nOr for example you can purge your task queue with:\n\n```bash\npython manage.py taskmonitorctl purge queue\n```\n\nTo get an overview of available commands run:\n\n```bash\npython manage.py taskmonitorctl --help\n```\n\nTo get an overview of available targets for a command run:\n\n```bash\npython manage.py taskmonitorctl inspect --help\n```\n\n## Settings\n\nHere is a list of available settings for this app. They can be configured by adding them to your AA settings file (`local.py`).\n\nNote that all settings are optional and the app will use the documented default settings if they are not used.\n\nName|Description|Default\n--|--|--\n`TASKMONITOR_APP_NAME_MAPPING_CONFIG`|Ability to map tasks to the same app name. Map must be a dictionary with string keys and list of strings as value. All app names in the list will be replaced by it's key.|`{}`\n`TASKMONITOR_DATA_MAX_AGE`|Max age of logged tasks in hours. Older logs be deleted automatically.|`24`\n`TASKMONITOR_DELETE_STALE_BATCH_SIZE`|Size of task logs deleted together in one batch.|`5000`\n`TASKMONITOR_ENABLED`|Global switch to enable/disable task monitor.|`True`\n`TASKMONITOR_HOUSEKEEPING_FREQUENCY`|Frequency of house keeping runs in minutes.|`30`\n`TASKMONITOR_QUEUED_TASKS_ADMIN_LIMIT`|The admin page will stop showing the list of queued tasks above this limit to protect against crashing caused by too high memory consumption.|`100000`\n`TASKMONITOR_QUEUED_TASKS_CACHE_TIMEOUT`|Timeout for caching queued tasks in seconds. 0 will deactivate the cache.|`10`\n`TASKMONITOR_REPORTS_MAX_AGE`|Max age of cached reports in minutes.|`60`\n`TASKMONITOR_REPORTS_MAX_TOP`|Max items to show in the top reports. e.g. 10 will shop the top ten items.|`20`\n`TASKMONITOR_TRUNCATE_NESTED_DATA`|Whether deeply nested task params and results are truncated.|`True`\n\n## FAQ\n\n### Is it possible to store task logs longer then for just 24 hours?\n\nYes, there is a setting, which you can increase according to your needs. However, please keep in mind that your storage needs will increase accordingly. The current approx. usage is 0.5 KB per entry, so e.g. you need approx. 500 MB to store 1.000.000 task logs. Note that you can use the command line utility to find out how much space your logs are currently using.\n\n### How is this app different from celery analytics?\n\nCelery Analytics seams to be designed mainly as data source for reports on Grafana. It apparently works great if you want to integrate analysis about your task executions into a Grafana dashboards. But it's usability without Grafana is limited.\n\nTask Monitor on the other hand aims to be fully functional standalone by providing reports and many useful features for analyzing your task logs directly on the admin site. It also provides a more complete picture, since Celery Analytics ignores retried tasks.\n\n## How is this app different from celery's flower?\n\nFlower offers more detailed and technical information about task runs and might therefore be more most for developers. However, it not designed to store a larger number of task logs (default is only 10K) and is appears therefore to be less suited for Alliance Auth, where you typically have 100K+ tasks per day.\n\n## What does data truncating do exactly?\n\nTask Monitor has data truncating enabled by default. It is applied when storing args, kwargs and results in task logs. This helps to reduce the storage consumptions and also makes the task log better readable. But it can be turned off.\n\nTask args are truncated by clearing all nested containers.\n\nExample: `[1, [2, 3], 4]` becomes `[1, [], 4]`\n\nTask kwargs are truncated by clearing all nested containers in values.\n\nExample: `{\"a\": [1, 2], \"b\": 3}` becomes `{\"a\": [], \"b\": 3}`\n\nFinally, task results are truncated like args and kwargs depending on their type. In addition lists of empty containers are compressed.\n\nExample: `[ [], [], [] ]` becomes `[]`\n\n",
"bugtrack_url": null,
"license": "",
"summary": "An Alliance Auth app for monitoring celery tasks.",
"version": "0.23.0",
"project_urls": {
"Changelog": "https://gitlab.com/ErikKalkoken/aa-taskmonitor/-/blob/master/CHANGELOG.md",
"Homepage": "https://gitlab.com/ErikKalkoken/aa-taskmonitor",
"Source": "https://gitlab.com/ErikKalkoken/aa-taskmonitor",
"Tracker": "https://gitlab.com/ErikKalkoken/aa-taskmonitor/-/issues"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "cbb1ef8fd0ef509738fc0fd5240b409f407d52b853cdb0f243f9efb491e92e65",
"md5": "2054163e17efa12cd0107c700eb09941",
"sha256": "acd74c59005237a10aae8e836ff0bcb20271c0fa25fcb5272be035b29396ed74"
},
"downloads": -1,
"filename": "aa_taskmonitor-0.23.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "2054163e17efa12cd0107c700eb09941",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 485535,
"upload_time": "2023-12-13T07:24:54",
"upload_time_iso_8601": "2023-12-13T07:24:54.884941Z",
"url": "https://files.pythonhosted.org/packages/cb/b1/ef8fd0ef509738fc0fd5240b409f407d52b853cdb0f243f9efb491e92e65/aa_taskmonitor-0.23.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "adcba8d9ef0c2d8cf22fde026a0250a4f65e6b9f40216d3e268a35230e1847cd",
"md5": "b391ab33da43205f0b508aad42aa1d42",
"sha256": "0fbd4a4065e3c90e3d7f6336fdb5ceeffb8a84f21501c945ec86d021ac0cb87d"
},
"downloads": -1,
"filename": "aa_taskmonitor-0.23.0.tar.gz",
"has_sig": false,
"md5_digest": "b391ab33da43205f0b508aad42aa1d42",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 458353,
"upload_time": "2023-12-13T07:24:57",
"upload_time_iso_8601": "2023-12-13T07:24:57.156630Z",
"url": "https://files.pythonhosted.org/packages/ad/cb/a8d9ef0c2d8cf22fde026a0250a4f65e6b9f40216d3e268a35230e1847cd/aa_taskmonitor-0.23.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-12-13 07:24:57",
"github": false,
"gitlab": true,
"bitbucket": false,
"codeberg": false,
"gitlab_user": "ErikKalkoken",
"gitlab_project": "aa-taskmonitor",
"lcname": "aa-taskmonitor"
}
