| Name | tutor-contrib-celery JSON |
| Version |
19.0.0
JSON |
| download |
| home_page | None |
| Summary | A Tutor plugin to manage celery deployments |
| upload_time | 2024-12-18 19:14:53 |
| maintainer | None |
| docs_url | None |
| author | eduNEXT |
| requires_python | >=3.9 |
| license | AGPLv3 |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Celery plugin for [Tutor](https://docs.tutor.edly.io)
A tutor plugin to extend the default LMS and CMS celery workers included in Tutor.
It adds and configures extra deployments running LMS and CMS celery workers where
every deployment will process async tasks routed to a specific queue. Having this
workers separation per queue can help to define the scaling requirements for the Celery
deployments, since having a single queue (the default one) with a single deployment can
lead to unexpected behaviors when running large-scale sites.
## Installation
```shell
pip install tutor-contrib-celery
```
## Usage
```shell
tutor plugins enable celery
```
## Configuration
### Celery queues
By default, in a standard OpenedX installation with Tutor in Kubernetes, all the LMS/CMS async tasks are executed
by a single celery deployment. This plugin allows to distribute async workload by configuring additional deployments
to execute celery tasks sent to a specific queues. This can help to:
- Achieve a better performance when having high volume of async tasks to process
- Configure different scaling parameters according to the nature of the tasks processed by a queue (I/O bound tasks,
CPU tasks, etc.)
To achieve this, the `CELERY_WORKERS_CONFIG` filter is implemented to add extra queues whose tasks require to be
processed by a separated deployment.
## Recommended multiqueue configuration
From checking the LMS and CMS codebase, the queues for every service are described below:
- **CMS**: default, high, low (taken from CMS settings [here](https://github.com/openedx/edx-platform/blob/open-release/redwood.master/cms/envs/common.py#L1578-L1582))
- **LMS**: default, high, high_mem (taken from LMS settings [here](https://github.com/openedx/edx-platform/blob/open-release/redwood.master/lms/envs/common.py#L2913-L2917))
By default Tutor implements a single deployment to process tasks on all queues in LMS/CMS. The `CELERY_WORKERS_CONFIG` filter
can be used to add the extra queues from LMS/CMS configuration.
```python
from tutorcelery.hooks import CELERY_WORKERS_CONFIG
@CELERY_WORKERS_CONFIG.add()
def _add_celery_workers_config(workers_config):
# Adding LMS extra queues
workers_config["lms"]["high"] = {} # Make sure to match the key with the queue name: edx.lms.core.high
workers_config["lms"]["high_mem"] = {}
# Adding CMS extra queues
workers_config["cms"]["high"] = {}
workers_config["cms"]["low"] = {}
return workers_config
```
With this configuration, 4 new deployments will be created (one for every new queue) to process the tasks
separately according to the queue they are sent to. Additionally, the default Tutor LMS/CMS celery deployments
are patched to ONLY process the tasks sent to the "default" queue.
This is the recommended configuration for a multiqueue approach with LMS and CMS given the queues every
service proposes in its settings files by default. However, the usage of the `CELERY_WORKERS_CONFIG` filter
can be adapted for different configuration scenarios.
This plugin also provides a setting to directly route LMS/CMS tasks to an specific queue. It can extends/overrides
the default `EXPLICIT_QUEUES` setting:
```yaml
CELERY_LMS_EXPLICIT_QUEUES:
lms.djangoapps.grades.tasks.compute_all_grades_for_course:
queue: edx.lms.core.high_mem
CELERY_CMS_EXPLICIT_QUEUES:
cms.djangoapps.contentstore.tasks.import_olx:
queue: edx.cms.core.high
```
### Custom parameters
Each deployment can be configured to run with different paramaters to override the defaults, the setting `extra_param`
is a list that can be used to pass custom parameters to the Celery workers. e.g changing the Celery's pool parameter
for the high_mem lms worker deployment:
```python
@CELERY_WORKERS_CONFIG.add()
def _add_celery_workers_config(workers_config):
# Adding LMS extra queues
workers_config["lms"]["high_mem"]["extra_params"] = {
"--pool=gevent",
"--concurrency=100",
}
return workers_config
```
### Autoscaling
As an alternative to the CPU/memory based autoscaling offered by the plugin [tutor-contrib-pod-autoscaling](https://github.com/eduNEXT/tutor-contrib-pod-autoscaling),
this plugins supports Celery workers autoscaling based on the size of the celery queue of a given worker. We are using
Keda autoscaling for this purposes, check the [Keda documentation](https://keda.sh/docs) to find out more.
To enable autoscaling you need to enable the `enable_keda` key for every queue variant. The defaults parameters are the following:
```python
{
"min_replicas": 0,
"max_replicas": 30,
"list_length": 40,
"enable_keda": False,
}
```
> [!NOTE]
> You can use the filter `CELERY_WORKERS_CONFIG` as shown above to modify the scaling parameters for every queue.
> This has been test against Keda v2.15.0 and above.
If you are using [tutor-contrib-pod-autoscaling](https://github.com/eduNEXT/tutor-contrib-pod-autoscaling) and want to setup Keda autoscaling, make sure to disable HPA for the `lms-worker` and the `cms-worker` as **using both autoscalers at the same time is not recommended**.
```python
from tutorpod_autoscaling.hooks import AUTOSCALING_CONFIG
@AUTOSCALING_CONFIG.add()
def _add_my_autoscaling(autoscaling_config):
autoscaling_config["lms-worker"].update({
"enable_hpa": False,
})
autoscaling_config["cms-worker"].update({
"enable_hpa": False,
})
return autoscaling_config
```
### Enable flower
For troubleshooting purposes, you can enable a flower deployment to monitor in realtime the Celery queues
times and performance:
```yaml
CELERY_FLOWER: true
```
#### Enable Flower Prometheus Integration
If you are running grafana you can use the attached [config map](resources/configmap.yaml) to import a custom Grafana dashboard to monitor
celery metrics such as:
- Total Queue Length
- Queue Length by task name
- Celery Worker Status
- Number of Tasks Currently Executing at Worker
- Average Task Runtime at Worker
- Task Prefetch Time at Worker
- Number of Tasks Prefetched at Worker
- Tasks Success Ratio
- Tasks Failure Ratio
If you are using the [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator) you can enable a ServiceMonitor resource to automatically configure a scrape target for the flower service.
```yaml
CELERY_FLOWER_SERVICE_MONITOR: true
```
License
---
This software is licensed under the terms of the AGPLv3.
Raw data
{
"_id": null,
"home_page": null,
"name": "tutor-contrib-celery",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": null,
"author": "eduNEXT",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/be/eb/daf47c074f61d345c142b5ddfa9b19ba17d2b1b50e95996fcf1431b10110/tutor_contrib_celery-19.0.0.tar.gz",
"platform": null,
"description": "# Celery plugin for [Tutor](https://docs.tutor.edly.io)\n\nA tutor plugin to extend the default LMS and CMS celery workers included in Tutor.\nIt adds and configures extra deployments running LMS and CMS celery workers where\nevery deployment will process async tasks routed to a specific queue. Having this\nworkers separation per queue can help to define the scaling requirements for the Celery\ndeployments, since having a single queue (the default one) with a single deployment can\nlead to unexpected behaviors when running large-scale sites.\n\n## Installation\n\n```shell\npip install tutor-contrib-celery\n```\n\n## Usage\n\n```shell\ntutor plugins enable celery\n```\n\n## Configuration\n\n### Celery queues\n\nBy default, in a standard OpenedX installation with Tutor in Kubernetes, all the LMS/CMS async tasks are executed\nby a single celery deployment. This plugin allows to distribute async workload by configuring additional deployments\nto execute celery tasks sent to a specific queues. This can help to:\n\n- Achieve a better performance when having high volume of async tasks to process\n- Configure different scaling parameters according to the nature of the tasks processed by a queue (I/O bound tasks,\nCPU tasks, etc.)\n\nTo achieve this, the `CELERY_WORKERS_CONFIG` filter is implemented to add extra queues whose tasks require to be\nprocessed by a separated deployment.\n\n## Recommended multiqueue configuration\n\nFrom checking the LMS and CMS codebase, the queues for every service are described below:\n\n- **CMS**: default, high, low (taken from CMS settings [here](https://github.com/openedx/edx-platform/blob/open-release/redwood.master/cms/envs/common.py#L1578-L1582))\n- **LMS**: default, high, high_mem (taken from LMS settings [here](https://github.com/openedx/edx-platform/blob/open-release/redwood.master/lms/envs/common.py#L2913-L2917))\n\nBy default Tutor implements a single deployment to process tasks on all queues in LMS/CMS. The `CELERY_WORKERS_CONFIG` filter\ncan be used to add the extra queues from LMS/CMS configuration.\n\n```python\n\nfrom tutorcelery.hooks import CELERY_WORKERS_CONFIG\n\n@CELERY_WORKERS_CONFIG.add()\ndef _add_celery_workers_config(workers_config):\n # Adding LMS extra queues\n workers_config[\"lms\"][\"high\"] = {} # Make sure to match the key with the queue name: edx.lms.core.high\n workers_config[\"lms\"][\"high_mem\"] = {}\n\n # Adding CMS extra queues\n workers_config[\"cms\"][\"high\"] = {}\n workers_config[\"cms\"][\"low\"] = {}\n return workers_config\n```\nWith this configuration, 4 new deployments will be created (one for every new queue) to process the tasks\nseparately according to the queue they are sent to. Additionally, the default Tutor LMS/CMS celery deployments\nare patched to ONLY process the tasks sent to the \"default\" queue.\n\nThis is the recommended configuration for a multiqueue approach with LMS and CMS given the queues every\nservice proposes in its settings files by default. However, the usage of the `CELERY_WORKERS_CONFIG` filter\ncan be adapted for different configuration scenarios.\n\nThis plugin also provides a setting to directly route LMS/CMS tasks to an specific queue. It can extends/overrides\nthe default `EXPLICIT_QUEUES` setting:\n\n```yaml\nCELERY_LMS_EXPLICIT_QUEUES:\n lms.djangoapps.grades.tasks.compute_all_grades_for_course:\n queue: edx.lms.core.high_mem\nCELERY_CMS_EXPLICIT_QUEUES:\n cms.djangoapps.contentstore.tasks.import_olx:\n queue: edx.cms.core.high\n```\n\n### Custom parameters\n\nEach deployment can be configured to run with different paramaters to override the defaults, the setting `extra_param`\nis a list that can be used to pass custom parameters to the Celery workers. e.g changing the Celery's pool parameter\nfor the high_mem lms worker deployment:\n\n```python\n@CELERY_WORKERS_CONFIG.add()\ndef _add_celery_workers_config(workers_config):\n # Adding LMS extra queues\n workers_config[\"lms\"][\"high_mem\"][\"extra_params\"] = {\n \"--pool=gevent\",\n \"--concurrency=100\",\n }\n\n return workers_config\n```\n\n### Autoscaling\n\nAs an alternative to the CPU/memory based autoscaling offered by the plugin [tutor-contrib-pod-autoscaling](https://github.com/eduNEXT/tutor-contrib-pod-autoscaling),\nthis plugins supports Celery workers autoscaling based on the size of the celery queue of a given worker. We are using\nKeda autoscaling for this purposes, check the [Keda documentation](https://keda.sh/docs) to find out more.\n\nTo enable autoscaling you need to enable the `enable_keda` key for every queue variant. The defaults parameters are the following:\n\n```python\n{\n \"min_replicas\": 0,\n \"max_replicas\": 30,\n \"list_length\": 40,\n \"enable_keda\": False,\n}\n```\n\n> [!NOTE]\n> You can use the filter `CELERY_WORKERS_CONFIG` as shown above to modify the scaling parameters for every queue.\n> This has been test against Keda v2.15.0 and above.\n\nIf you are using [tutor-contrib-pod-autoscaling](https://github.com/eduNEXT/tutor-contrib-pod-autoscaling) and want to setup Keda autoscaling, make sure to disable HPA for the `lms-worker` and the `cms-worker` as **using both autoscalers at the same time is not recommended**.\n\n```python\nfrom tutorpod_autoscaling.hooks import AUTOSCALING_CONFIG\n\n@AUTOSCALING_CONFIG.add()\ndef _add_my_autoscaling(autoscaling_config):\n autoscaling_config[\"lms-worker\"].update({\n \"enable_hpa\": False,\n })\n autoscaling_config[\"cms-worker\"].update({\n \"enable_hpa\": False,\n })\n return autoscaling_config\n```\n\n### Enable flower\n\nFor troubleshooting purposes, you can enable a flower deployment to monitor in realtime the Celery queues\ntimes and performance:\n\n```yaml\nCELERY_FLOWER: true\n```\n\n#### Enable Flower Prometheus Integration\n\nIf you are running grafana you can use the attached [config map](resources/configmap.yaml) to import a custom Grafana dashboard to monitor\ncelery metrics such as:\n\n- Total Queue Length\n- Queue Length by task name\n- Celery Worker Status\n- Number of Tasks Currently Executing at Worker\n- Average Task Runtime at Worker\n- Task Prefetch Time at Worker\n- Number of Tasks Prefetched at Worker\n- Tasks Success Ratio\n- Tasks Failure Ratio\n\nIf you are using the [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator) you can enable a ServiceMonitor resource to automatically configure a scrape target for the flower service.\n\n```yaml\nCELERY_FLOWER_SERVICE_MONITOR: true\n```\n\nLicense\n\n---\n\nThis software is licensed under the terms of the AGPLv3.\n",
"bugtrack_url": null,
"license": "AGPLv3",
"summary": "A Tutor plugin to manage celery deployments",
"version": "19.0.0",
"project_urls": {
"Code": "https://github.com/edunext/tutor-contrib-celery",
"Homepage": "https://github.com/edunext/tutor-contrib-celery",
"Issue tracker": "https://github.com/edunext/tutor-contrib-celery/issues"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "56022b51a61b75d570053f73069f3ac444d270dcc76951655f237224c925b878",
"md5": "bddfa8de95d051718e8bc0ea42ed71a7",
"sha256": "acbdafea876ac249362b0547bc328ecfb063a95a8ba8c6fff73e41ac8de5ef75"
},
"downloads": -1,
"filename": "tutor_contrib_celery-19.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "bddfa8de95d051718e8bc0ea42ed71a7",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 24653,
"upload_time": "2024-12-18T19:14:52",
"upload_time_iso_8601": "2024-12-18T19:14:52.550581Z",
"url": "https://files.pythonhosted.org/packages/56/02/2b51a61b75d570053f73069f3ac444d270dcc76951655f237224c925b878/tutor_contrib_celery-19.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "beebdaf47c074f61d345c142b5ddfa9b19ba17d2b1b50e95996fcf1431b10110",
"md5": "dfe17687330150e756f276dc29b71b3b",
"sha256": "610a20deca67e2bd9284e8eea6d41ad855a5a8bad2a78e987ca06f6a7678256d"
},
"downloads": -1,
"filename": "tutor_contrib_celery-19.0.0.tar.gz",
"has_sig": false,
"md5_digest": "dfe17687330150e756f276dc29b71b3b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 22049,
"upload_time": "2024-12-18T19:14:53",
"upload_time_iso_8601": "2024-12-18T19:14:53.980434Z",
"url": "https://files.pythonhosted.org/packages/be/eb/daf47c074f61d345c142b5ddfa9b19ba17d2b1b50e95996fcf1431b10110/tutor_contrib_celery-19.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-18 19:14:53",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "edunext",
"github_project": "tutor-contrib-celery",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "tutor-contrib-celery"
}
