Summary
-------
Run, monitor and log celery tasks.
Installation and setup
----------------------
Declare tasks using celery task or cubicweb-celery cwtasks.
On worker side, install cw-celerytask-helpers_.
celeryconfig.py example::
CELERY_BROKER_URL = 'redis://localhost:6379/0'
CELERY_RESULT_BACKEND = BROKER_URL
CUBICWEB_CELERYTASK_REDIS_URL = CELERY_BROKER_URL
CELERY_IMPORTS = ('cw_celerytask_helpers.helpers', 'module.containing.tasks')
In this configuration example, the ``cw_celerytask_helpers`` in
``CELERY_IMPORTS`` is required to have logging data (in the task) sent
back to the Cubicweb instance via Redis. The
``CUBICWEB_CELERYTASK_REDIS_URL`` is the Redis endpoint used for this
logging handling mechanism.
Start a worker::
# running cubicweb tasks (celeryconfig.py will be imported from your instance config directory)
celery -A cubicweb_celery -i <CW_INSTANCE_NAME> worker -l info
# running pure celery tasks
celery worker -l info
Task state synchronization requires to run the `celery-monitor` command::
cubicweb-ctl celery-monitor <instance-name>
Ensure to have the celeryconfig.py loaded for both cubicweb instance and
celery worker, enforce by settings with CELERY_CONFIG_MODULE environment
variable (it must be an importable python module).
.. _cw-celerytask-helpers: https://www.cubicweb.org/project/cw-celerytask-helpers
Running tasks
-------------
Create a task:
.. code-block:: python
from celery import current_app as app
from celery.utils.log import get_task_logger
logger = get_task_logger(__name__)
@app.task(name='hi_there')
def my_task(arg, kw=0):
logger.info('HI %s %s!', arg, kw)
return 42
Run a task:
.. code-block:: python
from cubicweb_celerytask.entities import start_async_task
cwtask = start_async_task(cnx, 'hi_there', 'THERE', kw=42)
cnx.commit()
start_async_task() accept task names, task objects or task signatures:
http://docs.celeryproject.org/en/latest/userguide/canvas.html#signatures
For instance, to start the above task in a dedicated queue named `myqueue`:
.. code-block:: python
import celery
start_async_task(cnx, celery.signature('hi_there', args=('THERE',),
kwargs={'kw': 42}, queue='myqueue'))
Testing task based application
------------------------------
In CubicWeb test mode, tasks don't run automatically, use
`cubicweb_celerytask.entities.get_tasks()` to introspect them and
`cubicweb_celerytask.entities.run_all_tasks()` to run them.
Also, CELERY_ALWAYS_EAGER and CELERY_EAGER_PROPAGATES_EXCEPTIONS are set to
True by default.
Raw data
{
"_id": null,
"home_page": "https://forge.extranet.logilab.fr/cubicweb/cubes/cubicweb-celerytask",
"name": "cubicweb-celerytask",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9.2",
"maintainer_email": null,
"keywords": null,
"author": "LOGILAB S.A. (Paris, FRANCE)",
"author_email": "contact@logilab.fr",
"download_url": "https://files.pythonhosted.org/packages/01/6b/9f412c2a8f978332b66318df1986194ef29ae7dbf08233b94919a586e849/cubicweb-celerytask-2.0.1.tar.gz",
"platform": null,
"description": "Summary\n-------\n\nRun, monitor and log celery tasks.\n\n\nInstallation and setup\n----------------------\n\nDeclare tasks using celery task or cubicweb-celery cwtasks.\n\nOn worker side, install cw-celerytask-helpers_.\n\nceleryconfig.py example::\n\n CELERY_BROKER_URL = 'redis://localhost:6379/0'\n CELERY_RESULT_BACKEND = BROKER_URL\n CUBICWEB_CELERYTASK_REDIS_URL = CELERY_BROKER_URL\n CELERY_IMPORTS = ('cw_celerytask_helpers.helpers', 'module.containing.tasks')\n\n\nIn this configuration example, the ``cw_celerytask_helpers`` in\n``CELERY_IMPORTS`` is required to have logging data (in the task) sent\nback to the Cubicweb instance via Redis. The\n``CUBICWEB_CELERYTASK_REDIS_URL`` is the Redis endpoint used for this\nlogging handling mechanism.\n\n \nStart a worker::\n\n # running cubicweb tasks (celeryconfig.py will be imported from your instance config directory)\n celery -A cubicweb_celery -i <CW_INSTANCE_NAME> worker -l info\n\n # running pure celery tasks\n celery worker -l info\n\n\nTask state synchronization requires to run the `celery-monitor` command::\n\n cubicweb-ctl celery-monitor <instance-name>\n\n\nEnsure to have the celeryconfig.py loaded for both cubicweb instance and\ncelery worker, enforce by settings with CELERY_CONFIG_MODULE environment\nvariable (it must be an importable python module).\n\n.. _cw-celerytask-helpers: https://www.cubicweb.org/project/cw-celerytask-helpers\n\nRunning tasks\n-------------\n\nCreate a task:\n\n.. code-block:: python\n\n from celery import current_app as app\n from celery.utils.log import get_task_logger\n\n logger = get_task_logger(__name__)\n\n @app.task(name='hi_there')\n def my_task(arg, kw=0):\n logger.info('HI %s %s!', arg, kw)\n return 42\n\n\nRun a task:\n\n.. code-block:: python\n\n from cubicweb_celerytask.entities import start_async_task\n\n cwtask = start_async_task(cnx, 'hi_there', 'THERE', kw=42)\n cnx.commit()\n\n\nstart_async_task() accept task names, task objects or task signatures:\nhttp://docs.celeryproject.org/en/latest/userguide/canvas.html#signatures\n\nFor instance, to start the above task in a dedicated queue named `myqueue`:\n\n.. code-block:: python\n\n import celery\n\n start_async_task(cnx, celery.signature('hi_there', args=('THERE',),\n kwargs={'kw': 42}, queue='myqueue'))\n\n\nTesting task based application\n------------------------------\n\nIn CubicWeb test mode, tasks don't run automatically, use\n`cubicweb_celerytask.entities.get_tasks()` to introspect them and\n`cubicweb_celerytask.entities.run_all_tasks()` to run them.\n\nAlso, CELERY_ALWAYS_EAGER and CELERY_EAGER_PROPAGATES_EXCEPTIONS are set to\nTrue by default.\n",
"bugtrack_url": null,
"license": "LGPL",
"summary": "Run and monitor celery tasks",
"version": "2.0.1",
"project_urls": {
"Homepage": "https://forge.extranet.logilab.fr/cubicweb/cubes/cubicweb-celerytask"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "016b9f412c2a8f978332b66318df1986194ef29ae7dbf08233b94919a586e849",
"md5": "f8a935c6649486bd0d29a4f7d767f701",
"sha256": "1ffd3504d354d4ebefa30631f0c72c17f620ba51c4d6c6ba42be52a229650b5e"
},
"downloads": -1,
"filename": "cubicweb-celerytask-2.0.1.tar.gz",
"has_sig": false,
"md5_digest": "f8a935c6649486bd0d29a4f7d767f701",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9.2",
"size": 21303,
"upload_time": "2024-09-18T13:07:13",
"upload_time_iso_8601": "2024-09-18T13:07:13.663603Z",
"url": "https://files.pythonhosted.org/packages/01/6b/9f412c2a8f978332b66318df1986194ef29ae7dbf08233b94919a586e849/cubicweb-celerytask-2.0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-18 13:07:13",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "cubicweb-celerytask"
}
JSON