django-toosimple-q


Namedjango-toosimple-q JSON
Version 0.2.0 PyPI version JSON
download
home_pagehttps://github.com/olivierdalang/django-toosimple-q
SummaryA simplistic task queue and cron-like scheduler for Django
upload_time2021-06-11 08:36:47
maintainer
docs_urlNone
authorOlivier Dalang
requires_python
licenseMIT
keywords django-toosimple-q
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # Django Too Simple Queue

[![PyPI version](https://badge.fury.io/py/django-toosimple-q.svg)](https://pypi.org/project/django-toosimple-q/) ![Workflow](https://github.com/olivierdalang/django-toosimple-q/workflows/ci/badge.svg)

This packages provides a simplistic task queue and scheduler for Django.

If execution of your tasks is mission critical, do not use this library, and turn to more complex solutions such as Celery, as this package doesn't guarantee task execution nor unique execution.

It is geared towards basic apps, where simplicity primes over reliability. The package offers simple decorator syntax, including cron-like schedules.

Features :

- no celery/redis/rabbitmq/whatever... just Django !
- clean decorator syntax to register tasks and schedules
- simple queuing syntax
- cron-like scheduling
- tasks.py autodiscovery
- django admin integration

Limitations :

- probably not extremely reliable because of race conditions
- no multithreading yet (but running multiple workers should work)

## Installation

Install the library :
```shell
$ pip install django-toosimple-q
```

Enable the app in `settings.py` :
```python
INSTALLED_APPS = [
    ...
    'django_toosimple_q',
    ...
]
```

## Quickstart

Tasks need to be registered using the `@register_task()` decorator. Once registered, they can be added to the queue by calling the `.queue()` function.

```python
from django_toosimple_q.decorators import register_task

# Register a task
@register_task()
def my_task(name):
    return f"Hello {name} !"

# Enqueue tasks
my_task.queue("John")
my_task.queue("Peter")
```

Registered tasks can be scheduled from code using this cron-like syntax :
```python
from django_toosimple_q.decorators import register_task, schedule

# Register and schedule tasks
@schedule(cron="30 8 * * *", args=['John'])
@register_task()
def morning_routine(name):
    return f"Good morning {name} !"
```

To consume the tasks, you need to run at least one worker :
```shell
$ python manage.py worker
```
The workers will take care of adding scheduled tasks to the queue when needed, and will execute the tasks.

The package autoloads `tasks.py` from all installed apps. While this is the recommended place to define your tasks, you can do so from anywhere in your code.

## Advanced usage

### Tasks

You can optionnaly give a custom name to your tasks. This is required when your task is defined in a local scope.
```python
@register_task("my_favourite_task")
def my_task(name):
    return f"Good morning {name} !"
```

You can set task priorities.
```python
@register_task(priority=0)
def my_favourite_task(name):
    return f"Good bye {name} !"

@register_task(priority=1)
def my_other_task(name):
    return f"Hello {name} !"

# Enqueue tasks
my_other_task.queue("John")
my_favourite_task.queue("Peter")  # will be executed before the other one
```

You can define `retries=N` and `retry_delay=S` to retry the task in case of failure. The delay (in second) will double on each failure.

```python
@register_task(retries=10, retry_delay=60)
def send_email():
    ...
```

You can mark a task as `unique=True` if the task shouldn't be queued again if already queued with the same arguments. This is usefull for tasks such as cleaning or refreshing.

```python
@register_task(unique=True)
def cleanup():
    ...

cleanup.queue()
cleanup.queue()  # this will be ignored as long as the first one is still queued
```

You can assign tasks to specific queues, and then have your worker only consume tasks from specific queues using `--queue myqueue` or `--exclude_queue myqueue`. By default, workers consume all tasks.

```python
@register_task(queue='long_running')
def long_task():
    ...

@register_task()
def short_task():
    ...

# Then run those with these workers, so that long
# running tasks don't prevent short running tasks
# from being run :
# manage.py worker --exclude_queue long_running
# manage.py worker
```

### Schedules

By default, `last_check` is set to `now()` on schedule creation. This means they will only run on next cron occurence. If you need your schedules to be run as soon as possible after initialisation, you can specify `last_check=None`.

```python
@schedule(cron="30 8 * * *", last_check=None)
@register_task()
def my_task(name):
    return f"Good morning {name} !"
```

By default, if some crons where missed (e.g. after a server shutdown or if the workers can't keep up with all tasks), the missed tasks will be lost. If you need the tasks to catch up, set `catch_up=True`.

```python
@schedule(cron="30 8 * * *", catch_up=True)
@register_task()
def my_task(name):
    ...
```

You may define multiple schedules for the same task. In this case, it is mandatory to specify a unique name :

```python
@schedule(name="morning_routine", cron="30 16 * * *", args=['morning'])
@schedule(name="afternoon_routine", cron="30 8 * * *", args=['afternoon'])
@register_task()
def my_task(time):
    return f"Good {time} John !"
```

### Management comment

Besides standard django management commands arguments, the management command supports following arguments.

```
usage: manage.py worker [--queue QUEUE | --exclude_queue EXCLUDE_QUEUE]
                        [--tick TICK]
                        [--once | --until_done]
                        [--no_recreate | --recreate_only]

optional arguments:
  --queue QUEUE         which queue to run (can be used several times, all
                        queues are run if not provided)
  --exclude_queue EXCLUDE_QUEUE
                        which queue not to run (can be used several times, all
                        queues are run if not provided)
  --tick TICK           frequency in seconds at which the database is checked
                        for new tasks/schedules
  --once                run once then exit (useful for debugging)
  --until_done          run until no tasks are available then exit (useful for
                        debugging)
  --no_recreate         do not (re)populate the schedule table (useful for
                        debugging)
  --recreate_only       populates the schedule table then exit (useful for
                        debugging)
```

## Dev

### Tests

```shell
$ pip install -r requirements-dev.txt
$ python manage.py test
```

### Contribute

Code style is done with pre-commit :
```
$ pip install -r requirements-dev.txt
$ pre-commit install
```

## Changelog

- 2021-06-11 : v0.2.0
  - added `retries`, `retry_delay` options for tasks
  - improve logging

- 2020-11-12 : v0.1.0
  - fixed bug where updating schedule failed
  - fixed worker not doing all available tasks for each tick
  - added --tick argument
  - enforce uniqueness of schedule



            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/olivierdalang/django-toosimple-q",
    "name": "django-toosimple-q",
    "maintainer": "",
    "docs_url": null,
    "requires_python": "",
    "maintainer_email": "",
    "keywords": "django-toosimple-q",
    "author": "Olivier Dalang",
    "author_email": "olivier.dalang@gmail.com",
    "download_url": "https://files.pythonhosted.org/packages/5b/19/626309124c4518e26e487efd98b220b91003a49413187337ec42d0541d68/django-toosimple-q-0.2.0.tar.gz",
    "platform": "",
    "description": "# Django Too Simple Queue\n\n[![PyPI version](https://badge.fury.io/py/django-toosimple-q.svg)](https://pypi.org/project/django-toosimple-q/) ![Workflow](https://github.com/olivierdalang/django-toosimple-q/workflows/ci/badge.svg)\n\nThis packages provides a simplistic task queue and scheduler for Django.\n\nIf execution of your tasks is mission critical, do not use this library, and turn to more complex solutions such as Celery, as this package doesn't guarantee task execution nor unique execution.\n\nIt is geared towards basic apps, where simplicity primes over reliability. The package offers simple decorator syntax, including cron-like schedules.\n\nFeatures :\n\n- no celery/redis/rabbitmq/whatever... just Django !\n- clean decorator syntax to register tasks and schedules\n- simple queuing syntax\n- cron-like scheduling\n- tasks.py autodiscovery\n- django admin integration\n\nLimitations :\n\n- probably not extremely reliable because of race conditions\n- no multithreading yet (but running multiple workers should work)\n\n## Installation\n\nInstall the library :\n```shell\n$ pip install django-toosimple-q\n```\n\nEnable the app in `settings.py` :\n```python\nINSTALLED_APPS = [\n    ...\n    'django_toosimple_q',\n    ...\n]\n```\n\n## Quickstart\n\nTasks need to be registered using the `@register_task()` decorator. Once registered, they can be added to the queue by calling the `.queue()` function.\n\n```python\nfrom django_toosimple_q.decorators import register_task\n\n# Register a task\n@register_task()\ndef my_task(name):\n    return f\"Hello {name} !\"\n\n# Enqueue tasks\nmy_task.queue(\"John\")\nmy_task.queue(\"Peter\")\n```\n\nRegistered tasks can be scheduled from code using this cron-like syntax :\n```python\nfrom django_toosimple_q.decorators import register_task, schedule\n\n# Register and schedule tasks\n@schedule(cron=\"30 8 * * *\", args=['John'])\n@register_task()\ndef morning_routine(name):\n    return f\"Good morning {name} !\"\n```\n\nTo consume the tasks, you need to run at least one worker :\n```shell\n$ python manage.py worker\n```\nThe workers will take care of adding scheduled tasks to the queue when needed, and will execute the tasks.\n\nThe package autoloads `tasks.py` from all installed apps. While this is the recommended place to define your tasks, you can do so from anywhere in your code.\n\n## Advanced usage\n\n### Tasks\n\nYou can optionnaly give a custom name to your tasks. This is required when your task is defined in a local scope.\n```python\n@register_task(\"my_favourite_task\")\ndef my_task(name):\n    return f\"Good morning {name} !\"\n```\n\nYou can set task priorities.\n```python\n@register_task(priority=0)\ndef my_favourite_task(name):\n    return f\"Good bye {name} !\"\n\n@register_task(priority=1)\ndef my_other_task(name):\n    return f\"Hello {name} !\"\n\n# Enqueue tasks\nmy_other_task.queue(\"John\")\nmy_favourite_task.queue(\"Peter\")  # will be executed before the other one\n```\n\nYou can define `retries=N` and `retry_delay=S` to retry the task in case of failure. The delay (in second) will double on each failure.\n\n```python\n@register_task(retries=10, retry_delay=60)\ndef send_email():\n    ...\n```\n\nYou can mark a task as `unique=True` if the task shouldn't be queued again if already queued with the same arguments. This is usefull for tasks such as cleaning or refreshing.\n\n```python\n@register_task(unique=True)\ndef cleanup():\n    ...\n\ncleanup.queue()\ncleanup.queue()  # this will be ignored as long as the first one is still queued\n```\n\nYou can assign tasks to specific queues, and then have your worker only consume tasks from specific queues using `--queue myqueue` or `--exclude_queue myqueue`. By default, workers consume all tasks.\n\n```python\n@register_task(queue='long_running')\ndef long_task():\n    ...\n\n@register_task()\ndef short_task():\n    ...\n\n# Then run those with these workers, so that long\n# running tasks don't prevent short running tasks\n# from being run :\n# manage.py worker --exclude_queue long_running\n# manage.py worker\n```\n\n### Schedules\n\nBy default, `last_check` is set to `now()` on schedule creation. This means they will only run on next cron occurence. If you need your schedules to be run as soon as possible after initialisation, you can specify `last_check=None`.\n\n```python\n@schedule(cron=\"30 8 * * *\", last_check=None)\n@register_task()\ndef my_task(name):\n    return f\"Good morning {name} !\"\n```\n\nBy default, if some crons where missed (e.g. after a server shutdown or if the workers can't keep up with all tasks), the missed tasks will be lost. If you need the tasks to catch up, set `catch_up=True`.\n\n```python\n@schedule(cron=\"30 8 * * *\", catch_up=True)\n@register_task()\ndef my_task(name):\n    ...\n```\n\nYou may define multiple schedules for the same task. In this case, it is mandatory to specify a unique name :\n\n```python\n@schedule(name=\"morning_routine\", cron=\"30 16 * * *\", args=['morning'])\n@schedule(name=\"afternoon_routine\", cron=\"30 8 * * *\", args=['afternoon'])\n@register_task()\ndef my_task(time):\n    return f\"Good {time} John !\"\n```\n\n### Management comment\n\nBesides standard django management commands arguments, the management command supports following arguments.\n\n```\nusage: manage.py worker [--queue QUEUE | --exclude_queue EXCLUDE_QUEUE]\n                        [--tick TICK]\n                        [--once | --until_done]\n                        [--no_recreate | --recreate_only]\n\noptional arguments:\n  --queue QUEUE         which queue to run (can be used several times, all\n                        queues are run if not provided)\n  --exclude_queue EXCLUDE_QUEUE\n                        which queue not to run (can be used several times, all\n                        queues are run if not provided)\n  --tick TICK           frequency in seconds at which the database is checked\n                        for new tasks/schedules\n  --once                run once then exit (useful for debugging)\n  --until_done          run until no tasks are available then exit (useful for\n                        debugging)\n  --no_recreate         do not (re)populate the schedule table (useful for\n                        debugging)\n  --recreate_only       populates the schedule table then exit (useful for\n                        debugging)\n```\n\n## Dev\n\n### Tests\n\n```shell\n$ pip install -r requirements-dev.txt\n$ python manage.py test\n```\n\n### Contribute\n\nCode style is done with pre-commit :\n```\n$ pip install -r requirements-dev.txt\n$ pre-commit install\n```\n\n## Changelog\n\n- 2021-06-11 : v0.2.0\n  - added `retries`, `retry_delay` options for tasks\n  - improve logging\n\n- 2020-11-12 : v0.1.0\n  - fixed bug where updating schedule failed\n  - fixed worker not doing all available tasks for each tick\n  - added --tick argument\n  - enforce uniqueness of schedule\n\n\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "A simplistic task queue and cron-like scheduler for Django",
    "version": "0.2.0",
    "split_keywords": [
        "django-toosimple-q"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "265f280f16cb386908a2fdcb1f33f27c",
                "sha256": "97a1cd141b747d11389ebfce0d2b5c07b739ed5b0e3b78c2e46940dc96eb3435"
            },
            "downloads": -1,
            "filename": "django_toosimple_q-0.2.0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "265f280f16cb386908a2fdcb1f33f27c",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 19225,
            "upload_time": "2021-06-11T08:36:46",
            "upload_time_iso_8601": "2021-06-11T08:36:46.049430Z",
            "url": "https://files.pythonhosted.org/packages/a6/2f/8a534caa72af2dc59ce585db46095978a1c0afbf8513edd4c22205a5e08d/django_toosimple_q-0.2.0-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "0fa32f4a28365241aafb6780fcd05b72",
                "sha256": "f0f9a9dbfb74abdeb1cc0bcb4daac7083e06a22d6ae6c893b6d73e81c125fb43"
            },
            "downloads": -1,
            "filename": "django-toosimple-q-0.2.0.tar.gz",
            "has_sig": false,
            "md5_digest": "0fa32f4a28365241aafb6780fcd05b72",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 15831,
            "upload_time": "2021-06-11T08:36:47",
            "upload_time_iso_8601": "2021-06-11T08:36:47.855260Z",
            "url": "https://files.pythonhosted.org/packages/5b/19/626309124c4518e26e487efd98b220b91003a49413187337ec42d0541d68/django-toosimple-q-0.2.0.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2021-06-11 08:36:47",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "github_user": null,
    "github_project": "olivierdalang",
    "error": "Could not fetch GitHub repository",
    "lcname": "django-toosimple-q"
}
        
Elapsed time: 0.31830s