.. image:: docs/_static/logo.png
:align: center
:alt: Q logo
:target: https://django-q.readthedocs.org/
A multiprocessing distributed task queue for Django
---------------------------------------------------
|image0| |image1| |docs| |image2|
Features
~~~~~~~~
- Multiprocessing worker pool
- Asynchronous tasks
- Scheduled, cron and repeated tasks
- Signed and compressed packages
- Failure and success database or cache
- Result hooks, groups and chains
- Django Admin integration
- PaaS compatible with multiple instances
- Multi cluster monitor
- Redis, Disque, IronMQ, SQS, MongoDB or ORM
- Rollbar and Sentry support
Requirements
~~~~~~~~~~~~
- `Django <https://www.djangoproject.com>`__ > = 2.2
- `Django-picklefield <https://github.com/gintas/django-picklefield>`__
- `Arrow <https://github.com/crsmithdev/arrow>`__
- `Blessed <https://github.com/jquast/blessed>`__
Tested with: Python 3.7, 3.8, 3.9 Django 2.2.X and 3.2.X
.. warning:: Since Python 3.7 `async` became a reserved keyword and was refactored to `async_task`
Brokers
~~~~~~~
- `Redis <https://django-q.readthedocs.org/en/latest/brokers.html#redis>`__
- `Disque <https://django-q.readthedocs.org/en/latest/brokers.html#disque>`__
- `IronMQ <https://django-q.readthedocs.org/en/latest/brokers.html#ironmq>`__
- `Amazon SQS <https://django-q.readthedocs.org/en/latest/brokers.html#amazon-sqs>`__
- `MongoDB <https://django-q.readthedocs.org/en/latest/brokers.html#mongodb>`__
- `Django ORM <https://django-q.readthedocs.org/en/latest/brokers.html#django-orm>`__
Installation
~~~~~~~~~~~~
- Install the latest version with pip::
$ pip install django-q
- Add `django_q` to your `INSTALLED_APPS` in your projects `settings.py`::
INSTALLED_APPS = (
# other apps
'django_q',
)
- Run Django migrations to create the database tables::
$ python manage.py migrate
- Choose a message `broker <https://django-q.readthedocs.org/en/latest/brokers.html>`__ , configure and install the appropriate client library.
Read the full documentation at `https://django-q.readthedocs.org <https://django-q.readthedocs.org>`__
Configuration
~~~~~~~~~~~~~
All configuration settings are optional. e.g:
.. code:: python
# settings.py example
Q_CLUSTER = {
'name': 'myproject',
'workers': 8,
'recycle': 500,
'timeout': 60,
'compress': True,
'cpu_affinity': 1,
'save_limit': 250,
'queue_limit': 500,
'label': 'Django Q',
'redis': {
'host': '127.0.0.1',
'port': 6379,
'db': 0, }
}
For full configuration options, see the `configuration documentation <https://django-q.readthedocs.org/en/latest/configure.html>`__.
Management Commands
~~~~~~~~~~~~~~~~~~~
Start a cluster with::
$ python manage.py qcluster
Monitor your clusters with::
$ python manage.py qmonitor
Monitor your clusters' memory usage with::
$ python manage.py qmemory
Check overall statistics with::
$ python manage.py qinfo
Creating Tasks
~~~~~~~~~~~~~~
Use `async_task` from your code to quickly offload tasks:
.. code:: python
from django_q.tasks import async_task, result
# create the task
async_task('math.copysign', 2, -2)
# or with a reference
import math.copysign
task_id = async_task(copysign, 2, -2)
# get the result
task_result = result(task_id)
# result returns None if the task has not been executed yet
# you can wait for it
task_result = result(task_id, 200)
# but in most cases you will want to use a hook:
async_task('math.modf', 2.5, hook='hooks.print_result')
# hooks.py
def print_result(task):
print(task.result)
For more info see `Tasks <https://django-q.readthedocs.org/en/latest/tasks.html>`__
Schedule
~~~~~~~~
Schedules are regular Django models. You can manage them through the
Admin page or directly from your code:
.. code:: python
# Use the schedule function
from django_q.tasks import schedule
schedule('math.copysign',
2, -2,
hook='hooks.print_result',
schedule_type=Schedule.DAILY)
# Or create the object directly
from django_q.models import Schedule
Schedule.objects.create(func='math.copysign',
hook='hooks.print_result',
args='2,-2',
schedule_type=Schedule.DAILY
)
# Run a task every 5 minutes, starting at 6 today
# for 2 hours
import arrow
schedule('math.hypot',
3, 4,
schedule_type=Schedule.MINUTES,
minutes=5,
repeats=24,
next_run=arrow.utcnow().replace(hour=18, minute=0))
# Use a cron expression
schedule('math.hypot',
3, 4,
schedule_type=Schedule.CRON,
cron = '0 22 * * 1-5')
For more info check the `Schedules <https://django-q.readthedocs.org/en/latest/schedules.html>`__ documentation.
Testing
~~~~~~~
To run the tests you will need the following in addition to install requirements:
* `py.test <http://pytest.org/latest/>`__
* `pytest-django <https://github.com/pytest-dev/pytest-django>`__
* Disque from https://github.com/antirez/disque.git
* Redis
* MongoDB
Or you can use the included Docker Compose file.
The following commands can be used to run the tests:
.. code:: bash
# Create virtual environment
python -m venv venv
# Install requirements
venv/bin/pip install -r requirements.txt
# Install test dependencies
venv/bin/pip install pytest pytest-django
# Install django-q
venv/bin/python setup.py develop
# Run required services (you need to have docker-compose installed)
docker-compose -f test-services-docker-compose.yaml up -d
# Run tests
venv/bin/pytest
# Stop the services required by tests (when you no longer plan to run tests)
docker-compose -f test-services-docker-compose.yaml down
Locale
~~~~~~
Currently available in English, German and French.
Translation pull requests are always welcome.
Todo
~~~~
- Better tests and coverage
- Less dependencies?
Acknowledgements
~~~~~~~~~~~~~~~~
- Django Q was inspired by working with
`Django-RQ <https://github.com/ui/django-rq>`__ and
`RQ <https://github.com/ui/django-rq>`__
- Human readable hashes by
`HumanHash <https://github.com/zacharyvoase/humanhash>`__
- Redditors feedback at `r/django <https://www.reddit.com/r/django/>`__
- JetBrains for their `Open Source Support Program <https://www.jetbrains.com/community/opensource>`__
.. |image0| image:: https://github.com/koed00/django-q/workflows/Tests/badge.svg?branche=master
:target: https://github.com/Koed00/django-q/actions?query=workflow%3Atests
.. |image1| image:: http://codecov.io/github/Koed00/django-q/coverage.svg?branch=master
:target: http://codecov.io/github/Koed00/django-q?branch=master
.. |image2| image:: http://badges.gitter.im/Join%20Chat.svg
:target: https://gitter.im/Koed00/django-q
.. |docs| image:: https://readthedocs.org/projects/docs/badge/?version=latest
:alt: Documentation Status
:scale: 100
:target: https://django-q.readthedocs.org/
Raw data
{
"_id": null,
"home_page": "https://django-q.readthedocs.org",
"name": "django-q",
"maintainer": "Ilan Steemers",
"docs_url": null,
"requires_python": ">=3.6.2,<4",
"maintainer_email": "koed00@gmail.com",
"keywords": "django,distributed,multiprocessing,queue,scheduler",
"author": "Ilan Steemers",
"author_email": "koed00@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/3f/49/d54c6631bce6fea588ad31cd4f1b7538a4f5e6e7b85a5187c5de72c9fa75/django-q-1.3.9.tar.gz",
"platform": "",
"description": ".. image:: docs/_static/logo.png\n :align: center\n :alt: Q logo\n :target: https://django-q.readthedocs.org/\n\nA multiprocessing distributed task queue for Django\n---------------------------------------------------\n\n|image0| |image1| |docs| |image2|\n\nFeatures\n~~~~~~~~\n\n- Multiprocessing worker pool\n- Asynchronous tasks\n- Scheduled, cron and repeated tasks\n- Signed and compressed packages\n- Failure and success database or cache\n- Result hooks, groups and chains\n- Django Admin integration\n- PaaS compatible with multiple instances\n- Multi cluster monitor\n- Redis, Disque, IronMQ, SQS, MongoDB or ORM\n- Rollbar and Sentry support\n\nRequirements\n~~~~~~~~~~~~\n\n- `Django <https://www.djangoproject.com>`__ > = 2.2\n- `Django-picklefield <https://github.com/gintas/django-picklefield>`__\n- `Arrow <https://github.com/crsmithdev/arrow>`__\n- `Blessed <https://github.com/jquast/blessed>`__\n\nTested with: Python 3.7, 3.8, 3.9 Django 2.2.X and 3.2.X\n\n.. warning:: Since Python 3.7 `async` became a reserved keyword and was refactored to `async_task`\n\nBrokers\n~~~~~~~\n- `Redis <https://django-q.readthedocs.org/en/latest/brokers.html#redis>`__\n- `Disque <https://django-q.readthedocs.org/en/latest/brokers.html#disque>`__\n- `IronMQ <https://django-q.readthedocs.org/en/latest/brokers.html#ironmq>`__\n- `Amazon SQS <https://django-q.readthedocs.org/en/latest/brokers.html#amazon-sqs>`__\n- `MongoDB <https://django-q.readthedocs.org/en/latest/brokers.html#mongodb>`__\n- `Django ORM <https://django-q.readthedocs.org/en/latest/brokers.html#django-orm>`__\n\nInstallation\n~~~~~~~~~~~~\n\n- Install the latest version with pip::\n\n $ pip install django-q\n\n\n- Add `django_q` to your `INSTALLED_APPS` in your projects `settings.py`::\n\n INSTALLED_APPS = (\n # other apps\n 'django_q',\n )\n\n- Run Django migrations to create the database tables::\n\n $ python manage.py migrate\n\n- Choose a message `broker <https://django-q.readthedocs.org/en/latest/brokers.html>`__ , configure and install the appropriate client library.\n\nRead the full documentation at `https://django-q.readthedocs.org <https://django-q.readthedocs.org>`__\n\n\nConfiguration\n~~~~~~~~~~~~~\n\nAll configuration settings are optional. e.g:\n\n.. code:: python\n\n # settings.py example\n Q_CLUSTER = {\n 'name': 'myproject',\n 'workers': 8,\n 'recycle': 500,\n 'timeout': 60,\n 'compress': True,\n 'cpu_affinity': 1,\n 'save_limit': 250,\n 'queue_limit': 500,\n 'label': 'Django Q',\n 'redis': {\n 'host': '127.0.0.1',\n 'port': 6379,\n 'db': 0, }\n }\n\nFor full configuration options, see the `configuration documentation <https://django-q.readthedocs.org/en/latest/configure.html>`__.\n\nManagement Commands\n~~~~~~~~~~~~~~~~~~~\n\nStart a cluster with::\n\n $ python manage.py qcluster\n\nMonitor your clusters with::\n\n $ python manage.py qmonitor\n\nMonitor your clusters' memory usage with::\n\n $ python manage.py qmemory\n\nCheck overall statistics with::\n\n $ python manage.py qinfo\n\nCreating Tasks\n~~~~~~~~~~~~~~\n\nUse `async_task` from your code to quickly offload tasks:\n\n.. code:: python\n\n from django_q.tasks import async_task, result\n\n # create the task\n async_task('math.copysign', 2, -2)\n\n # or with a reference\n import math.copysign\n\n task_id = async_task(copysign, 2, -2)\n\n # get the result\n task_result = result(task_id)\n\n # result returns None if the task has not been executed yet\n # you can wait for it\n task_result = result(task_id, 200)\n\n # but in most cases you will want to use a hook:\n\n async_task('math.modf', 2.5, hook='hooks.print_result')\n\n # hooks.py\n def print_result(task):\n print(task.result)\n\nFor more info see `Tasks <https://django-q.readthedocs.org/en/latest/tasks.html>`__\n\n\nSchedule\n~~~~~~~~\n\nSchedules are regular Django models. You can manage them through the\nAdmin page or directly from your code:\n\n.. code:: python\n\n # Use the schedule function\n from django_q.tasks import schedule\n\n schedule('math.copysign',\n 2, -2,\n hook='hooks.print_result',\n schedule_type=Schedule.DAILY)\n\n # Or create the object directly\n from django_q.models import Schedule\n\n Schedule.objects.create(func='math.copysign',\n hook='hooks.print_result',\n args='2,-2',\n schedule_type=Schedule.DAILY\n )\n\n # Run a task every 5 minutes, starting at 6 today\n # for 2 hours\n import arrow\n\n schedule('math.hypot',\n 3, 4,\n schedule_type=Schedule.MINUTES,\n minutes=5,\n repeats=24,\n next_run=arrow.utcnow().replace(hour=18, minute=0))\n\n # Use a cron expression\n schedule('math.hypot',\n 3, 4,\n schedule_type=Schedule.CRON,\n cron = '0 22 * * 1-5')\n\nFor more info check the `Schedules <https://django-q.readthedocs.org/en/latest/schedules.html>`__ documentation.\n\n\nTesting\n~~~~~~~\n\nTo run the tests you will need the following in addition to install requirements:\n\n* `py.test <http://pytest.org/latest/>`__\n* `pytest-django <https://github.com/pytest-dev/pytest-django>`__\n* Disque from https://github.com/antirez/disque.git\n* Redis\n* MongoDB\n\nOr you can use the included Docker Compose file.\n\nThe following commands can be used to run the tests:\n\n.. code:: bash\n\n # Create virtual environment\n python -m venv venv\n\n # Install requirements\n venv/bin/pip install -r requirements.txt\n\n # Install test dependencies\n venv/bin/pip install pytest pytest-django\n\n # Install django-q\n venv/bin/python setup.py develop\n\n # Run required services (you need to have docker-compose installed)\n docker-compose -f test-services-docker-compose.yaml up -d\n\n # Run tests\n venv/bin/pytest\n\n # Stop the services required by tests (when you no longer plan to run tests)\n docker-compose -f test-services-docker-compose.yaml down\n\nLocale\n~~~~~~\n\nCurrently available in English, German and French.\nTranslation pull requests are always welcome.\n\nTodo\n~~~~\n\n- Better tests and coverage\n- Less dependencies?\n\nAcknowledgements\n~~~~~~~~~~~~~~~~\n\n- Django Q was inspired by working with\n `Django-RQ <https://github.com/ui/django-rq>`__ and\n `RQ <https://github.com/ui/django-rq>`__\n- Human readable hashes by\n `HumanHash <https://github.com/zacharyvoase/humanhash>`__\n- Redditors feedback at `r/django <https://www.reddit.com/r/django/>`__\n\n- JetBrains for their `Open Source Support Program <https://www.jetbrains.com/community/opensource>`__\n\n.. |image0| image:: https://github.com/koed00/django-q/workflows/Tests/badge.svg?branche=master\n :target: https://github.com/Koed00/django-q/actions?query=workflow%3Atests\n.. |image1| image:: http://codecov.io/github/Koed00/django-q/coverage.svg?branch=master\n :target: http://codecov.io/github/Koed00/django-q?branch=master\n.. |image2| image:: http://badges.gitter.im/Join%20Chat.svg\n :target: https://gitter.im/Koed00/django-q\n.. |docs| image:: https://readthedocs.org/projects/docs/badge/?version=latest\n :alt: Documentation Status\n :scale: 100\n :target: https://django-q.readthedocs.org/\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A multiprocessing distributed task queue for Django",
"version": "1.3.9",
"split_keywords": [
"django",
"distributed",
"multiprocessing",
"queue",
"scheduler"
],
"urls": [
{
"comment_text": "",
"digests": {
"md5": "dcb19a11f1d89c13e40a9268b8cf6bd2",
"sha256": "1b74ce3a8931990b136903e3a7bc9b07243282a2b5355117246f05ed5d076e68"
},
"downloads": -1,
"filename": "django_q-1.3.9-py3-none-any.whl",
"has_sig": false,
"md5_digest": "dcb19a11f1d89c13e40a9268b8cf6bd2",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6.2,<4",
"size": 89959,
"upload_time": "2021-06-27T12:04:53",
"upload_time_iso_8601": "2021-06-27T12:04:53.831233Z",
"url": "https://files.pythonhosted.org/packages/bf/4c/150dcb7bb9e5e0c67da2b1773823f44378f1160973cef3fe2923b840aa72/django_q-1.3.9-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"md5": "75def6d4d98612e4fc1245c18a82e65a",
"sha256": "5c6b4d530aa3aabf9c6aa57376da1ca2abf89a1562b77038b7a04e52a4a0a91b"
},
"downloads": -1,
"filename": "django-q-1.3.9.tar.gz",
"has_sig": false,
"md5_digest": "75def6d4d98612e4fc1245c18a82e65a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6.2,<4",
"size": 72104,
"upload_time": "2021-06-27T12:04:51",
"upload_time_iso_8601": "2021-06-27T12:04:51.131353Z",
"url": "https://files.pythonhosted.org/packages/3f/49/d54c6631bce6fea588ad31cd4f1b7538a4f5e6e7b85a5187c5de72c9fa75/django-q-1.3.9.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2021-06-27 12:04:51",
"github": false,
"gitlab": false,
"bitbucket": false,
"lcname": "django-q"
}
JSON