.. image:: https://img.shields.io/pypi/l/Torrelque.svg
:target: https://spdx.org/licenses/LGPL-3.0-only.html
:alt: PyPI - License
.. image:: https://heptapod.host/saajns/torrelque/badges/branch/default/pipeline.svg
:target: https://heptapod.host/saajns/torrelque/-/commits/branch/default
:alt: Pipeline status
.. image:: https://heptapod.host/saajns/torrelque/badges/branch/default/coverage.svg
:target: https://heptapod.host/saajns/torrelque/-/commits/branch/default
:alt: Test code coverage
.. image:: https://badge.fury.io/py/Torrelque.svg
:target: https://pypi.python.org/pypi/Torrelque
:alt: PyPI
.. image:: https://readthedocs.org/projects/torrelque/badge/?version=latest
:target: https://torrelque.readthedocs.io/en/latest/?badge=latest
:alt: RTFD
*********
Torrelque
*********
Torrelque is a Python package that provides minimal asynchronous reliable
distributed Redis-backed (or a protocol-compatible alternative) work queues.
It is built:
1. Lock-free. It relies on Redis transactions and its single-threaded
execution model.
2. Poll-free. Waiting subset of the Python API relies either on blocking Redis
commands or notifications.
3. Bulk-friendly. Tasks can be produced and consumed in bulk.
4. Introspectable. Task stats, task status transition watching API, and
the data model comprehensible directly in Redis.
Supported Redis server implementations: Redis, KeyDB.
Install
=======
::
pip install Torrelque
Quickstart
==========
Producer:
.. sourcecode:: python
import redis.asyncio
import torrelque
client = redis.asyncio.Redis()
queue = torrelque.Torrelque(client, queue='email')
queue.schedule_sweep() # to make due requeued tasks available again
task_data = {'addr': 'joe@doe.com', 'subj': 'hello', 'body': '...'}
task_id = await queue.enqueue(task_data)
print('Email task enqueued', task_id)
Consumer:
.. sourcecode:: python
import redis.asyncio
import torrelque
client = redis.asyncio.Redis()
queue = torrelque.Torrelque(client, queue='email')
while True:
task_id, task_data = await queue.dequeue()
try:
await some_email_client.send(**task_data)
except Exception:
print('Email sending error, retrying in 30s', task_id)
await queue.requeue(task_id, delay=30)
else:
print('Email sent', task_id)
await queue.release(task_id)
Example list
============
- `Producer-consumer <e1_>`_. Infinite producing and consuming loops.
- `Batch processing <e2_>`_. Finite number of tasks, consumers stop with a
poison pill, bulk enqueue. This example can be used as a synthetic benchmark.
Because there's no IO-bound workload, it'll be CPU-bound which isn't normal
mode of operation for an asynchronous application. But it can be used to
compare between CPython, PyPy and concurrency parameters.
- `Web application background task <e3_>`_. This tornado application allows
to start a task and push server-sent events (SSE) to UI about its status. UI
starts a task and waits for it to complete. When a task fails it's re-queued
with exponential back-off.
.. _e1: https://heptapod.host/saajns/torrelque/blob/branch/default/example/producer_consumer.py
.. _e2: https://heptapod.host/saajns/torrelque/blob/branch/default/example/batch_processing.py
.. _e3: https://heptapod.host/saajns/torrelque/blob/branch/default/example/wait_until_complete.py
Raw data
{
"_id": null,
"home_page": "https://heptapod.host/saajns/torrelque",
"name": "Torrelque",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3",
"maintainer_email": null,
"keywords": "python redis asynchronous job-queue work-queue",
"author": "saaj",
"author_email": "mail@saaj.me",
"download_url": "https://files.pythonhosted.org/packages/2d/55/a693144625632521b8faebc4545d39f6bfe3a0cd7cba31c3aeb110cdc2a3/Torrelque-0.7.1.tar.gz",
"platform": "Any",
"description": ".. image:: https://img.shields.io/pypi/l/Torrelque.svg\n :target: https://spdx.org/licenses/LGPL-3.0-only.html\n :alt: PyPI - License\n.. image:: https://heptapod.host/saajns/torrelque/badges/branch/default/pipeline.svg\n :target: https://heptapod.host/saajns/torrelque/-/commits/branch/default\n :alt: Pipeline status\n.. image:: https://heptapod.host/saajns/torrelque/badges/branch/default/coverage.svg\n :target: https://heptapod.host/saajns/torrelque/-/commits/branch/default\n :alt: Test code coverage\n.. image:: https://badge.fury.io/py/Torrelque.svg\n :target: https://pypi.python.org/pypi/Torrelque\n :alt: PyPI\n.. image:: https://readthedocs.org/projects/torrelque/badge/?version=latest\n :target: https://torrelque.readthedocs.io/en/latest/?badge=latest\n :alt: RTFD\n\n*********\nTorrelque\n*********\nTorrelque is a Python package that provides minimal asynchronous reliable\ndistributed Redis-backed (or a protocol-compatible alternative) work queues.\nIt is built:\n\n1. Lock-free. It relies on Redis transactions and its single-threaded\n execution model.\n2. Poll-free. Waiting subset of the Python API relies either on blocking Redis\n commands or notifications.\n3. Bulk-friendly. Tasks can be produced and consumed in bulk.\n4. Introspectable. Task stats, task status transition watching API, and\n the data model comprehensible directly in Redis.\n\nSupported Redis server implementations: Redis, KeyDB.\n\nInstall\n=======\n::\n\n pip install Torrelque\n\nQuickstart\n==========\nProducer:\n\n.. sourcecode:: python\n\n import redis.asyncio\n import torrelque\n\n client = redis.asyncio.Redis()\n queue = torrelque.Torrelque(client, queue='email')\n queue.schedule_sweep() # to make due requeued tasks available again\n\n task_data = {'addr': 'joe@doe.com', 'subj': 'hello', 'body': '...'}\n task_id = await queue.enqueue(task_data)\n print('Email task enqueued', task_id)\n\nConsumer:\n\n.. sourcecode:: python\n\n import redis.asyncio\n import torrelque\n\n client = redis.asyncio.Redis()\n queue = torrelque.Torrelque(client, queue='email')\n\n while True:\n task_id, task_data = await queue.dequeue()\n try:\n await some_email_client.send(**task_data)\n except Exception:\n print('Email sending error, retrying in 30s', task_id)\n await queue.requeue(task_id, delay=30)\n else:\n print('Email sent', task_id)\n await queue.release(task_id)\n\nExample list\n============\n- `Producer-consumer <e1_>`_. Infinite producing and consuming loops.\n- `Batch processing <e2_>`_. Finite number of tasks, consumers stop with a\n poison pill, bulk enqueue. This example can be used as a synthetic benchmark.\n Because there's no IO-bound workload, it'll be CPU-bound which isn't normal\n mode of operation for an asynchronous application. But it can be used to\n compare between CPython, PyPy and concurrency parameters.\n- `Web application background task <e3_>`_. This tornado application allows\n to start a task and push server-sent events (SSE) to UI about its status. UI\n starts a task and waits for it to complete. When a task fails it's re-queued\n with exponential back-off.\n\n\n.. _e1: https://heptapod.host/saajns/torrelque/blob/branch/default/example/producer_consumer.py\n.. _e2: https://heptapod.host/saajns/torrelque/blob/branch/default/example/batch_processing.py\n.. _e3: https://heptapod.host/saajns/torrelque/blob/branch/default/example/wait_until_complete.py\n",
"bugtrack_url": null,
"license": "LGPL-3.0-only",
"summary": "Asynchronous Redis-backed reliable queue package",
"version": "0.7.1",
"project_urls": {
"Documentation": "https://torrelque.readthedocs.io/",
"Homepage": "https://heptapod.host/saajns/torrelque",
"Release Notes": "https://torrelque.readthedocs.io/en/latest/history.html",
"Source Code": "https://heptapod.host/saajns/torrelque"
},
"split_keywords": [
"python",
"redis",
"asynchronous",
"job-queue",
"work-queue"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2d55a693144625632521b8faebc4545d39f6bfe3a0cd7cba31c3aeb110cdc2a3",
"md5": "ada76d5441ca560b1f32794318a238f6",
"sha256": "43f6957f4bf087b6dc6bdc133bcf1bdce5c68fe854e8700c885cc24edca7fd97"
},
"downloads": -1,
"filename": "Torrelque-0.7.1.tar.gz",
"has_sig": false,
"md5_digest": "ada76d5441ca560b1f32794318a238f6",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3",
"size": 14802,
"upload_time": "2024-04-05T13:25:39",
"upload_time_iso_8601": "2024-04-05T13:25:39.215811Z",
"url": "https://files.pythonhosted.org/packages/2d/55/a693144625632521b8faebc4545d39f6bfe3a0cd7cba31c3aeb110cdc2a3/Torrelque-0.7.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-05 13:25:39",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "torrelque"
}
JSON