# redis-tq
redis-tq is a [Redis-based][redis] multi-producer, multi-consumer Queue.
Allows for sharing data between multiple processes or hosts.
Tasks support a "lease time". After that time other workers may consider this
client to have crashed or stalled and pick up the item instead. The number of
retries can be configured as well.
Based on [this example][source] but with many improvements added.
[source]: http://peter-hoffmann.com/2012/python-simple-queue-redis-queue.html
[redis]: https://redis.io/
## Installing
redis-tq is available on [PyPI][] so you can simply install via:
```sh
$ pip install redis-tq
```
[PyPI]: https://pypi.org/project/redis-tq/
## How to use
On the producing side, populate the queue with tasks and a respective lease
timeout:
```python
from redistq import TaskQueue
tq = TaskQueue('localhost', 'myqueue')
for i in range(10):
tq.add(some task, lease_timeout, ttl=3)
```
On the consuming side:
```python
from redistq import TaskQueue
tq = TaskQueue('localhost', 'myqueue')
while True:
task, task_id = tq.get()
if task is not None:
# do something with task and mark it as complete afterwards
tq.complete(task_id)
if tq.is_empty():
break
# tq.get is non-blocking, so you may want to sleep a
# bit before the next iteration
time.sleep(1)
```
If the consumer crashes (i.e. the task is not marked as completed after
`lease_timeout` seconds), the task will be put back into the task queue. This
rescheduling will happen at most `ttl` times and then the task will be
dropped. A callback can be provided if you want to monitor such cases.
## Running the tests
The tests will check the presence of a Redis instance on localhost, you can
use
docker run --rm -d -p 6379:6379 redis
to get one. Then use `make test`, it will take care of creating an appropriate
virtualenv and use it to run the tests.
Raw data
{
"_id": null,
"home_page": "https://github.com/flix-tech/redis-tq",
"name": "redis-tq",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "redis task queue",
"author": "FlixTech",
"author_email": "open-source@flixbus.com",
"download_url": "https://files.pythonhosted.org/packages/e8/ac/6fd24a63cf82b823449d08abf3aae0885c8814f279ce02def78bfab2f077/redis-tq-0.0.11.tar.gz",
"platform": null,
"description": "# redis-tq\n\nredis-tq is a [Redis-based][redis] multi-producer, multi-consumer Queue.\nAllows for sharing data between multiple processes or hosts.\n\nTasks support a \"lease time\". After that time other workers may consider this\nclient to have crashed or stalled and pick up the item instead. The number of\nretries can be configured as well.\n\nBased on [this example][source] but with many improvements added.\n\n[source]: http://peter-hoffmann.com/2012/python-simple-queue-redis-queue.html\n[redis]: https://redis.io/\n\n\n## Installing\n\nredis-tq is available on [PyPI][] so you can simply install via:\n\n```sh\n$ pip install redis-tq\n```\n\n[PyPI]: https://pypi.org/project/redis-tq/\n\n\n## How to use\n\nOn the producing side, populate the queue with tasks and a respective lease\ntimeout:\n\n```python\nfrom redistq import TaskQueue\n\ntq = TaskQueue('localhost', 'myqueue')\nfor i in range(10):\n tq.add(some task, lease_timeout, ttl=3)\n```\n\nOn the consuming side:\n\n```python\nfrom redistq import TaskQueue\n\ntq = TaskQueue('localhost', 'myqueue')\nwhile True:\n task, task_id = tq.get()\n if task is not None:\n # do something with task and mark it as complete afterwards\n tq.complete(task_id)\n if tq.is_empty():\n break\n # tq.get is non-blocking, so you may want to sleep a\n # bit before the next iteration\n time.sleep(1)\n```\n\nIf the consumer crashes (i.e. the task is not marked as completed after\n`lease_timeout` seconds), the task will be put back into the task queue. This\nrescheduling will happen at most `ttl` times and then the task will be\ndropped. A callback can be provided if you want to monitor such cases.\n\n\n## Running the tests\n\nThe tests will check the presence of a Redis instance on localhost, you can\nuse\n\n docker run --rm -d -p 6379:6379 redis\n\nto get one. Then use `make test`, it will take care of creating an appropriate\nvirtualenv and use it to run the tests.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Redis Based Task Queue",
"version": "0.0.11",
"split_keywords": [
"redis",
"task",
"queue"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4c919584dfb834de88376025417f58753fdb8cedfd580b50ecb69a61982bbe55",
"md5": "1693dbab2deb6e27d2a3a833b609fcba",
"sha256": "4815d3bcfec709821f3e0cc8a908a9bec2934c22b23ca92cd1090bb4fdfcf41a"
},
"downloads": -1,
"filename": "redis_tq-0.0.11-py3-none-any.whl",
"has_sig": false,
"md5_digest": "1693dbab2deb6e27d2a3a833b609fcba",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 8352,
"upload_time": "2023-01-25T10:03:21",
"upload_time_iso_8601": "2023-01-25T10:03:21.293979Z",
"url": "https://files.pythonhosted.org/packages/4c/91/9584dfb834de88376025417f58753fdb8cedfd580b50ecb69a61982bbe55/redis_tq-0.0.11-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "e8ac6fd24a63cf82b823449d08abf3aae0885c8814f279ce02def78bfab2f077",
"md5": "df5b0ec36841909540179b6e6a1c75eb",
"sha256": "13fdb4d8679c58383698e9d152bdcc29b8e00a89acf2578e3d36b84ff0bf7b37"
},
"downloads": -1,
"filename": "redis-tq-0.0.11.tar.gz",
"has_sig": false,
"md5_digest": "df5b0ec36841909540179b6e6a1c75eb",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 7934,
"upload_time": "2023-01-25T10:03:22",
"upload_time_iso_8601": "2023-01-25T10:03:22.773368Z",
"url": "https://files.pythonhosted.org/packages/e8/ac/6fd24a63cf82b823449d08abf3aae0885c8814f279ce02def78bfab2f077/redis-tq-0.0.11.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-01-25 10:03:22",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "flix-tech",
"github_project": "redis-tq",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "redis",
"specs": [
[
"==",
"4.4.2"
]
]
}
],
"lcname": "redis-tq"
}
JSON
