# redispatcher
<a href="https://rafalstapinski.github.io/redispatcher">
<img src="https://rafalstapinski.github.io/redispatcher/img/logo.svg" alt="redispatcher logo" />
</a>
<p align="center">
<strong>
<em>
Dispatch and run distributed work asynchronously, brokered by Redis
</em>
</strong>
</p>
---
**Documentation**: <a href="https://rafalstapinski.github.io/redispatcher">https://rafalstapinski.github.io/redispatcher</a>
**Source Code**: <a href="https://github.com/rafalstapinski/redispatcher">https://github.com/rafalstapinski/redispatcher</a>
---
<p align="center">
<a href="https://github.com/rafalstapinski/redispatcher/actions/workflows/test.yml" target="_blank">
<img src="https://github.com/rafalstapinski/redispatcher/actions/workflows/test.yml/badge.svg" alt="Test Status" />
</a>
<a href="https://pypi.org/project/redispatcher" target="_blank">
<img src="https://img.shields.io/pypi/v/redispatcher?color=%2334D058" alt="pypi" />
</a>
<a href="https://pypi.org/project/redispatcher" target="_blank">
<img src="https://img.shields.io/pypi/pyversions/redispatcher?color=%23334D058" alt="Supported Python Versions: 3.8, 3.9, 3.10" />
</a>
<a href="https://github.com/rafalstapinski/redispatcher/blob/master/LICENSE" target="_blank">
<img src="https://img.shields.io/pypi/l/redispatcher?color=%23334D058" alt="MIT License" />
</a>
</p>
## What is redispatcher
redispatcher allows you to dispatch work that needs to be done in a process separate from your main program loop. This is useful in cases when you need to process some long running work that doesn't necessarily need to be done synchronously within your code. A classic example of this is sending a welcome email to a user as they sign up for your service. It's not necessary to wait for the results of sending an email, and it may take a few seconds to do so. redispatcher lets you fire-and-forget this work (as a message put into a Redis server) and have it be executed in the background, asynchronously, in a separate process (or machine) entirely.
redispatcher comes in two parts:
1. A library that lets you define workers, define strongly typed messages sent to workers, and provides helper functions to facilitate dispatching that work
2. A daemon that runs your workers in a pool, consistently listening for any incoming messages to be processed
## Why use it
There are certainly other solutions for orchestrating distributed workers. redispatcher aims to be super lightweight, very fast and simple to set up (there are many free cloud-hosted Redis solutions available), and has robust type validation and intellisense support.
## Features
* Full intellisense support across your code, despite a distributed workload
* Strongly typed message contract between your publishing code and consumer
* Minimal boilerplate required to setup and start publishing compared than most alternatives
* Minimal performance overhead and completely non-blocking, thanks to `asyncio` (and works with alternatives like `uvloop`)
### Dependencies
* `aioredis` is used under the hood to publish message to and read messages from Redis
* `pydantic` is used to to validate messages conform to your strongly typed contracts
## Installation
Install with `poetry`
```bash
$ poetry add redispatcher
```
or with `pip`
```bash
$ pip install redispatcher
```
## Basic Usage
### Defining your worker
```python
from redispatcher import BaseConsumer
class SendWelcomeEmail(BaseConsumer):
QUEUE = "send-welcome-email"
class Message(BaseConsumer.Message):
email: str
name: str
async def process_message(self, message: Message):
# construct an email and send it to the `message.email` address
```
### Dispatching work
```python
from clients import my_aioredis_client
@app.post("/register")
async def register(...)
...
message = SendWelcomeEmail.Message(email=..., name=..., registered=True)
await SendWelcomeEmail.dispatch(message, my_aioredis_client)
...
```
### Running redispatcher
```python
from redispatcher import Redispatcher, RedispatcherConfig, ConsumerConfig
config = RedispatcherConfig(
redis_dsn="redis://localhost:6379/0",
consumers=[
ConsumerConfig(
consumer_class=SendWelcomeEmail,
count=2
)
]
)
if __name__ == "__main__":
dispatcher = Redispatcher(config)
dispatcher.start()
```
### Contributing
`redispatcher` is already used in production, but is still in its infancy.
If you find a bug, <a href="https://github.com/rafalstapinski/redispatcher/issues/new">open an issue</a> with a detailed description and steps to reproduce.
If you're looking for a feature, <a href="https://github.com/rafalstapinski/redispatcher/issues/new">open an issue</a> with a detailed description and use case. Feel free <a href="https://github.com/rafalstapinski/redispatcher/pulls">open a pull request</a> if you want to contribure directly!
Raw data
{
"_id": null,
"home_page": "https://rafalstapinski.github.io/redispatcher",
"name": "redispatcher",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8,<4.0",
"maintainer_email": "",
"keywords": "redis,queue,broker,pub sub,asynchronous message,celery,dispatcher,worker",
"author": "Rafal Stapinski",
"author_email": "stapinskirafal@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/3c/fe/e18f2820daf2fce82d6f71562ed91f55fbd0713206d1d99b7905f078c7fe/redispatcher-0.3.0.tar.gz",
"platform": null,
"description": "# redispatcher\n\n<a href=\"https://rafalstapinski.github.io/redispatcher\">\n <img src=\"https://rafalstapinski.github.io/redispatcher/img/logo.svg\" alt=\"redispatcher logo\" />\n</a>\n\n<p align=\"center\">\n <strong>\n <em>\n Dispatch and run distributed work asynchronously, brokered by Redis\n </em>\n </strong>\n</p>\n\n---\n\n**Documentation**: <a href=\"https://rafalstapinski.github.io/redispatcher\">https://rafalstapinski.github.io/redispatcher</a>\n\n**Source Code**: <a href=\"https://github.com/rafalstapinski/redispatcher\">https://github.com/rafalstapinski/redispatcher</a>\n\n---\n\n<p align=\"center\">\n <a href=\"https://github.com/rafalstapinski/redispatcher/actions/workflows/test.yml\" target=\"_blank\">\n <img src=\"https://github.com/rafalstapinski/redispatcher/actions/workflows/test.yml/badge.svg\" alt=\"Test Status\" />\n </a>\n <a href=\"https://pypi.org/project/redispatcher\" target=\"_blank\">\n <img src=\"https://img.shields.io/pypi/v/redispatcher?color=%2334D058\" alt=\"pypi\" />\n </a>\n <a href=\"https://pypi.org/project/redispatcher\" target=\"_blank\">\n <img src=\"https://img.shields.io/pypi/pyversions/redispatcher?color=%23334D058\" alt=\"Supported Python Versions: 3.8, 3.9, 3.10\" />\n </a>\n <a href=\"https://github.com/rafalstapinski/redispatcher/blob/master/LICENSE\" target=\"_blank\">\n <img src=\"https://img.shields.io/pypi/l/redispatcher?color=%23334D058\" alt=\"MIT License\" />\n </a>\n</p>\n\n## What is redispatcher\n\nredispatcher allows you to dispatch work that needs to be done in a process separate from your main program loop. This is useful in cases when you need to process some long running work that doesn't necessarily need to be done synchronously within your code. A classic example of this is sending a welcome email to a user as they sign up for your service. It's not necessary to wait for the results of sending an email, and it may take a few seconds to do so. redispatcher lets you fire-and-forget this work (as a message put into a Redis server) and have it be executed in the background, asynchronously, in a separate process (or machine) entirely.\n\nredispatcher comes in two parts:\n1. A library that lets you define workers, define strongly typed messages sent to workers, and provides helper functions to facilitate dispatching that work\n2. A daemon that runs your workers in a pool, consistently listening for any incoming messages to be processed\n\n\n## Why use it\n\nThere are certainly other solutions for orchestrating distributed workers. redispatcher aims to be super lightweight, very fast and simple to set up (there are many free cloud-hosted Redis solutions available), and has robust type validation and intellisense support.\n## Features\n* Full intellisense support across your code, despite a distributed workload\n* Strongly typed message contract between your publishing code and consumer\n* Minimal boilerplate required to setup and start publishing compared than most alternatives\n* Minimal performance overhead and completely non-blocking, thanks to `asyncio` (and works with alternatives like `uvloop`)\n\n### Dependencies\n* `aioredis` is used under the hood to publish message to and read messages from Redis\n* `pydantic` is used to to validate messages conform to your strongly typed contracts\n\n\n## Installation\nInstall with `poetry`\n```bash\n$ poetry add redispatcher\n```\nor with `pip`\n```bash\n$ pip install redispatcher\n```\n## Basic Usage\n### Defining your worker\n```python\nfrom redispatcher import BaseConsumer\n\nclass SendWelcomeEmail(BaseConsumer):\n\n QUEUE = \"send-welcome-email\"\n\n class Message(BaseConsumer.Message):\n email: str\n name: str\n \n async def process_message(self, message: Message):\n # construct an email and send it to the `message.email` address\n```\n\n### Dispatching work\n```python\nfrom clients import my_aioredis_client\n\n@app.post(\"/register\")\nasync def register(...)\n ...\n message = SendWelcomeEmail.Message(email=..., name=..., registered=True)\n await SendWelcomeEmail.dispatch(message, my_aioredis_client)\n ...\n```\n\n### Running redispatcher\n```python\nfrom redispatcher import Redispatcher, RedispatcherConfig, ConsumerConfig\n\nconfig = RedispatcherConfig(\n redis_dsn=\"redis://localhost:6379/0\",\n consumers=[\n ConsumerConfig(\n consumer_class=SendWelcomeEmail,\n count=2\n )\n ]\n)\n\nif __name__ == \"__main__\":\n dispatcher = Redispatcher(config)\n dispatcher.start() \n```\n\n\n### Contributing\n\n`redispatcher` is already used in production, but is still in its infancy.\n\nIf you find a bug, <a href=\"https://github.com/rafalstapinski/redispatcher/issues/new\">open an issue</a> with a detailed description and steps to reproduce.\n\nIf you're looking for a feature, <a href=\"https://github.com/rafalstapinski/redispatcher/issues/new\">open an issue</a> with a detailed description and use case. Feel free <a href=\"https://github.com/rafalstapinski/redispatcher/pulls\">open a pull request</a> if you want to contribure directly!\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Dispatch and run distributed work asynchronously, brokered by Redis",
"version": "0.3.0",
"project_urls": {
"Homepage": "https://rafalstapinski.github.io/redispatcher",
"Repository": "https://github.com/rafalstapinski/redispatcher"
},
"split_keywords": [
"redis",
"queue",
"broker",
"pub sub",
"asynchronous message",
"celery",
"dispatcher",
"worker"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "829a6d72b17734c9e95a00dbaef3a30421c4aec58e13c29b60a43aa4cec6896c",
"md5": "7ec6cd0c662883adb195a701aba0137e",
"sha256": "b9c2165921f184d23355ab01e30e026b7aa1b2a69b50769b7a5920befe073690"
},
"downloads": -1,
"filename": "redispatcher-0.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "7ec6cd0c662883adb195a701aba0137e",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8,<4.0",
"size": 9085,
"upload_time": "2024-02-22T19:11:35",
"upload_time_iso_8601": "2024-02-22T19:11:35.699833Z",
"url": "https://files.pythonhosted.org/packages/82/9a/6d72b17734c9e95a00dbaef3a30421c4aec58e13c29b60a43aa4cec6896c/redispatcher-0.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3cfee18f2820daf2fce82d6f71562ed91f55fbd0713206d1d99b7905f078c7fe",
"md5": "958ea7ef8692b19281207a0665e92b8a",
"sha256": "95771fd9812498e25c7bd02dea4562a3f2f50e25052e12c0163cf16695a960e5"
},
"downloads": -1,
"filename": "redispatcher-0.3.0.tar.gz",
"has_sig": false,
"md5_digest": "958ea7ef8692b19281207a0665e92b8a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8,<4.0",
"size": 8728,
"upload_time": "2024-02-22T19:11:37",
"upload_time_iso_8601": "2024-02-22T19:11:37.198238Z",
"url": "https://files.pythonhosted.org/packages/3c/fe/e18f2820daf2fce82d6f71562ed91f55fbd0713206d1d99b7905f078c7fe/redispatcher-0.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-22 19:11:37",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "rafalstapinski",
"github_project": "redispatcher",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "redispatcher"
}
JSON
