# ReArq
[](https://pypi.python.org/pypi/rearq)
[](https://github.com/long2ice/rearq)
[](https://github.com/long2ice/rearq/actions?query=workflow:pypi)
[](https://github.com/long2ice/rearq/actions?query=workflow:ci)
## Introduction
ReArq is a distributed task queue with asyncio and redis, which rewrite from [arq](https://github.com/samuelcolvin/arq)
to make improvement and include web interface.
You can try [Demo Online](https://rearq.long2ice.io) here.
## Features
- AsyncIO support, easy integration with [FastAPI](https://github.com/tiangolo/fastapi).
- Delay task, cron task and async task support.
- Full-featured build-in web interface.
- Built-in distributed task lock to make same task only run one at the same time.
- Other powerful features to be discovered.
## Screenshots
## Requirements
- Redis >= 5.0
## Install
Use MySQL backend:
```shell
pip install rearq[mysql]
```
Use PostgreSQL backend:
```shell
pip install rearq[postgres]
```
## Quick Start
### Task Definition
```python
# main.py
from rearq import ReArq
rearq = ReArq(db_url='mysql://root:123456@127.0.0.1:3306/rearq')
@rearq.on_shutdown
async def on_shutdown():
# you can do some clean work here like close db and so on...
print("shutdown")
@rearq.on_startup
async def on_startup():
# you should do some initialization work here
print("startup")
# you must init Tortoise ORM here
await Tortoise.init(
db_url=settings.DB_URL,
modules={"rearq": ["rearq.server.models"]},
)
@rearq.task(queue="q1")
async def add(self, a, b):
return a + b
@rearq.task(cron="*/5 * * * * * *") # run task per 5 seconds
async def timer(self):
return "timer"
```
### Run rearq worker
```shell
> rearq main:rearq worker -q q1 -q q2 # consume tasks from q1 and q2 as the same time
```
```log
2021-03-29 09:54:50.464 | INFO | rearq.worker:_main:95 - Start worker success with queue: rearq:queue:default
2021-03-29 09:54:50.465 | INFO | rearq.worker:_main:96 - Registered tasks: add, sleep, timer_add
2021-03-29 09:54:50.465 | INFO | rearq.worker:log_redis_info:86 - redis_version=6.2.1 mem_usage=1.43M clients_connected=5 db_keys=6
```
### Run rearq timer
If you have timing task or delay task, you should run another command also:
```shell
> rearq main:rearq timer
```
```log
2021-03-29 09:54:43.878 | INFO | rearq.worker:_main:275 - Start timer success
2021-03-29 09:54:43.887 | INFO | rearq.worker:_main:277 - Registered timer tasks: timer_add
2021-03-29 09:54:43.894 | INFO | rearq.worker:log_redis_info:86 - redis_version=6.2.1 mem_usage=1.25M clients_connected=2 db_keys=6
```
Also, you can run timer with worker together by `rearq main:rearq worker -t`.
### Integration in FastAPI
```python
from fastapi import FastAPI
app = FastAPI()
@app.on_event("startup")
async def startup() -> None:
await Tortoise.init(
db_url=settings.DB_URL,
modules={"rearq": ["rearq.server.models"]},
)
@app.on_event("shutdown")
async def shutdown() -> None:
await rearq.close()
# then run task in view
@app.get("/test")
async def test():
job = await add.delay(args=(1, 2))
# or
job = await add.delay(kwargs={"a": 1, "b": 2})
# or
job = await add.delay(1, 2)
# or
job = await add.delay(a=1, b=2)
result = await job.result(timeout=5) # wait result for 5 seconds
print(result.result)
return result
```
## Start web interface
```shell
> rearq main:rearq server
Usage: rearq server [OPTIONS]
Start rest api server.
Options:
--host TEXT Listen host. [default: 0.0.0.0]
-p, --port INTEGER Listen port. [default: 8000]
-h, --help Show this message and exit..
```
After server run, you can visit [https://127.0.0.1:8000/docs](https://127.0.0.1:8000/docs) to see all apis
and [https://127.0.0.1:8000](https://127.0.0.1:8000) to see web interface.
Other options will pass into `uvicorn` directly, such as `--root-path` etc.
```shell
rearq main:rearq server --host 0.0.0.0 --root-path /rearq
```
### Mount as FastAPI sub app
You can also mount rearq server as FastAPI sub app.
```python
from fastapi import FastAPI
from examples.tasks import rearq
from rearq.server.app import app as rearq_app
app = FastAPI()
app.mount("/rearq", rearq_app)
rearq_app.set_rearq(rearq)
```
### Start worker inside app
You can also start worker inside your app.
```python
@app.on_event("startup")
async def startup():
await rearq.init()
await rearq_app.start_worker(with_timer=True, block=False)
```
## ThanksTo
- [arq](https://github.com/samuelcolvin/arq), Fast job queuing and RPC in python with asyncio and redis.
## License
This project is licensed under the [Apache-2.0](./LICENSE) License.
Raw data
{
"_id": null,
"home_page": "https://github.com/long2ice/rearq.git",
"name": "rearq",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8,<4.0",
"maintainer_email": "",
"keywords": "asyncio,task,arq,queue,distributed",
"author": "long2ice",
"author_email": "long2ice@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/e5/db/3296837440cdfc96696e9d754a548dca2d97e65fe0d26c1da8d089646547/rearq-0.2.9.tar.gz",
"platform": null,
"description": "# ReArq\n\n[](https://pypi.python.org/pypi/rearq)\n[](https://github.com/long2ice/rearq)\n[](https://github.com/long2ice/rearq/actions?query=workflow:pypi)\n[](https://github.com/long2ice/rearq/actions?query=workflow:ci)\n\n## Introduction\n\nReArq is a distributed task queue with asyncio and redis, which rewrite from [arq](https://github.com/samuelcolvin/arq)\nto make improvement and include web interface.\n\nYou can try [Demo Online](https://rearq.long2ice.io) here.\n\n## Features\n\n- AsyncIO support, easy integration with [FastAPI](https://github.com/tiangolo/fastapi).\n- Delay task, cron task and async task support.\n- Full-featured build-in web interface.\n- Built-in distributed task lock to make same task only run one at the same time.\n- Other powerful features to be discovered.\n\n## Screenshots\n\n\n\n\n\n\n\n## Requirements\n\n- Redis >= 5.0\n\n## Install\n\nUse MySQL backend:\n\n```shell\npip install rearq[mysql]\n```\n\nUse PostgreSQL backend:\n\n```shell\npip install rearq[postgres]\n```\n\n## Quick Start\n\n### Task Definition\n\n```python\n# main.py\nfrom rearq import ReArq\n\nrearq = ReArq(db_url='mysql://root:123456@127.0.0.1:3306/rearq')\n\n\n@rearq.on_shutdown\nasync def on_shutdown():\n # you can do some clean work here like close db and so on...\n print(\"shutdown\")\n\n\n@rearq.on_startup\nasync def on_startup():\n # you should do some initialization work here\n print(\"startup\")\n # you must init Tortoise ORM here\n await Tortoise.init(\n db_url=settings.DB_URL,\n modules={\"rearq\": [\"rearq.server.models\"]},\n )\n\n\n@rearq.task(queue=\"q1\")\nasync def add(self, a, b):\n return a + b\n\n\n@rearq.task(cron=\"*/5 * * * * * *\") # run task per 5 seconds\nasync def timer(self):\n return \"timer\"\n```\n\n### Run rearq worker\n\n```shell\n> rearq main:rearq worker -q q1 -q q2 # consume tasks from q1 and q2 as the same time\n```\n\n```log\n2021-03-29 09:54:50.464 | INFO | rearq.worker:_main:95 - Start worker success with queue: rearq:queue:default\n2021-03-29 09:54:50.465 | INFO | rearq.worker:_main:96 - Registered tasks: add, sleep, timer_add\n2021-03-29 09:54:50.465 | INFO | rearq.worker:log_redis_info:86 - redis_version=6.2.1 mem_usage=1.43M clients_connected=5 db_keys=6\n```\n\n### Run rearq timer\n\nIf you have timing task or delay task, you should run another command also:\n\n```shell\n> rearq main:rearq timer\n```\n\n```log\n2021-03-29 09:54:43.878 | INFO | rearq.worker:_main:275 - Start timer success\n2021-03-29 09:54:43.887 | INFO | rearq.worker:_main:277 - Registered timer tasks: timer_add\n2021-03-29 09:54:43.894 | INFO | rearq.worker:log_redis_info:86 - redis_version=6.2.1 mem_usage=1.25M clients_connected=2 db_keys=6\n```\n\nAlso, you can run timer with worker together by `rearq main:rearq worker -t`.\n\n### Integration in FastAPI\n\n```python\nfrom fastapi import FastAPI\n\napp = FastAPI()\n\n\n@app.on_event(\"startup\")\nasync def startup() -> None:\n await Tortoise.init(\n db_url=settings.DB_URL,\n modules={\"rearq\": [\"rearq.server.models\"]},\n )\n\n\n@app.on_event(\"shutdown\")\nasync def shutdown() -> None:\n await rearq.close()\n\n\n# then run task in view\n@app.get(\"/test\")\nasync def test():\n job = await add.delay(args=(1, 2))\n # or\n job = await add.delay(kwargs={\"a\": 1, \"b\": 2})\n # or\n job = await add.delay(1, 2)\n # or\n job = await add.delay(a=1, b=2)\n result = await job.result(timeout=5) # wait result for 5 seconds\n print(result.result)\n return result\n```\n\n## Start web interface\n\n```shell\n> rearq main:rearq server\nUsage: rearq server [OPTIONS]\n\n Start rest api server.\n\nOptions:\n --host TEXT Listen host. [default: 0.0.0.0]\n -p, --port INTEGER Listen port. [default: 8000]\n -h, --help Show this message and exit..\n```\n\nAfter server run, you can visit [https://127.0.0.1:8000/docs](https://127.0.0.1:8000/docs) to see all apis\nand [https://127.0.0.1:8000](https://127.0.0.1:8000) to see web interface.\n\nOther options will pass into `uvicorn` directly, such as `--root-path` etc.\n\n```shell\nrearq main:rearq server --host 0.0.0.0 --root-path /rearq\n```\n\n### Mount as FastAPI sub app\n\nYou can also mount rearq server as FastAPI sub app.\n\n```python\n\nfrom fastapi import FastAPI\n\nfrom examples.tasks import rearq\nfrom rearq.server.app import app as rearq_app\n\napp = FastAPI()\n\napp.mount(\"/rearq\", rearq_app)\nrearq_app.set_rearq(rearq)\n```\n\n### Start worker inside app\n\nYou can also start worker inside your app.\n\n```python\n@app.on_event(\"startup\")\nasync def startup():\n await rearq.init()\n await rearq_app.start_worker(with_timer=True, block=False)\n```\n\n## ThanksTo\n\n- [arq](https://github.com/samuelcolvin/arq), Fast job queuing and RPC in python with asyncio and redis.\n\n## License\n\nThis project is licensed under the [Apache-2.0](./LICENSE) License.\n\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "A distributed task queue built with asyncio and redis, with built-in web interface",
"version": "0.2.9",
"project_urls": {
"Documentation": "https://github.com/long2ice/rearq/blob/master/README.md",
"Homepage": "https://github.com/long2ice/rearq.git",
"Repository": "https://github.com/long2ice/rearq.git"
},
"split_keywords": [
"asyncio",
"task",
"arq",
"queue",
"distributed"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c9a2461249d139492b29fbf1a64161be782eda132cf86c5cab58bf273ab63035",
"md5": "d4a0d8e957107e3cb4b980aa83b878e3",
"sha256": "4536c04db1bd366dd2c5ba6fea713e7552b592edaa8ea086b2038c26e46cdfae"
},
"downloads": -1,
"filename": "rearq-0.2.9-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d4a0d8e957107e3cb4b980aa83b878e3",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8,<4.0",
"size": 49133,
"upload_time": "2024-02-20T13:30:12",
"upload_time_iso_8601": "2024-02-20T13:30:12.342180Z",
"url": "https://files.pythonhosted.org/packages/c9/a2/461249d139492b29fbf1a64161be782eda132cf86c5cab58bf273ab63035/rearq-0.2.9-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "e5db3296837440cdfc96696e9d754a548dca2d97e65fe0d26c1da8d089646547",
"md5": "c794271611e1f8ec740c888e495ec8cb",
"sha256": "a7626b8b20243583905b48a998a5b7494f050a95532f89e15cd34996486308f8"
},
"downloads": -1,
"filename": "rearq-0.2.9.tar.gz",
"has_sig": false,
"md5_digest": "c794271611e1f8ec740c888e495ec8cb",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8,<4.0",
"size": 33131,
"upload_time": "2024-02-20T13:30:14",
"upload_time_iso_8601": "2024-02-20T13:30:14.374447Z",
"url": "https://files.pythonhosted.org/packages/e5/db/3296837440cdfc96696e9d754a548dca2d97e65fe0d26c1da8d089646547/rearq-0.2.9.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-20 13:30:14",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "long2ice",
"github_project": "rearq",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "rearq"
}
JSON
