| Name | celery-callback-service JSON |
| Version |
0.1.8
JSON |
| download |
| home_page | None |
| Summary | 基于celery的回调服务。业务系统创建celery回调任务，celery-callback-worker执行回调任务，业务系统在回调任务中处理异步任务。 |
| upload_time | 2024-08-17 01:29:44 |
| maintainer | Sun XiaoWei |
| docs_url | None |
| author | Sun XiaoWei |
| requires_python | None |
| license | MIT |
| keywords |
celery
callback
service
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# celery-callback-service
基于celery的回调服务。业务系统创建celery回调任务，celery-callback-worker执行回调任务，业务系统在回调任务中处理异步任务。
## 使用方法
### 启动`celery-callback-worker`（celery worker）
*celeryconfig.py*
```python
worker_concurrency = 10
worker_pool = "threads"
broker_url = "redis://redis/0"
result_backend = "redis://redis/1"
accept_content = ["application/json"]
task_serializer = "json"
result_accept_content = ["application/json"]
result_serializer = "json"
timezone = "Asia/Shanghai"
broker_connection_retry_on_startup = True
task_track_started = True
task_acks_late = True
task_acks_on_failure_or_timeout = True
task_reject_on_worker_lost = True
# 额外新增的配置项
# 配置后所有任务都使用不同的队列
use_different_queue = True
from hybrid_cipher import HybridCipher
celery_callback_service_cipher_key = """
-----BEGIN RSA PRIVATE KEY-----
xxxxxx 请自行生成rsa证书
xxxxxx from Crypto.PublicKey import RSA
xxxxxx sk = RSA.generate(1024)
xxxxxx print(sk.export_key().decode())
xxxxxx
xxxxxx celery-callback-worker侧使用私钥
xxxxxx celery-callback-service业务侧使用公钥
-----END RSA PRIVATE KEY-----
""".strip()
celery_callback_service_cipher = HybridCipher(celery_callback_service_cipher_key)
```
*start.sh*
```
#!/bin/bash
celery -l celery_callback_service.celery_tasks:app worker -l DEBUG
```
执行`./start.sh`启动`celery-callback-worker`。
### 业务程序中引入`celery-callback-service`
*celeryconfig.py*
```python
# 与celery-callback-worker中的celeryconfig.py保持一致
# 这是celery-callback-worker中的私钥需要更新为相应的公钥
```
*pro/settings.py*
```python
INSTALLED_APPS = [
...
"django_admin_daterange_listfilter",
"celery_callback_service",
...
]
# 这里用于设置回调接口访问的APIKEYS
# 允许多个，请定期更换
CELERY_CALLBACK_SERVICE_APIKEYS = [
"yEcU2IrtVGslTgw6JmkoTo4Trkplnyg8",
"kBhZB8yKKFmXoAzFHP7HVembYsAeOyBk",
]
# 这里的服务地址必须`celery-callback-worker`能够访问的地址
CELERY_CALLBACK_SERVICE_ADDRESS = "http://127.0.0.1:8000"
```
*初始化`celery-callback-service`相关数据表*
```shell
python manage.py migrate celery_callback_service
```
`app/tasks.py`
```python
def task1(arg1, arg2):
pass
task1.execution_lock_timeout = 60 # 设置回调任务的锁定时间。默认60秒。
task1.delay_seconds = 5 # 设置n秒后再回调。默认是5秒后再回调。主要是避免在业务接口中创建的Task还没有commit到数据库，导致回调时Task任务不可见。
```
*注意：task1等回调函数，必须全局可见。*
`app/services.py`
```python
from celery_callback_service.client import start_callback_service
from .tasks import task1
def service_func1(*args, **kwargs):
...
start_callback_service(task1, arg1, arg2...)
...
```
## 配置项
### 业务侧`settings.py`额外配置项
- CELERY_CALLBACK_SERVICE_APIKEYS
- 不设置的话，优先继承`DJANGO_APIS_APIKEYS`配置项的值
- 如果`DJANGO_APIS_APIKEYS`也没有设置的话，则自动生成随机授权码，并打印输出。
- CELERY_CALLBACK_SERVICE_ADDRESS
- 如果不设置，则在启动时告警。不设置的话，是没有办法正常回调的。
### 业务侧`celeryconfig.py`额外配置项
- celery_callback_service_cipher: 无默认值，必须的配置项
- celery_callback_retry_countdown_step: 5
- celery_callback_retry_countdown_max: 300
- celery_callback_max_retries: 2048
## 回调异常的说明
- 如果回调执行过程中出现程序逻辑异常，则会结束该回调任务，并把错误信息记录在记录表中。
- 如果回调执行过程中出现网络异常、网关异常等HTTP状态码非200的情况，则会将任务纳入重试。
- 重试延迟规则是：5秒（可配置） * 重试次数，最大延迟300秒（可配置）。
## 版本记录
### v0.1.6
- 版本首发。
### v0.1.7
- 管理界面添加重新推送、添加删除标记、取消删除标记等指处理动作。
### v0.1.8
- django-apis兼容性改造。
Raw data
{
"_id": null,
"home_page": null,
"name": "celery-callback-service",
"maintainer": "Sun XiaoWei",
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "celery callback service",
"author": "Sun XiaoWei",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/5e/48/f6ae488eb9930f9beb8b67ed0f38e08c0a7019597bf5d3cb94b01f26a0ba/celery-callback-service-0.1.8.tar.gz",
"platform": null,
"description": "# celery-callback-service\n\n\u57fa\u4e8ecelery\u7684\u56de\u8c03\u670d\u52a1\u3002\u4e1a\u52a1\u7cfb\u7edf\u521b\u5efacelery\u56de\u8c03\u4efb\u52a1\uff0ccelery-callback-worker\u6267\u884c\u56de\u8c03\u4efb\u52a1\uff0c\u4e1a\u52a1\u7cfb\u7edf\u5728\u56de\u8c03\u4efb\u52a1\u4e2d\u5904\u7406\u5f02\u6b65\u4efb\u52a1\u3002\n\n\n## \u4f7f\u7528\u65b9\u6cd5\n\n### \u542f\u52a8`celery-callback-worker`\uff08celery worker\uff09\n\n*celeryconfig.py*\n\n```python\nworker_concurrency = 10\nworker_pool = \"threads\"\nbroker_url = \"redis://redis/0\"\nresult_backend = \"redis://redis/1\"\naccept_content = [\"application/json\"]\ntask_serializer = \"json\"\nresult_accept_content = [\"application/json\"]\nresult_serializer = \"json\"\ntimezone = \"Asia/Shanghai\"\nbroker_connection_retry_on_startup = True\ntask_track_started = True\ntask_acks_late = True\ntask_acks_on_failure_or_timeout = True\ntask_reject_on_worker_lost = True\n# \u989d\u5916\u65b0\u589e\u7684\u914d\u7f6e\u9879\n# \u914d\u7f6e\u540e\u6240\u6709\u4efb\u52a1\u90fd\u4f7f\u7528\u4e0d\u540c\u7684\u961f\u5217\nuse_different_queue = True\n\nfrom hybrid_cipher import HybridCipher\n\ncelery_callback_service_cipher_key = \"\"\"\n-----BEGIN RSA PRIVATE KEY-----\nxxxxxx \u8bf7\u81ea\u884c\u751f\u6210rsa\u8bc1\u4e66\nxxxxxx from Crypto.PublicKey import RSA\nxxxxxx sk = RSA.generate(1024)\nxxxxxx print(sk.export_key().decode())\nxxxxxx \nxxxxxx celery-callback-worker\u4fa7\u4f7f\u7528\u79c1\u94a5\nxxxxxx celery-callback-service\u4e1a\u52a1\u4fa7\u4f7f\u7528\u516c\u94a5\n-----END RSA PRIVATE KEY-----\n\"\"\".strip()\ncelery_callback_service_cipher = HybridCipher(celery_callback_service_cipher_key)\n```\n\n*start.sh*\n```\n#!/bin/bash\ncelery -l celery_callback_service.celery_tasks:app worker -l DEBUG\n```\n\n\u6267\u884c`./start.sh`\u542f\u52a8`celery-callback-worker`\u3002\n\n### \u4e1a\u52a1\u7a0b\u5e8f\u4e2d\u5f15\u5165`celery-callback-service`\n\n*celeryconfig.py*\n\n```python\n# \u4e0ecelery-callback-worker\u4e2d\u7684celeryconfig.py\u4fdd\u6301\u4e00\u81f4\n# \u8fd9\u662fcelery-callback-worker\u4e2d\u7684\u79c1\u94a5\u9700\u8981\u66f4\u65b0\u4e3a\u76f8\u5e94\u7684\u516c\u94a5\n```\n\n*pro/settings.py*\n\n```python\nINSTALLED_APPS = [\n ...\n \"django_admin_daterange_listfilter\",\n \"celery_callback_service\",\n ...\n]\n\n# \u8fd9\u91cc\u7528\u4e8e\u8bbe\u7f6e\u56de\u8c03\u63a5\u53e3\u8bbf\u95ee\u7684APIKEYS\n# \u5141\u8bb8\u591a\u4e2a\uff0c\u8bf7\u5b9a\u671f\u66f4\u6362\nCELERY_CALLBACK_SERVICE_APIKEYS = [\n \"yEcU2IrtVGslTgw6JmkoTo4Trkplnyg8\",\n \"kBhZB8yKKFmXoAzFHP7HVembYsAeOyBk\",\n]\n# \u8fd9\u91cc\u7684\u670d\u52a1\u5730\u5740\u5fc5\u987b`celery-callback-worker`\u80fd\u591f\u8bbf\u95ee\u7684\u5730\u5740\nCELERY_CALLBACK_SERVICE_ADDRESS = \"http://127.0.0.1:8000\"\n```\n\n*\u521d\u59cb\u5316`celery-callback-service`\u76f8\u5173\u6570\u636e\u8868*\n\n```shell\npython manage.py migrate celery_callback_service\n```\n\n`app/tasks.py`\n\n```python\ndef task1(arg1, arg2):\n pass\n\ntask1.execution_lock_timeout = 60 # \u8bbe\u7f6e\u56de\u8c03\u4efb\u52a1\u7684\u9501\u5b9a\u65f6\u95f4\u3002\u9ed8\u8ba460\u79d2\u3002\ntask1.delay_seconds = 5 # \u8bbe\u7f6en\u79d2\u540e\u518d\u56de\u8c03\u3002\u9ed8\u8ba4\u662f5\u79d2\u540e\u518d\u56de\u8c03\u3002\u4e3b\u8981\u662f\u907f\u514d\u5728\u4e1a\u52a1\u63a5\u53e3\u4e2d\u521b\u5efa\u7684Task\u8fd8\u6ca1\u6709commit\u5230\u6570\u636e\u5e93\uff0c\u5bfc\u81f4\u56de\u8c03\u65f6Task\u4efb\u52a1\u4e0d\u53ef\u89c1\u3002\n```\n\n*\u6ce8\u610f\uff1atask1\u7b49\u56de\u8c03\u51fd\u6570\uff0c\u5fc5\u987b\u5168\u5c40\u53ef\u89c1\u3002*\n\n\n`app/services.py`\n\n```python\nfrom celery_callback_service.client import start_callback_service\nfrom .tasks import task1\n\ndef service_func1(*args, **kwargs):\n ...\n start_callback_service(task1, arg1, arg2...)\n ...\n```\n\n\n## \u914d\u7f6e\u9879\n\n### \u4e1a\u52a1\u4fa7`settings.py`\u989d\u5916\u914d\u7f6e\u9879\n\n- CELERY_CALLBACK_SERVICE_APIKEYS\n - \u4e0d\u8bbe\u7f6e\u7684\u8bdd\uff0c\u4f18\u5148\u7ee7\u627f`DJANGO_APIS_APIKEYS`\u914d\u7f6e\u9879\u7684\u503c\n - \u5982\u679c`DJANGO_APIS_APIKEYS`\u4e5f\u6ca1\u6709\u8bbe\u7f6e\u7684\u8bdd\uff0c\u5219\u81ea\u52a8\u751f\u6210\u968f\u673a\u6388\u6743\u7801\uff0c\u5e76\u6253\u5370\u8f93\u51fa\u3002\n- CELERY_CALLBACK_SERVICE_ADDRESS\n - \u5982\u679c\u4e0d\u8bbe\u7f6e\uff0c\u5219\u5728\u542f\u52a8\u65f6\u544a\u8b66\u3002\u4e0d\u8bbe\u7f6e\u7684\u8bdd\uff0c\u662f\u6ca1\u6709\u529e\u6cd5\u6b63\u5e38\u56de\u8c03\u7684\u3002\n\n### \u4e1a\u52a1\u4fa7`celeryconfig.py`\u989d\u5916\u914d\u7f6e\u9879\n\n- celery_callback_service_cipher: \u65e0\u9ed8\u8ba4\u503c\uff0c\u5fc5\u987b\u7684\u914d\u7f6e\u9879\n- celery_callback_retry_countdown_step: 5\n- celery_callback_retry_countdown_max: 300\n- celery_callback_max_retries: 2048\n\n\n## \u56de\u8c03\u5f02\u5e38\u7684\u8bf4\u660e\n\n- \u5982\u679c\u56de\u8c03\u6267\u884c\u8fc7\u7a0b\u4e2d\u51fa\u73b0\u7a0b\u5e8f\u903b\u8f91\u5f02\u5e38\uff0c\u5219\u4f1a\u7ed3\u675f\u8be5\u56de\u8c03\u4efb\u52a1\uff0c\u5e76\u628a\u9519\u8bef\u4fe1\u606f\u8bb0\u5f55\u5728\u8bb0\u5f55\u8868\u4e2d\u3002\n- \u5982\u679c\u56de\u8c03\u6267\u884c\u8fc7\u7a0b\u4e2d\u51fa\u73b0\u7f51\u7edc\u5f02\u5e38\u3001\u7f51\u5173\u5f02\u5e38\u7b49HTTP\u72b6\u6001\u7801\u975e200\u7684\u60c5\u51b5\uff0c\u5219\u4f1a\u5c06\u4efb\u52a1\u7eb3\u5165\u91cd\u8bd5\u3002\n - \u91cd\u8bd5\u5ef6\u8fdf\u89c4\u5219\u662f\uff1a5\u79d2\uff08\u53ef\u914d\u7f6e\uff09 * \u91cd\u8bd5\u6b21\u6570\uff0c\u6700\u5927\u5ef6\u8fdf300\u79d2\uff08\u53ef\u914d\u7f6e\uff09\u3002\n\n## \u7248\u672c\u8bb0\u5f55\n\n### v0.1.6\n\n- \u7248\u672c\u9996\u53d1\u3002\n\n### v0.1.7\n\n- \u7ba1\u7406\u754c\u9762\u6dfb\u52a0\u91cd\u65b0\u63a8\u9001\u3001\u6dfb\u52a0\u5220\u9664\u6807\u8bb0\u3001\u53d6\u6d88\u5220\u9664\u6807\u8bb0\u7b49\u6307\u5904\u7406\u52a8\u4f5c\u3002\n\n### v0.1.8\n\n- django-apis\u517c\u5bb9\u6027\u6539\u9020\u3002\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "\u57fa\u4e8ecelery\u7684\u56de\u8c03\u670d\u52a1\u3002\u4e1a\u52a1\u7cfb\u7edf\u521b\u5efacelery\u56de\u8c03\u4efb\u52a1\uff0ccelery-callback-worker\u6267\u884c\u56de\u8c03\u4efb\u52a1\uff0c\u4e1a\u52a1\u7cfb\u7edf\u5728\u56de\u8c03\u4efb\u52a1\u4e2d\u5904\u7406\u5f02\u6b65\u4efb\u52a1\u3002",
"version": "0.1.8",
"project_urls": null,
"split_keywords": [
"celery",
"callback",
"service"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a4a1d3219bf53c81cf02f26841df736daae88ddaf879cb10cf70010c548c9017",
"md5": "0e1e8aa8936aae28efeddce150a9725b",
"sha256": "a9872b4b69d05dfedb1095975a0f3c8d1aa5216f4d74e49aeed23f8fd5de1a8d"
},
"downloads": -1,
"filename": "celery_callback_service-0.1.8-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0e1e8aa8936aae28efeddce150a9725b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 20550,
"upload_time": "2024-08-17T01:29:43",
"upload_time_iso_8601": "2024-08-17T01:29:43.105540Z",
"url": "https://files.pythonhosted.org/packages/a4/a1/d3219bf53c81cf02f26841df736daae88ddaf879cb10cf70010c548c9017/celery_callback_service-0.1.8-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "5e48f6ae488eb9930f9beb8b67ed0f38e08c0a7019597bf5d3cb94b01f26a0ba",
"md5": "f67f404fd6999a22ee4893b7fcab83e9",
"sha256": "cf8fb4ecb6515ad3de2b7260176696ad1827379c14f091b14f9d27eb2afe7d67"
},
"downloads": -1,
"filename": "celery-callback-service-0.1.8.tar.gz",
"has_sig": false,
"md5_digest": "f67f404fd6999a22ee4893b7fcab83e9",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 15801,
"upload_time": "2024-08-17T01:29:44",
"upload_time_iso_8601": "2024-08-17T01:29:44.964718Z",
"url": "https://files.pythonhosted.org/packages/5e/48/f6ae488eb9930f9beb8b67ed0f38e08c0a7019597bf5d3cb94b01f26a0ba/celery-callback-service-0.1.8.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-17 01:29:44",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "celery-callback-service"
}
