celery-callback-service


Namecelery-callback-service JSON
Version 0.1.7 PyPI version JSON
download
home_pageNone
Summary基于celery的回调服务。业务系统创建celery回调任务,celery-callback-worker执行回调任务,业务系统在回调任务中处理异步任务。
upload_time2024-05-27 06:55:21
maintainerSun XiaoWei
docs_urlNone
authorSun XiaoWei
requires_pythonNone
licenseMIT
keywords celery callback service
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # celery-callback-service

基于celery的回调服务。业务系统创建celery回调任务,celery-callback-worker执行回调任务,业务系统在回调任务中处理异步任务。


## 使用方法

###  启动`celery-callback-worker`(celery worker)

*celeryconfig.py*

```python
worker_concurrency = 10
worker_pool = "threads"
broker_url = "redis://redis/0"
result_backend = "redis://redis/1"
accept_content = ["application/json"]
task_serializer = "json"
result_accept_content = ["application/json"]
result_serializer = "json"
timezone = "Asia/Shanghai"
broker_connection_retry_on_startup = True
task_track_started = True
task_acks_late = True
task_acks_on_failure_or_timeout = True
task_reject_on_worker_lost = True
# 额外新增的配置项
# 配置后所有任务都使用不同的队列
use_different_queue = True

from hybrid_cipher import HybridCipher

celery_callback_service_cipher_key = """
-----BEGIN RSA PRIVATE KEY-----
xxxxxx 请自行生成rsa证书
xxxxxx from Crypto.PublicKey import RSA
xxxxxx sk = RSA.generate(1024)
xxxxxx print(sk.export_key().decode())
xxxxxx 
xxxxxx celery-callback-worker侧使用私钥
xxxxxx celery-callback-service业务侧使用公钥
-----END RSA PRIVATE KEY-----
""".strip()
celery_callback_service_cipher = HybridCipher(celery_callback_service_cipher_key)
```

*start.sh*
```
#!/bin/bash
celery -l celery_callback_service.celery_tasks:app worker -l DEBUG
```

执行`./start.sh`启动`celery-callback-worker`。

### 业务程序中引入`celery-callback-service`

*celeryconfig.py*

```python
# 与celery-callback-worker中的celeryconfig.py保持一致
# 这是celery-callback-worker中的私钥需要更新为相应的公钥
```

*pro/settings.py*

```python
INSTALLED_APPS = [
    ...
    "django_admin_daterange_listfilter",
    "celery_callback_service",
    ...
]

# 这里用于设置回调接口访问的APIKEYS
# 允许多个,请定期更换
CELERY_CALLBACK_SERVICE_APIKEYS = [
    "yEcU2IrtVGslTgw6JmkoTo4Trkplnyg8",
    "kBhZB8yKKFmXoAzFHP7HVembYsAeOyBk",
]
# 这里的服务地址必须`celery-callback-worker`能够访问的地址
CELERY_CALLBACK_SERVICE_ADDRESS = "http://127.0.0.1:8000"
```

*初始化`celery-callback-service`相关数据表*

```shell
python manage.py migrate celery_callback_service
```

`app/tasks.py`

```python
def task1(arg1, arg2):
   pass

task1.execution_lock_timeout = 60 # 设置回调任务的锁定时间。默认60秒。
task1.delay_seconds = 5 # 设置n秒后再回调。默认是5秒后再回调。主要是避免在业务接口中创建的Task还没有commit到数据库,导致回调时Task任务不可见。
```

*注意:task1等回调函数,必须全局可见。*


`app/services.py`

```python
from celery_callback_service.client import start_callback_service
from .tasks import task1

def service_func1(*args, **kwargs):
    ...
    start_callback_service(task1, arg1, arg2...)
    ...
```


## 配置项

### 业务侧`settings.py`额外配置项

- CELERY_CALLBACK_SERVICE_APIKEYS
    - 不设置的话,优先继承`DJANGO_APIS_APIKEYS`配置项的值
    - 如果`DJANGO_APIS_APIKEYS`也没有设置的话,则自动生成随机授权码,并打印输出。
- CELERY_CALLBACK_SERVICE_ADDRESS
    - 如果不设置,则在启动时告警。不设置的话,是没有办法正常回调的。

### 业务侧`celeryconfig.py`额外配置项

- celery_callback_service_cipher: 无默认值,必须的配置项
- celery_callback_retry_countdown_step: 5
- celery_callback_retry_countdown_max: 300
- celery_callback_max_retries: 2048


## 回调异常的说明

- 如果回调执行过程中出现程序逻辑异常,则会结束该回调任务,并把错误信息记录在记录表中。
- 如果回调执行过程中出现网络异常、网关异常等HTTP状态码非200的情况,则会将任务纳入重试。
    - 重试延迟规则是:5秒(可配置) * 重试次数,最大延迟300秒(可配置)。

## 版本记录

### v0.1.6

- 版本首发。

### v0.1.7

- 管理界面添加重新推送、添加删除标记、取消删除标记等指处理动作。

            

Raw data

            {
    "_id": null,
    "home_page": null,
    "name": "celery-callback-service",
    "maintainer": "Sun XiaoWei",
    "docs_url": null,
    "requires_python": null,
    "maintainer_email": null,
    "keywords": "celery callback service",
    "author": "Sun XiaoWei",
    "author_email": null,
    "download_url": "https://files.pythonhosted.org/packages/eb/98/7d45ca4f6a2ae4a790ed5c69253135b1425c901d90f13e2d7d70df079ee4/celery-callback-service-0.1.7.tar.gz",
    "platform": null,
    "description": "# celery-callback-service\n\n\u57fa\u4e8ecelery\u7684\u56de\u8c03\u670d\u52a1\u3002\u4e1a\u52a1\u7cfb\u7edf\u521b\u5efacelery\u56de\u8c03\u4efb\u52a1\uff0ccelery-callback-worker\u6267\u884c\u56de\u8c03\u4efb\u52a1\uff0c\u4e1a\u52a1\u7cfb\u7edf\u5728\u56de\u8c03\u4efb\u52a1\u4e2d\u5904\u7406\u5f02\u6b65\u4efb\u52a1\u3002\n\n\n## \u4f7f\u7528\u65b9\u6cd5\n\n###  \u542f\u52a8`celery-callback-worker`\uff08celery worker\uff09\n\n*celeryconfig.py*\n\n```python\nworker_concurrency = 10\nworker_pool = \"threads\"\nbroker_url = \"redis://redis/0\"\nresult_backend = \"redis://redis/1\"\naccept_content = [\"application/json\"]\ntask_serializer = \"json\"\nresult_accept_content = [\"application/json\"]\nresult_serializer = \"json\"\ntimezone = \"Asia/Shanghai\"\nbroker_connection_retry_on_startup = True\ntask_track_started = True\ntask_acks_late = True\ntask_acks_on_failure_or_timeout = True\ntask_reject_on_worker_lost = True\n# \u989d\u5916\u65b0\u589e\u7684\u914d\u7f6e\u9879\n# \u914d\u7f6e\u540e\u6240\u6709\u4efb\u52a1\u90fd\u4f7f\u7528\u4e0d\u540c\u7684\u961f\u5217\nuse_different_queue = True\n\nfrom hybrid_cipher import HybridCipher\n\ncelery_callback_service_cipher_key = \"\"\"\n-----BEGIN RSA PRIVATE KEY-----\nxxxxxx \u8bf7\u81ea\u884c\u751f\u6210rsa\u8bc1\u4e66\nxxxxxx from Crypto.PublicKey import RSA\nxxxxxx sk = RSA.generate(1024)\nxxxxxx print(sk.export_key().decode())\nxxxxxx \nxxxxxx celery-callback-worker\u4fa7\u4f7f\u7528\u79c1\u94a5\nxxxxxx celery-callback-service\u4e1a\u52a1\u4fa7\u4f7f\u7528\u516c\u94a5\n-----END RSA PRIVATE KEY-----\n\"\"\".strip()\ncelery_callback_service_cipher = HybridCipher(celery_callback_service_cipher_key)\n```\n\n*start.sh*\n```\n#!/bin/bash\ncelery -l celery_callback_service.celery_tasks:app worker -l DEBUG\n```\n\n\u6267\u884c`./start.sh`\u542f\u52a8`celery-callback-worker`\u3002\n\n### \u4e1a\u52a1\u7a0b\u5e8f\u4e2d\u5f15\u5165`celery-callback-service`\n\n*celeryconfig.py*\n\n```python\n# \u4e0ecelery-callback-worker\u4e2d\u7684celeryconfig.py\u4fdd\u6301\u4e00\u81f4\n# \u8fd9\u662fcelery-callback-worker\u4e2d\u7684\u79c1\u94a5\u9700\u8981\u66f4\u65b0\u4e3a\u76f8\u5e94\u7684\u516c\u94a5\n```\n\n*pro/settings.py*\n\n```python\nINSTALLED_APPS = [\n    ...\n    \"django_admin_daterange_listfilter\",\n    \"celery_callback_service\",\n    ...\n]\n\n# \u8fd9\u91cc\u7528\u4e8e\u8bbe\u7f6e\u56de\u8c03\u63a5\u53e3\u8bbf\u95ee\u7684APIKEYS\n# \u5141\u8bb8\u591a\u4e2a\uff0c\u8bf7\u5b9a\u671f\u66f4\u6362\nCELERY_CALLBACK_SERVICE_APIKEYS = [\n    \"yEcU2IrtVGslTgw6JmkoTo4Trkplnyg8\",\n    \"kBhZB8yKKFmXoAzFHP7HVembYsAeOyBk\",\n]\n# \u8fd9\u91cc\u7684\u670d\u52a1\u5730\u5740\u5fc5\u987b`celery-callback-worker`\u80fd\u591f\u8bbf\u95ee\u7684\u5730\u5740\nCELERY_CALLBACK_SERVICE_ADDRESS = \"http://127.0.0.1:8000\"\n```\n\n*\u521d\u59cb\u5316`celery-callback-service`\u76f8\u5173\u6570\u636e\u8868*\n\n```shell\npython manage.py migrate celery_callback_service\n```\n\n`app/tasks.py`\n\n```python\ndef task1(arg1, arg2):\n   pass\n\ntask1.execution_lock_timeout = 60 # \u8bbe\u7f6e\u56de\u8c03\u4efb\u52a1\u7684\u9501\u5b9a\u65f6\u95f4\u3002\u9ed8\u8ba460\u79d2\u3002\ntask1.delay_seconds = 5 # \u8bbe\u7f6en\u79d2\u540e\u518d\u56de\u8c03\u3002\u9ed8\u8ba4\u662f5\u79d2\u540e\u518d\u56de\u8c03\u3002\u4e3b\u8981\u662f\u907f\u514d\u5728\u4e1a\u52a1\u63a5\u53e3\u4e2d\u521b\u5efa\u7684Task\u8fd8\u6ca1\u6709commit\u5230\u6570\u636e\u5e93\uff0c\u5bfc\u81f4\u56de\u8c03\u65f6Task\u4efb\u52a1\u4e0d\u53ef\u89c1\u3002\n```\n\n*\u6ce8\u610f\uff1atask1\u7b49\u56de\u8c03\u51fd\u6570\uff0c\u5fc5\u987b\u5168\u5c40\u53ef\u89c1\u3002*\n\n\n`app/services.py`\n\n```python\nfrom celery_callback_service.client import start_callback_service\nfrom .tasks import task1\n\ndef service_func1(*args, **kwargs):\n    ...\n    start_callback_service(task1, arg1, arg2...)\n    ...\n```\n\n\n## \u914d\u7f6e\u9879\n\n### \u4e1a\u52a1\u4fa7`settings.py`\u989d\u5916\u914d\u7f6e\u9879\n\n- CELERY_CALLBACK_SERVICE_APIKEYS\n    - \u4e0d\u8bbe\u7f6e\u7684\u8bdd\uff0c\u4f18\u5148\u7ee7\u627f`DJANGO_APIS_APIKEYS`\u914d\u7f6e\u9879\u7684\u503c\n    - \u5982\u679c`DJANGO_APIS_APIKEYS`\u4e5f\u6ca1\u6709\u8bbe\u7f6e\u7684\u8bdd\uff0c\u5219\u81ea\u52a8\u751f\u6210\u968f\u673a\u6388\u6743\u7801\uff0c\u5e76\u6253\u5370\u8f93\u51fa\u3002\n- CELERY_CALLBACK_SERVICE_ADDRESS\n    - \u5982\u679c\u4e0d\u8bbe\u7f6e\uff0c\u5219\u5728\u542f\u52a8\u65f6\u544a\u8b66\u3002\u4e0d\u8bbe\u7f6e\u7684\u8bdd\uff0c\u662f\u6ca1\u6709\u529e\u6cd5\u6b63\u5e38\u56de\u8c03\u7684\u3002\n\n### \u4e1a\u52a1\u4fa7`celeryconfig.py`\u989d\u5916\u914d\u7f6e\u9879\n\n- celery_callback_service_cipher: \u65e0\u9ed8\u8ba4\u503c\uff0c\u5fc5\u987b\u7684\u914d\u7f6e\u9879\n- celery_callback_retry_countdown_step: 5\n- celery_callback_retry_countdown_max: 300\n- celery_callback_max_retries: 2048\n\n\n## \u56de\u8c03\u5f02\u5e38\u7684\u8bf4\u660e\n\n- \u5982\u679c\u56de\u8c03\u6267\u884c\u8fc7\u7a0b\u4e2d\u51fa\u73b0\u7a0b\u5e8f\u903b\u8f91\u5f02\u5e38\uff0c\u5219\u4f1a\u7ed3\u675f\u8be5\u56de\u8c03\u4efb\u52a1\uff0c\u5e76\u628a\u9519\u8bef\u4fe1\u606f\u8bb0\u5f55\u5728\u8bb0\u5f55\u8868\u4e2d\u3002\n- \u5982\u679c\u56de\u8c03\u6267\u884c\u8fc7\u7a0b\u4e2d\u51fa\u73b0\u7f51\u7edc\u5f02\u5e38\u3001\u7f51\u5173\u5f02\u5e38\u7b49HTTP\u72b6\u6001\u7801\u975e200\u7684\u60c5\u51b5\uff0c\u5219\u4f1a\u5c06\u4efb\u52a1\u7eb3\u5165\u91cd\u8bd5\u3002\n    - \u91cd\u8bd5\u5ef6\u8fdf\u89c4\u5219\u662f\uff1a5\u79d2\uff08\u53ef\u914d\u7f6e\uff09 * \u91cd\u8bd5\u6b21\u6570\uff0c\u6700\u5927\u5ef6\u8fdf300\u79d2\uff08\u53ef\u914d\u7f6e\uff09\u3002\n\n## \u7248\u672c\u8bb0\u5f55\n\n### v0.1.6\n\n- \u7248\u672c\u9996\u53d1\u3002\n\n### v0.1.7\n\n- \u7ba1\u7406\u754c\u9762\u6dfb\u52a0\u91cd\u65b0\u63a8\u9001\u3001\u6dfb\u52a0\u5220\u9664\u6807\u8bb0\u3001\u53d6\u6d88\u5220\u9664\u6807\u8bb0\u7b49\u6307\u5904\u7406\u52a8\u4f5c\u3002\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "\u57fa\u4e8ecelery\u7684\u56de\u8c03\u670d\u52a1\u3002\u4e1a\u52a1\u7cfb\u7edf\u521b\u5efacelery\u56de\u8c03\u4efb\u52a1\uff0ccelery-callback-worker\u6267\u884c\u56de\u8c03\u4efb\u52a1\uff0c\u4e1a\u52a1\u7cfb\u7edf\u5728\u56de\u8c03\u4efb\u52a1\u4e2d\u5904\u7406\u5f02\u6b65\u4efb\u52a1\u3002",
    "version": "0.1.7",
    "project_urls": null,
    "split_keywords": [
        "celery",
        "callback",
        "service"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "2a014364938f335597f4fef808f69700f18e7e394385d48775765aeb813eab8d",
                "md5": "344b896e8dbccc0ef1163dc7a1d5c3c7",
                "sha256": "3cd1632d9b12f62af0e9a7a01568fa76f2baa0b5e3ac4399b44affa230f8bbd5"
            },
            "downloads": -1,
            "filename": "celery_callback_service-0.1.7-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "344b896e8dbccc0ef1163dc7a1d5c3c7",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 20539,
            "upload_time": "2024-05-27T06:55:18",
            "upload_time_iso_8601": "2024-05-27T06:55:18.577929Z",
            "url": "https://files.pythonhosted.org/packages/2a/01/4364938f335597f4fef808f69700f18e7e394385d48775765aeb813eab8d/celery_callback_service-0.1.7-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "eb987d45ca4f6a2ae4a790ed5c69253135b1425c901d90f13e2d7d70df079ee4",
                "md5": "8d3d5b2a6dc242b786ea38b8f92fdb4e",
                "sha256": "2357532fdecbb0540ff381ca0ba2e83540318387f6f9fccb08cd1f9adffaaeec"
            },
            "downloads": -1,
            "filename": "celery-callback-service-0.1.7.tar.gz",
            "has_sig": false,
            "md5_digest": "8d3d5b2a6dc242b786ea38b8f92fdb4e",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 15748,
            "upload_time": "2024-05-27T06:55:21",
            "upload_time_iso_8601": "2024-05-27T06:55:21.098934Z",
            "url": "https://files.pythonhosted.org/packages/eb/98/7d45ca4f6a2ae4a790ed5c69253135b1425c901d90f13e2d7d70df079ee4/celery-callback-service-0.1.7.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-05-27 06:55:21",
    "github": false,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "lcname": "celery-callback-service"
}
        
Elapsed time: 0.23897s