| Name | shihua-liquid JSON |
| Version |
0.1
JSON |
| download |
| home_page | https://github.com/redblue0216/Liquid |
| Summary | Liquid is a pipeline tool based on pluggy hook technology, which is mainly used for rapid construction of data science applications. |
| upload_time | 2023-03-14 09:54:55 |
| maintainer | |
| docs_url | None |
| author | shihua |
| requires_python | >=3.9.12 |
| license | MIT |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Liquid-算法应用流程管理工具
   
## 介绍
+ liquid是一个pipeline工具，主要功能提供算法应用快捷组织功能，主要技术使用基于pluggy的hook技术。
## 安装
liquid采用Python开发，得益于Python良好的社区环境，安装支持Pythonic风格的各种管理器。
```
$ pip install liquid-0.1-xxxxxxxxxxxx.whl
```
## 快速指南
### python-sdk使用
+ liquid提供python-sdk的使用方式，以下是liquid主程脚本示例：
```python
from liquid.node import Node
from liquid.pipeline import Pipeline
from liquid.io import DataCatalog
from liquid.hook import HookManager
from liquid.runner import SequentialRunner
### 开始liquid测试
### 设置函数节点对象
def first_func(a,b):
c = a + b
return c
def second_func(c,d):
e = c * d
return e
def third_func(e):
f = e + 1
return f
first_node = Node(func=first_func,inputs=['a','b'],outputs='c',name='firstfunc')
second_node = Node(func=second_func,inputs=['c','d'],outputs='e',name='secondfunc')
third_node = Node(func=third_func,inputs=['e'],outputs=['f'],name='thirdfunc')
### 创建一个pipeline
test_pipeline = Pipeline(nodes=[first_node,second_node,third_node])
### catalog加载初始参数
test_cache_data = DataCatalog()
test_cache_data.save(data_name='a',data_obj=5)
test_cache_data.save(data_name='b',data_obj=6)
test_cache_data.save(data_name='d',data_obj=7)
### hook_manager加载已挂载的前后处理功能函数
hook_manager = HookManager()
sequential_runner = SequentialRunner(pipeline=test_pipeline,catalog=test_cache_data,hook_manager=hook_manager)
# print(sequential_runner.pipeline,sequential_runner.catalog,sequential_runner.hook_manager)
print('--------------------------------------------------------------------------------------------------------------')
sequential_runner.execute(is_release=False)
print(test_cache_data.load(data_name='f'))
print(test_cache_data.cache_data)
### 测试过程(5+6)*7+1=78
```
## 设计
+ 基于pluggy的hook技术实现灵活扩展性
+ 设计关键概念，Node，Pipeline,Runner
+ 函数节点化-Node
+ 开放挂载点-Pipeline
+ 扩展运行方式(顺序运行和并行运行)-Runner
### 技术列表
+ property动态属性
+ 基于pluggy的hook
+ __call__
+ __add__
+ cache_dict
### 设计UML图
以下是设计的UML图：
Raw data
{
"_id": null,
"home_page": "https://github.com/redblue0216/Liquid",
"name": "shihua-liquid",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.9.12",
"maintainer_email": "",
"keywords": "",
"author": "shihua",
"author_email": "15021408795@163.com",
"download_url": "https://files.pythonhosted.org/packages/0e/22/7e8eef11022cb0a5fcae36204d9a89ee042f09e8b2304a8c9e7866575aaa/shihua-liquid-0.1.tar.gz",
"platform": null,
"description": "# Liquid-\u7b97\u6cd5\u5e94\u7528\u6d41\u7a0b\u7ba1\u7406\u5de5\u5177\n\n   \n\n\n\n## \u4ecb\u7ecd\n+ liquid\u662f\u4e00\u4e2apipeline\u5de5\u5177\uff0c\u4e3b\u8981\u529f\u80fd\u63d0\u4f9b\u7b97\u6cd5\u5e94\u7528\u5feb\u6377\u7ec4\u7ec7\u529f\u80fd\uff0c\u4e3b\u8981\u6280\u672f\u4f7f\u7528\u57fa\u4e8epluggy\u7684hook\u6280\u672f\u3002\n\n\n## \u5b89\u88c5\nliquid\u91c7\u7528Python\u5f00\u53d1\uff0c\u5f97\u76ca\u4e8ePython\u826f\u597d\u7684\u793e\u533a\u73af\u5883\uff0c\u5b89\u88c5\u652f\u6301Pythonic\u98ce\u683c\u7684\u5404\u79cd\u7ba1\u7406\u5668\u3002\n```\n\t$ pip install liquid-0.1-xxxxxxxxxxxx.whl\n```\n\n\n## \u5feb\u901f\u6307\u5357\n\n### python-sdk\u4f7f\u7528\n+ liquid\u63d0\u4f9bpython-sdk\u7684\u4f7f\u7528\u65b9\u5f0f\uff0c\u4ee5\u4e0b\u662fliquid\u4e3b\u7a0b\u811a\u672c\u793a\u4f8b\uff1a\n\n```python\nfrom liquid.node import Node\nfrom liquid.pipeline import Pipeline\nfrom liquid.io import DataCatalog\nfrom liquid.hook import HookManager\nfrom liquid.runner import SequentialRunner\n\n\n\n### \u5f00\u59cbliquid\u6d4b\u8bd5\n### \u8bbe\u7f6e\u51fd\u6570\u8282\u70b9\u5bf9\u8c61\ndef first_func(a,b):\n c = a + b\n return c\ndef second_func(c,d):\n e = c * d\n return e\ndef third_func(e):\n f = e + 1\n return f\nfirst_node = Node(func=first_func,inputs=['a','b'],outputs='c',name='firstfunc')\nsecond_node = Node(func=second_func,inputs=['c','d'],outputs='e',name='secondfunc')\nthird_node = Node(func=third_func,inputs=['e'],outputs=['f'],name='thirdfunc')\n### \u521b\u5efa\u4e00\u4e2apipeline\ntest_pipeline = Pipeline(nodes=[first_node,second_node,third_node])\n### catalog\u52a0\u8f7d\u521d\u59cb\u53c2\u6570\ntest_cache_data = DataCatalog()\ntest_cache_data.save(data_name='a',data_obj=5)\ntest_cache_data.save(data_name='b',data_obj=6)\ntest_cache_data.save(data_name='d',data_obj=7)\n### hook_manager\u52a0\u8f7d\u5df2\u6302\u8f7d\u7684\u524d\u540e\u5904\u7406\u529f\u80fd\u51fd\u6570\nhook_manager = HookManager()\nsequential_runner = SequentialRunner(pipeline=test_pipeline,catalog=test_cache_data,hook_manager=hook_manager)\n# print(sequential_runner.pipeline,sequential_runner.catalog,sequential_runner.hook_manager)\nprint('--------------------------------------------------------------------------------------------------------------')\nsequential_runner.execute(is_release=False)\nprint(test_cache_data.load(data_name='f'))\nprint(test_cache_data.cache_data)\n### \u6d4b\u8bd5\u8fc7\u7a0b(5+6)*7+1=78\n```\n\n## \u8bbe\u8ba1\n+ \u57fa\u4e8epluggy\u7684hook\u6280\u672f\u5b9e\u73b0\u7075\u6d3b\u6269\u5c55\u6027\n+ \u8bbe\u8ba1\u5173\u952e\u6982\u5ff5\uff0cNode\uff0cPipeline,Runner\n+ \u51fd\u6570\u8282\u70b9\u5316-Node\n+ \u5f00\u653e\u6302\u8f7d\u70b9-Pipeline\n+ \u6269\u5c55\u8fd0\u884c\u65b9\u5f0f(\u987a\u5e8f\u8fd0\u884c\u548c\u5e76\u884c\u8fd0\u884c)-Runner\n\n### \u6280\u672f\u5217\u8868\n+ property\u52a8\u6001\u5c5e\u6027\n+ \u57fa\u4e8epluggy\u7684hook\n+ __call__\n+ __add__\n+ cache_dict\n\n\n### \u8bbe\u8ba1UML\u56fe\n\u4ee5\u4e0b\u662f\u8bbe\u8ba1\u7684UML\u56fe\uff1a\n\n\n\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Liquid is a pipeline tool based on pluggy hook technology, which is mainly used for rapid construction of data science applications.",
"version": "0.1",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e63e7300ab99a2278d8c8dc15764c66726d3f230b0c64a8ee792c167531f5b5f",
"md5": "4db60ecebb016475191fef8c02d62d6c",
"sha256": "0e067cd2f7e7755e31edb0fefd5f0d1a04e1f80732e431829a2a50a156b2bf2c"
},
"downloads": -1,
"filename": "shihua_liquid-0.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "4db60ecebb016475191fef8c02d62d6c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9.12",
"size": 12778,
"upload_time": "2023-03-14T09:54:53",
"upload_time_iso_8601": "2023-03-14T09:54:53.044430Z",
"url": "https://files.pythonhosted.org/packages/e6/3e/7300ab99a2278d8c8dc15764c66726d3f230b0c64a8ee792c167531f5b5f/shihua_liquid-0.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "0e227e8eef11022cb0a5fcae36204d9a89ee042f09e8b2304a8c9e7866575aaa",
"md5": "d49ce697983704aeea824badf3b59b2b",
"sha256": "09ec5f50ca01b4c1f93adff183f2a20cbe3e12d35e4fe0028dc1cf6edbe2b84f"
},
"downloads": -1,
"filename": "shihua-liquid-0.1.tar.gz",
"has_sig": false,
"md5_digest": "d49ce697983704aeea824badf3b59b2b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9.12",
"size": 10141,
"upload_time": "2023-03-14T09:54:55",
"upload_time_iso_8601": "2023-03-14T09:54:55.213241Z",
"url": "https://files.pythonhosted.org/packages/0e/22/7e8eef11022cb0a5fcae36204d9a89ee042f09e8b2304a8c9e7866575aaa/shihua-liquid-0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-03-14 09:54:55",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "redblue0216",
"github_project": "Liquid",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "shihua-liquid"
}
