| Name | pipen JSON |
| Version |
0.14.6
JSON |
| download |
| home_page | https://github.com/pwwang/pipen |
| Summary | A pipeline framework for python |
| upload_time | 2024-04-05 22:43:25 |
| maintainer | None |
| docs_url | None |
| author | pwwang |
| requires_python | <4.0,>=3.9 |
| license | MIT |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
|
<div align="center">
<img src="./pipen.png" width="320px">
**A pipeline framework for python**
</div>
______________________________________________________________________
[![Pypi][6]][7] [![Github][8]][9] ![Building][10] [![Docs and API][11]][1] [![Codacy][12]][13] [![Codacy coverage][14]][13] [![Deps][5]][23]
[Documentation][1] | [ChangeLog][2] | [Examples][3] | [API][4]
## Features
- Easy to use
- Nearly zero-configuration
- Nice logging
- Highly extendable
## Installation
```bash
pip install -U pipen
```
## Quickstart
`example.py`
```python
from pipen import Proc, Pipen
class P1(Proc):
"""Sort input file"""
input = "infile"
input_data = ["/tmp/data.txt"]
output = "outfile:file:intermediate.txt"
script = "cat {{in.infile}} | sort > {{out.outfile}}"
class P2(Proc):
"""Paste line number"""
requires = P1
input = "infile"
output = "outfile:file:result.txt"
script = "paste <(seq 1 3) {{in.infile}} > {{out.outfile}}"
class MyPipeline(Pipen):
starts = P1
if __name__ == "__main__":
MyPipeline().run()
```
```shell
> echo -e "3\n2\n1" > /tmp/data.txt
> python example.py
```
```log
06-09 23:15:29 I core _____________________________________ __
06-09 23:15:29 I core ___ __ \___ _/__ __ \__ ____/__ | / /
06-09 23:15:29 I core __ /_/ /__ / __ /_/ /_ __/ __ |/ /
06-09 23:15:29 I core _ ____/__/ / _ ____/_ /___ _ /| /
06-09 23:15:29 I core /_/ /___/ /_/ /_____/ /_/ |_/
06-09 23:15:29 I core
06-09 23:15:29 I core version: 0.14.5
06-09 23:15:29 I core
06-09 23:15:29 I core ╔═══════════════════════════════════════════════════╗
06-09 23:15:29 I core ║ MYPIPELINE ║
06-09 23:15:29 I core ╚═══════════════════════════════════════════════════╝
06-09 23:15:29 I core plugins : verbose v0.11.0
06-09 23:15:29 I core # procs : 2
06-09 23:15:29 I core profile : default
06-09 23:15:29 I core outdir : /home/pwwang/github/pipen/MyPipeline-output
06-09 23:15:29 I core cache : True
06-09 23:15:29 I core dirsig : 1
06-09 23:15:29 I core error_strategy : ignore
06-09 23:15:29 I core forks : 1
06-09 23:15:29 I core lang : bash
06-09 23:15:29 I core loglevel : info
06-09 23:15:29 I core num_retries : 3
06-09 23:15:29 I core scheduler : local
06-09 23:15:29 I core submission_batch: 8
06-09 23:15:29 I core template : liquid
06-09 23:15:29 I core workdir : /home/pwwang/github/pipen/.pipen/MyPipeline
06-09 23:15:29 I core plugin_opts :
06-09 23:15:29 I core template_opts :
06-09 23:15:31 I core
06-09 23:15:31 I core ╭──────────────────────── P1 ───────────────────────╮
06-09 23:15:31 I core │ Sort input file │
06-09 23:15:31 I core ╰──────────────────────────────────────────────────╯
06-09 23:15:31 I core P1: Workdir: '/home/pwwang/github/pipen/.pipen/MyPipeline/P1'
06-09 23:15:31 I core P1: <<< [START]
06-09 23:15:31 I core P1: >>> ['P2']
06-09 23:15:31 I verbose P1: size: 1
06-09 23:15:31 I verbose P1: [0/0] in.infile: /tmp/data.txt
06-09 23:15:31 I verbose P1: [0/0] out.outfile:
/home/pwwang/github/pipen/.pipen/MyPipeline/P1/0/output/intermediate.txt
06-09 23:15:33 I verbose P1: Time elapsed: 00:00:02.018s
06-09 23:15:33 I core
06-09 23:15:33 I core ╭════════════════════════ P2 ═══════════════════════╮
06-09 23:15:33 I core ║ Paste line number ║
06-09 23:15:33 I core ╰══════════════════════════════════════════════════╯
06-09 23:15:33 I core P2: Workdir: '/home/pwwang/github/pipen/.pipen/MyPipeline/P2'
06-09 23:15:33 I core P2: <<< ['P1']
06-09 23:15:33 I core P2: >>> [END]
06-09 23:15:33 I verbose P2: size: 1
06-09 23:15:33 I verbose P2: [0/0] in.infile:
/home/pwwang/github/pipen/.pipen/MyPipeline/P1/0/output/intermediate.txt
06-09 23:15:33 I verbose P2: [0/0] out.outfile:
/home/pwwang/github/pipen/MyPipeline-output/P2/result.txt
06-09 23:15:35 I verbose P2: Time elapsed: 00:00:02.009s
06-09 23:15:35 I core
MYPIPELINE: 100%|█████████████████████████████| 2/2 [00:06<00:00, 0.36 procs/s]
```
```shell
> cat ./MyPipeline-output/P2/result.txt
1 1
2 2
3 3
```
## Examples
See more examples at `examples/` and a more realcase example at:
https://github.com/pwwang/pipen-report/tree/master/example
## Plugin gallery
Plugins make `pipen` even better.
- [`pipen-annotate`][26]: Use docstring to annotate pipen processes
- [`pipen-args`][19]: Command line argument parser for pipen
- [`pipen-board`][27]: Visualize configuration and running of pipen pipelines on the web
- [`pipen-diagram`][18]: Draw pipeline diagrams for pipen
- [`pipen-dry`][20]: Dry runner for pipen pipelines
- [`pipen-filters`][17]: Add a set of useful filters for pipen templates.
- [`pipen-lock`][25]: Process lock for pipen to prevent multiple runs at the same time.
- [`pipen-log2file`][28]: Save running logs to file for pipen
- [`pipen-poplog`][30]: Populate logs from jobs to running log of the pipeline
- [`pipen-report`][16]: Generate report for pipen
- [`pipen-runinfo`][29]: Save running information to file for pipen
- [`pipen-verbose`][15]: Add verbosal information in logs for pipen.
- [`pipen-cli-init`][21]: A pipen CLI plugin to create a pipen project (pipeline)
- [`pipen-cli-ref`][31]: Make reference documentation for processes
- [`pipen-cli-require`][24]: A pipen cli plugin check the requirements of a pipeline
- [`pipen-cli-run`][22]: A pipen cli plugin to run a process or a pipeline
[1]: https://pwwang.github.io/pipen
[2]: https://pwwang.github.io/pipen/CHANGELOG
[3]: https://pwwang.github.io/pipen/examples
[4]: https://pwwang.github.io/pipen/api/pipen
[5]: https://img.shields.io/librariesio/release/pypi/pipen?style=flat-square
[6]: https://img.shields.io/pypi/v/pipen?style=flat-square
[7]: https://pypi.org/project/pipen/
[8]: https://img.shields.io/github/v/tag/pwwang/pipen?style=flat-square
[9]: https://github.com/pwwang/pipen
[10]: https://img.shields.io/github/actions/workflow/status/pwwang/pipen/build.yml?style=flat-square
[11]: https://img.shields.io/github/actions/workflow/status/pwwang/pipen/docs.yml?label=docs&style=flat-square
[12]: https://img.shields.io/codacy/grade/cf1c6c97e5c4480386a05b42dec10c6e?style=flat-square
[13]: https://app.codacy.com/gh/pwwang/pipen
[14]: https://img.shields.io/codacy/coverage/cf1c6c97e5c4480386a05b42dec10c6e?style=flat-square
[15]: https://github.com/pwwang/pipen-verbose
[16]: https://github.com/pwwang/pipen-report
[17]: https://github.com/pwwang/pipen-filters
[18]: https://github.com/pwwang/pipen-diagram
[19]: https://github.com/pwwang/pipen-args
[20]: https://github.com/pwwang/pipen-dry
[21]: https://github.com/pwwang/pipen-cli-init
[22]: https://github.com/pwwang/pipen-cli-run
[23]: https://libraries.io/github/pwwang/pipen#repository_dependencies
[24]: https://github.com/pwwang/pipen-cli-require
[25]: https://github.com/pwwang/pipen-lock
[26]: https://github.com/pwwang/pipen-annotate
[27]: https://github.com/pwwang/pipen-board
[28]: https://github.com/pwwang/pipen-log2file
[29]: https://github.com/pwwang/pipen-runinfo
[30]: https://github.com/pwwang/pipen-poplog
[31]: https://github.com/pwwang/pipen-cli-ref
Raw data
{
"_id": null,
"home_page": "https://github.com/pwwang/pipen",
"name": "pipen",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.9",
"maintainer_email": null,
"keywords": null,
"author": "pwwang",
"author_email": "pwwang@pwwang.com",
"download_url": "https://files.pythonhosted.org/packages/8e/46/8ab4c05eba161d1155765c4419aaa181e94b9391ce32558bdf7ba40371d6/pipen-0.14.6.tar.gz",
"platform": null,
"description": "<div align=\"center\">\n <img src=\"./pipen.png\" width=\"320px\">\n\n**A pipeline framework for python**\n\n</div>\n\n______________________________________________________________________\n\n[![Pypi][6]][7] [![Github][8]][9] ![Building][10] [![Docs and API][11]][1] [![Codacy][12]][13] [![Codacy coverage][14]][13] [![Deps][5]][23]\n\n[Documentation][1] | [ChangeLog][2] | [Examples][3] | [API][4]\n\n## Features\n\n- Easy to use\n- Nearly zero-configuration\n- Nice logging\n- Highly extendable\n\n## Installation\n\n```bash\npip install -U pipen\n```\n\n## Quickstart\n\n`example.py`\n\n```python\nfrom pipen import Proc, Pipen\n\nclass P1(Proc):\n \"\"\"Sort input file\"\"\"\n input = \"infile\"\n input_data = [\"/tmp/data.txt\"]\n output = \"outfile:file:intermediate.txt\"\n script = \"cat {{in.infile}} | sort > {{out.outfile}}\"\n\nclass P2(Proc):\n \"\"\"Paste line number\"\"\"\n requires = P1\n input = \"infile\"\n output = \"outfile:file:result.txt\"\n script = \"paste <(seq 1 3) {{in.infile}} > {{out.outfile}}\"\n\nclass MyPipeline(Pipen):\n starts = P1\n\nif __name__ == \"__main__\":\n MyPipeline().run()\n```\n\n```shell\n> echo -e \"3\\n2\\n1\" > /tmp/data.txt\n> python example.py\n```\n\n```log\n06-09 23:15:29 I core _____________________________________ __\n06-09 23:15:29 I core ___ __ \\___ _/__ __ \\__ ____/__ | / /\n06-09 23:15:29 I core __ /_/ /__ / __ /_/ /_ __/ __ |/ /\n06-09 23:15:29 I core _ ____/__/ / _ ____/_ /___ _ /| /\n06-09 23:15:29 I core /_/ /___/ /_/ /_____/ /_/ |_/\n06-09 23:15:29 I core\n06-09 23:15:29 I core version: 0.14.5\n06-09 23:15:29 I core\n06-09 23:15:29 I core \u2554\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2557\n06-09 23:15:29 I core \u2551 MYPIPELINE \u2551\n06-09 23:15:29 I core \u255a\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u255d\n06-09 23:15:29 I core plugins : verbose v0.11.0\n06-09 23:15:29 I core # procs : 2\n06-09 23:15:29 I core profile : default\n06-09 23:15:29 I core outdir : /home/pwwang/github/pipen/MyPipeline-output\n06-09 23:15:29 I core cache : True\n06-09 23:15:29 I core dirsig : 1\n06-09 23:15:29 I core error_strategy : ignore\n06-09 23:15:29 I core forks : 1\n06-09 23:15:29 I core lang : bash\n06-09 23:15:29 I core loglevel : info\n06-09 23:15:29 I core num_retries : 3\n06-09 23:15:29 I core scheduler : local\n06-09 23:15:29 I core submission_batch: 8\n06-09 23:15:29 I core template : liquid\n06-09 23:15:29 I core workdir : /home/pwwang/github/pipen/.pipen/MyPipeline\n06-09 23:15:29 I core plugin_opts :\n06-09 23:15:29 I core template_opts :\n06-09 23:15:31 I core\n06-09 23:15:31 I core \u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 P1 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n06-09 23:15:31 I core \u2502 Sort input file \u2502\n06-09 23:15:31 I core \u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n06-09 23:15:31 I core P1: Workdir: '/home/pwwang/github/pipen/.pipen/MyPipeline/P1'\n06-09 23:15:31 I core P1: <<< [START]\n06-09 23:15:31 I core P1: >>> ['P2']\n06-09 23:15:31 I verbose P1: size: 1\n06-09 23:15:31 I verbose P1: [0/0] in.infile: /tmp/data.txt\n06-09 23:15:31 I verbose P1: [0/0] out.outfile:\n /home/pwwang/github/pipen/.pipen/MyPipeline/P1/0/output/intermediate.txt\n06-09 23:15:33 I verbose P1: Time elapsed: 00:00:02.018s\n06-09 23:15:33 I core\n06-09 23:15:33 I core \u256d\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550 P2 \u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u256e\n06-09 23:15:33 I core \u2551 Paste line number \u2551\n06-09 23:15:33 I core \u2570\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u256f\n06-09 23:15:33 I core P2: Workdir: '/home/pwwang/github/pipen/.pipen/MyPipeline/P2'\n06-09 23:15:33 I core P2: <<< ['P1']\n06-09 23:15:33 I core P2: >>> [END]\n06-09 23:15:33 I verbose P2: size: 1\n06-09 23:15:33 I verbose P2: [0/0] in.infile:\n /home/pwwang/github/pipen/.pipen/MyPipeline/P1/0/output/intermediate.txt\n06-09 23:15:33 I verbose P2: [0/0] out.outfile:\n /home/pwwang/github/pipen/MyPipeline-output/P2/result.txt\n06-09 23:15:35 I verbose P2: Time elapsed: 00:00:02.009s\n06-09 23:15:35 I core\n\n MYPIPELINE: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 2/2 [00:06<00:00, 0.36 procs/s]\n```\n\n```shell\n> cat ./MyPipeline-output/P2/result.txt\n1 1\n2 2\n3 3\n```\n\n## Examples\n\nSee more examples at `examples/` and a more realcase example at:\nhttps://github.com/pwwang/pipen-report/tree/master/example\n\n## Plugin gallery\n\nPlugins make `pipen` even better.\n\n- [`pipen-annotate`][26]: Use docstring to annotate pipen processes\n- [`pipen-args`][19]: Command line argument parser for pipen\n- [`pipen-board`][27]: Visualize configuration and running of pipen pipelines on the web\n- [`pipen-diagram`][18]: Draw pipeline diagrams for pipen\n- [`pipen-dry`][20]: Dry runner for pipen pipelines\n- [`pipen-filters`][17]: Add a set of useful filters for pipen templates.\n- [`pipen-lock`][25]: Process lock for pipen to prevent multiple runs at the same time.\n- [`pipen-log2file`][28]: Save running logs to file for pipen\n- [`pipen-poplog`][30]: Populate logs from jobs to running log of the pipeline\n- [`pipen-report`][16]: Generate report for pipen\n- [`pipen-runinfo`][29]: Save running information to file for pipen\n- [`pipen-verbose`][15]: Add verbosal information in logs for pipen.\n- [`pipen-cli-init`][21]: A pipen CLI plugin to create a pipen project (pipeline)\n- [`pipen-cli-ref`][31]: Make reference documentation for processes\n- [`pipen-cli-require`][24]: A pipen cli plugin check the requirements of a pipeline\n- [`pipen-cli-run`][22]: A pipen cli plugin to run a process or a pipeline\n\n\n[1]: https://pwwang.github.io/pipen\n[2]: https://pwwang.github.io/pipen/CHANGELOG\n[3]: https://pwwang.github.io/pipen/examples\n[4]: https://pwwang.github.io/pipen/api/pipen\n[5]: https://img.shields.io/librariesio/release/pypi/pipen?style=flat-square\n[6]: https://img.shields.io/pypi/v/pipen?style=flat-square\n[7]: https://pypi.org/project/pipen/\n[8]: https://img.shields.io/github/v/tag/pwwang/pipen?style=flat-square\n[9]: https://github.com/pwwang/pipen\n[10]: https://img.shields.io/github/actions/workflow/status/pwwang/pipen/build.yml?style=flat-square\n[11]: https://img.shields.io/github/actions/workflow/status/pwwang/pipen/docs.yml?label=docs&style=flat-square\n[12]: https://img.shields.io/codacy/grade/cf1c6c97e5c4480386a05b42dec10c6e?style=flat-square\n[13]: https://app.codacy.com/gh/pwwang/pipen\n[14]: https://img.shields.io/codacy/coverage/cf1c6c97e5c4480386a05b42dec10c6e?style=flat-square\n[15]: https://github.com/pwwang/pipen-verbose\n[16]: https://github.com/pwwang/pipen-report\n[17]: https://github.com/pwwang/pipen-filters\n[18]: https://github.com/pwwang/pipen-diagram\n[19]: https://github.com/pwwang/pipen-args\n[20]: https://github.com/pwwang/pipen-dry\n[21]: https://github.com/pwwang/pipen-cli-init\n[22]: https://github.com/pwwang/pipen-cli-run\n[23]: https://libraries.io/github/pwwang/pipen#repository_dependencies\n[24]: https://github.com/pwwang/pipen-cli-require\n[25]: https://github.com/pwwang/pipen-lock\n[26]: https://github.com/pwwang/pipen-annotate\n[27]: https://github.com/pwwang/pipen-board\n[28]: https://github.com/pwwang/pipen-log2file\n[29]: https://github.com/pwwang/pipen-runinfo\n[30]: https://github.com/pwwang/pipen-poplog\n[31]: https://github.com/pwwang/pipen-cli-ref\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A pipeline framework for python",
"version": "0.14.6",
"project_urls": {
"Homepage": "https://github.com/pwwang/pipen",
"Repository": "https://github.com/pwwang/pipen"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ae43116c5f7096a22469ba248b1bc2e4c47ecc44ea4bd880e19709b913c298df",
"md5": "339238bfba60728c857b40dc697fd2b0",
"sha256": "aa2322a82c7f45267f8bc686215cf56aa2755f578b16a053e846e286cd155268"
},
"downloads": -1,
"filename": "pipen-0.14.6-py3-none-any.whl",
"has_sig": false,
"md5_digest": "339238bfba60728c857b40dc697fd2b0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.9",
"size": 47519,
"upload_time": "2024-04-05T22:43:22",
"upload_time_iso_8601": "2024-04-05T22:43:22.829164Z",
"url": "https://files.pythonhosted.org/packages/ae/43/116c5f7096a22469ba248b1bc2e4c47ecc44ea4bd880e19709b913c298df/pipen-0.14.6-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "8e468ab4c05eba161d1155765c4419aaa181e94b9391ce32558bdf7ba40371d6",
"md5": "3dd89f064fb7898a92e934bbeb7d8884",
"sha256": "ce118feec0574e301dbfcf714ac67750d6e333b9f2a4452d64d2eb457e09ef02"
},
"downloads": -1,
"filename": "pipen-0.14.6.tar.gz",
"has_sig": false,
"md5_digest": "3dd89f064fb7898a92e934bbeb7d8884",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.9",
"size": 42419,
"upload_time": "2024-04-05T22:43:25",
"upload_time_iso_8601": "2024-04-05T22:43:25.369431Z",
"url": "https://files.pythonhosted.org/packages/8e/46/8ab4c05eba161d1155765c4419aaa181e94b9391ce32558bdf7ba40371d6/pipen-0.14.6.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-05 22:43:25",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "pwwang",
"github_project": "pipen",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"tox": true,
"lcname": "pipen"
}
