::
O
( ))
( )
( ) )
( ) )
( ) )
( )
( )
( )
──┐ ──┐
/ │ / │
/ │ / │
/ │ / │
/ │ / │
/ │/ │
/ │ │
┌─────────┴────────┴┐
│ │
│ ┌──┐ ┌┘ ┌──┐
│ │ │ ┌─┐ │ │ │ ┌──┐
│ └──┘ │ │ └┬─┴──┴──┴──┴───
│ │ │ │* * * * * * * O
└───────────┴─┴─────┘---------------
____ _ ___ ______ __
/ __ \(_)___ ___ / (_)___ ___ / ____/___ ______/ /_____ _______ __
/ /_/ / / __ \/ _ \/ / / __ \/ _ \ / /_ / __ `/ ___/ __/ __ \/ ___/ / / /
/ ____/ / /_/ / __/ / / / / / __/ / __/ / /_/ / /__/ /_/ /_/ / / / /_/ /
/_/ /_/ .___/\___/_/_/_/ /_/\___/ /_/ \__,_/\___/\__/\____/_/ \__, /
/_/ /____/
Pipeline Factory is a Python server framework to define, manage, schedule and execute custom defined code pipelines and series of pipeline across multiple worker processes.
- Based on Fastapi and Python multiprocessing
- Define pipelines of action steps to process data, make api calls, generate artifacts etc.
- Resumable pipelines where steps can define recovery action to avoid redoing work
- Run series of pipelines based on state/ results of previous pipelines
- Cron schedule tasks to auto run or resume pipelines
- Tasks that run pipelines execute on a worker process pool
- Add python callables as custom tasks, to extend pipeline execution capabilities or support execution of arbitrary work on worker pool
Installation
============
::
pip install pipeline-factory
Example usage
=============
`Example app <https://github.com/izo0x90/pipeline-factory/tree/main/example_app>`_
Raw data
{
"_id": null,
"home_page": null,
"name": "pipeline-factory",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "task, pipeline, schedule, workers, async, pipeline framework",
"author": null,
"author_email": "Hristo Gueorguiev <hristo.gueorguiev@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/60/db/3b438ce0a156bf9824c2b53c0177be7f9200536a96d10b00e38c21105229/pipeline_factory-0.0.1a0.tar.gz",
"platform": null,
"description": "::\n \n O \n ( )) \n ( ) \n ( ) ) \n ( ) ) \n ( ) ) \n ( ) \n ( ) \n ( ) \n \u2500\u2500\u2510 \u2500\u2500\u2510 \n / \u2502 / \u2502 \n / \u2502 / \u2502 \n / \u2502 / \u2502 \n / \u2502 / \u2502 \n / \u2502/ \u2502 \n / \u2502 \u2502 \n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2510 \n \u2502 \u2502 \n \u2502 \u250c\u2500\u2500\u2510 \u250c\u2518 \u250c\u2500\u2500\u2510 \n \u2502 \u2502 \u2502 \u250c\u2500\u2510 \u2502 \u2502 \u2502 \u250c\u2500\u2500\u2510 \n \u2502 \u2514\u2500\u2500\u2518 \u2502 \u2502 \u2514\u252c\u2500\u2534\u2500\u2500\u2534\u2500\u2500\u2534\u2500\u2500\u2534\u2500\u2500\u2500 \n \u2502 \u2502 \u2502 \u2502* * * * * * * O \n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2518--------------- \n ____ _ ___ ______ __ \n / __ \\(_)___ ___ / (_)___ ___ / ____/___ ______/ /_____ _______ __ \n / /_/ / / __ \\/ _ \\/ / / __ \\/ _ \\ / /_ / __ `/ ___/ __/ __ \\/ ___/ / / / \n / ____/ / /_/ / __/ / / / / / __/ / __/ / /_/ / /__/ /_/ /_/ / / / /_/ / \n /_/ /_/ .___/\\___/_/_/_/ /_/\\___/ /_/ \\__,_/\\___/\\__/\\____/_/ \\__, / \n /_/ /____/ \n \n\nPipeline Factory is a Python server framework to define, manage, schedule and execute custom defined code pipelines and series of pipeline across multiple worker processes.\n\n- Based on Fastapi and Python multiprocessing\n- Define pipelines of action steps to process data, make api calls, generate artifacts etc.\n- Resumable pipelines where steps can define recovery action to avoid redoing work\n- Run series of pipelines based on state/ results of previous pipelines\n- Cron schedule tasks to auto run or resume pipelines\n- Tasks that run pipelines execute on a worker process pool\n- Add python callables as custom tasks, to extend pipeline execution capabilities or support execution of arbitrary work on worker pool\n\nInstallation\n============\n::\n\n pip install pipeline-factory\n\nExample usage\n=============\n`Example app <https://github.com/izo0x90/pipeline-factory/tree/main/example_app>`_\n\n",
"bugtrack_url": null,
"license": "MIT License",
"summary": "Python server framework to define, manage, schedule and execute custom defined code pipelines and series of pipeline across multiple worker processes",
"version": "0.0.1a0",
"project_urls": {
"Issues": "https://github.com/izo0x90/pipeline-factory/issues",
"Repository": "https://github.com/izo0x90/pipeline-factory"
},
"split_keywords": [
"task",
" pipeline",
" schedule",
" workers",
" async",
" pipeline framework"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "de277709f4d38b20c2baebc5ac9f76573f60c348dd7913a94f3bb17b7724099a",
"md5": "5405b7245ae5492f65c28f41e1e60e27",
"sha256": "081bb2996c4cc36da018b5347e7e6201b64d0137f9ef390726d8b464f0174d0e"
},
"downloads": -1,
"filename": "pipeline_factory-0.0.1a0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5405b7245ae5492f65c28f41e1e60e27",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 18540,
"upload_time": "2024-07-23T21:12:49",
"upload_time_iso_8601": "2024-07-23T21:12:49.744594Z",
"url": "https://files.pythonhosted.org/packages/de/27/7709f4d38b20c2baebc5ac9f76573f60c348dd7913a94f3bb17b7724099a/pipeline_factory-0.0.1a0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "60db3b438ce0a156bf9824c2b53c0177be7f9200536a96d10b00e38c21105229",
"md5": "c052732cb46bacfa62e1d557367e4acf",
"sha256": "395b4f23d9faa4556e49bd45c7b066cb843b6a87c03f33d2981bdc1937731ad0"
},
"downloads": -1,
"filename": "pipeline_factory-0.0.1a0.tar.gz",
"has_sig": false,
"md5_digest": "c052732cb46bacfa62e1d557367e4acf",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 19744,
"upload_time": "2024-07-23T21:12:51",
"upload_time_iso_8601": "2024-07-23T21:12:51.024009Z",
"url": "https://files.pythonhosted.org/packages/60/db/3b438ce0a156bf9824c2b53c0177be7f9200536a96d10b00e38c21105229/pipeline_factory-0.0.1a0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-07-23 21:12:51",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "izo0x90",
"github_project": "pipeline-factory",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "pipeline-factory"
}
JSON
