# pipeline-flow
``pipeline-flow`` is a lightweight, scalable and extensible platform designed for managing ELT, ETL and ETLT data pipelines.
Being platform agnostic, it can run anywhere with a Python environment, offering a simple and yet flexible
solution for managing data workflows.
Ideal for small to medium-size data workflows without the overhead of full-scale orchestration tools,
``pipeline-flow`` makes building data pipelines simple and accessible.
With its YAML-based configuration and plugin-based architecture, you can easily define and run your data pipelines with minimal effort and maximum flexibility, being able to extend its functionality to various engines as needed. Whether you are using Spark, Polaris or any other data processing engine, you can easily integrate it with ``pipeline-flow``.
We recommend you to visit our [documentation](https://pipeline-flow.readthedocs.io/en/latest/index.html)
## Features
- YAML config based
- Plugin architecture based
- Supports ETL, ELT and ETL data pipelines.
- YAML suports env variables, local variables and secrets for re-usability and consistency.
## Installation
pipeline-flow is available on PyPI and can be installed using pip or poetry. To install using pip, run:
```bash
pip install pipeline-flow # or better use poetry
```
Make sure you know how to get started, [check out our docs](https://pipeline-flow.readthedocs.io/en/latest/pages/intro/quick_start.html)
## Contributing
See [Contributing Guide](CONTRIBUTING.md) for more details.
Raw data
{
"_id": null,
"home_page": "http://pipeline-flow.readthedocs.io/",
"name": "pipeline-flow",
"maintainer": "Jakub Pulaczewski",
"docs_url": null,
"requires_python": "<4.0,>=3.12",
"maintainer_email": "jakub.pulaczewski@example.com",
"keywords": "ETL, ELT, ETLT, data pipelines, plugins, asyncio, data engineering",
"author": "Jakub Pulaczewski",
"author_email": "jakub.pulaczewski@example.com",
"download_url": "https://files.pythonhosted.org/packages/fb/bc/a0cf739ec350aea48d0d3f2decc86d12e26ea9373d8296d922590246b427/pipeline_flow-1.0.7.tar.gz",
"platform": null,
"description": "# pipeline-flow\n\n``pipeline-flow`` is a lightweight, scalable and extensible platform designed for managing ELT, ETL and ETLT data pipelines.\nBeing platform agnostic, it can run anywhere with a Python environment, offering a simple and yet flexible\nsolution for managing data workflows.\n\n\n\nIdeal for small to medium-size data workflows without the overhead of full-scale orchestration tools, \n``pipeline-flow`` makes building data pipelines simple and accessible.\n\nWith its YAML-based configuration and plugin-based architecture, you can easily define and run your data pipelines with minimal effort and maximum flexibility, being able to extend its functionality to various engines as needed. Whether you are using Spark, Polaris or any other data processing engine, you can easily integrate it with ``pipeline-flow``.\n\nWe recommend you to visit our [documentation](https://pipeline-flow.readthedocs.io/en/latest/index.html)\n\n\n## Features\n\n- YAML config based\n- Plugin architecture based\n- Supports ETL, ELT and ETL data pipelines.\n- YAML suports env variables, local variables and secrets for re-usability and consistency.\n\n\n## Installation\npipeline-flow is available on PyPI and can be installed using pip or poetry. To install using pip, run:\n\n\n```bash\npip install pipeline-flow # or better use poetry\n```\n\nMake sure you know how to get started, [check out our docs](https://pipeline-flow.readthedocs.io/en/latest/pages/intro/quick_start.html)\n\n## Contributing\nSee [Contributing Guide](CONTRIBUTING.md) for more details.",
"bugtrack_url": null,
"license": "BSD-3-Clause",
"summary": "A framework for building ETL, ELT and ETLT data pipelines dynamically using plugins.",
"version": "1.0.7",
"project_urls": {
"Homepage": "http://pipeline-flow.readthedocs.io/",
"Repository": "https://github.com/jakubpulaczewski/pipeline-flow"
},
"split_keywords": [
"etl",
" elt",
" etlt",
" data pipelines",
" plugins",
" asyncio",
" data engineering"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "14e3c05db9d69a9af8c1f902a9a713d73a641671a81680ccf8d2eb51ee092052",
"md5": "41a148e6000c6a3cd74c5faf48de3be5",
"sha256": "0e929507479b2ed912c83ea02fcc0fc5db8b307a3fa2c491ef0ad7a8943de398"
},
"downloads": -1,
"filename": "pipeline_flow-1.0.7-py3-none-any.whl",
"has_sig": false,
"md5_digest": "41a148e6000c6a3cd74c5faf48de3be5",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.12",
"size": 30573,
"upload_time": "2025-07-15T19:27:32",
"upload_time_iso_8601": "2025-07-15T19:27:32.619527Z",
"url": "https://files.pythonhosted.org/packages/14/e3/c05db9d69a9af8c1f902a9a713d73a641671a81680ccf8d2eb51ee092052/pipeline_flow-1.0.7-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "fbbca0cf739ec350aea48d0d3f2decc86d12e26ea9373d8296d922590246b427",
"md5": "86713d35c4e293337810dc82a452773f",
"sha256": "3f3077c0d5498b39ea2ac9c300bfb79d8b2b58561c666a35bc6475a2de3bda79"
},
"downloads": -1,
"filename": "pipeline_flow-1.0.7.tar.gz",
"has_sig": false,
"md5_digest": "86713d35c4e293337810dc82a452773f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.12",
"size": 19571,
"upload_time": "2025-07-15T19:27:33",
"upload_time_iso_8601": "2025-07-15T19:27:33.918156Z",
"url": "https://files.pythonhosted.org/packages/fb/bc/a0cf739ec350aea48d0d3f2decc86d12e26ea9373d8296d922590246b427/pipeline_flow-1.0.7.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-15 19:27:33",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "jakubpulaczewski",
"github_project": "pipeline-flow",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "pipeline-flow"
}
JSON
