<div align="center">
<img src="https://gitlab.com/outflow-project/outflow/-/raw/develop/docs/sections/images/logo.svg" width="500" style="max-width: 500px;">
</div>
<div align="center">
<a href="https://pypi.org/project/outflow/">
<img src="https://img.shields.io/pypi/pyversions/outflow.svg" alt="python">
</a>
<a href="https://pypi.org/project/outflow/">
<img alt="PyPI" src="https://img.shields.io/pypi/v/outflow">
</a>
<a href="https://gitlab.com/outflow-project/outflow/-/pipelines/master/latest">
<img alt="pipeline status" src="https://gitlab.com/outflow-project/outflow/badges/master/pipeline.svg" />
</a>
<a href="https://gitlab.com/outflow-project/outflow/-/pipelines/master/latest">
<img alt="coverage report" src="https://gitlab.com/outflow-project/outflow/badges/master/coverage.svg" />
</a>
<a href=https://github.com/ambv/black>
<img src="https://img.shields.io/badge/code%20style-black-000000.svg">
</a>
<a href='https://docs.outflow.dev'>
<img src='https://readthedocs.org/projects/outflow/badge/?version=latest' alt='Documentation Status' />
</a>
<a href="https://pypi.python.org/pypi/outflow">
<img src="https://img.shields.io/pypi/l/outflow.svg" alt="license" />
</a>
<a href="https://discord.outflow.dev/">
<img src="https://img.shields.io/badge/discord-support-7389D8?logo=discord&style=flat&logoColor=fff" alt="chat-discord-support" />
</a>
</div>
Outflow is a framework that helps you build and run task workflows.
The api is as simple as possible while still giving the user full control over the definition and execution of the
workflows.
**Feature highlight :**
- Simple but powerful API
- Support for **parallelized and distributed execution**
- Centralized **command line interface** for your pipeline commands
- Integrated **database** access, sqlalchemy models and alembic migrations
- Executions and exceptions logging for **tracability**
- Strict type and input/output checking for a **robust** pipeline
Check out our [documentation][outflow readthedocs] for more information.
[outflow readthedocs]: https://docs.outflow.dev
# Installing
Install and update using [pip](https://pip.pypa.io/en/stable/):
```
pip install -U outflow
```
# Quick start
## One file starter
First, create a `pipeline.py` script:
```python
# -- pipeline.py
from outflow.core.commands import Command, RootCommand
from outflow.core.pipeline import Pipeline
from outflow.core.tasks import as_task
# with the as_task decorator, the function will be automatically converted into a Task subclass
# the signature of the function, including the return type, is used to determine task inputs and outputs
@as_task
def GetValues() -> {"word1": str, "word2": str}:
return {"word1": "Hello", "word2": "world!"}
# default values can also be used as inputs
@as_task
def Concatenate(word1: str, word2: str) -> {"result": str}:
result = f"{word1} {word2}"
return result # you can return the value directly if your task has only one output
# A task can have side-effects and returns nothing
@as_task
def PrintResult(result: str):
print(result)
@RootCommand.subcommand()
class HelloWorld(Command):
def setup_tasks(self):
# instantiate the tasks
get_values = GetValues()
concatenate = Concatenate(word2="outflow!") # you can override task inputs value at instantiation
print_result = PrintResult()
# build the workflow
get_values >> concatenate >> print_result
# instantiate and run the pipeline
with Pipeline(
root_directory=None,
settings_module="outflow.core.pipeline.default_settings",
force_dry_run=True,
) as pipeline:
result = pipeline.run()
```
and run your first Outflow pipeline:
```
$ python pipeline.py hello_world
```
## A robust, configurable and well-organized pipeline
You had a brief overview of Outflow's features and you want to go further. Outflow offers command line tools to help you to start your pipeline project.
First, we will need to auto-generate the pipeline structure -- a collection of files including the pipeline settings, the database and the cluster configuration, etc.
```
$ python -m outflow management create pipeline my_pipeline
```
Then, we have to create a plugin -- a dedicated folder regrouping the commands, the tasks as well as the description of the database (the models)
```
$ python -m outflow management create plugin my_namespace.my_plugin --plugin_dir my_pipeline/plugins/my_plugin
```
In the my_pipeline/settings.py file, add your new plugin to the plugin list:
```python
PLUGINS = [
'outflow.management',
'my_namespace.my_plugin',
]
```
and run the following command:
```
$ python ./my_pipeline/manage.py my_plugin
```
You'll see the following output on the command line:
```
* outflow.core.pipeline.pipeline - pipeline.py:325 - INFO - No cluster config found in configuration file, running in a local cluster
* my_namespace.my_plugin.commands - commands.py:49 - INFO - Hello from my_plugin
```
Your pipeline is up and running. You can now start adding new tasks and commands.
# Contributing
For guidance on setting up a development environment and how to make a contribution to Outflow, see the [contributing guidelines](https://gitlab.lam.fr/CONCERTO/outflow/-/blob/master/CONTRIBUTING.md).
Raw data
{
"_id": null,
"home_page": "https://outflow.dev",
"name": "outflow",
"maintainer": null,
"docs_url": null,
"requires_python": "<4,>=3.9",
"maintainer_email": null,
"keywords": "outflow, pipeline, distributed, parallel, workflow, data science, slurm",
"author": "Gregoire Duvauchelle",
"author_email": "gregoire.duvauchelle@lam.fr",
"download_url": "https://files.pythonhosted.org/packages/ff/53/2fc37b558d53f0e5f1544b6bdefe9ebf7181356a6fb4dd634edddd9a942c/outflow-0.9.1.tar.gz",
"platform": null,
"description": "<div align=\"center\">\n <img src=\"https://gitlab.com/outflow-project/outflow/-/raw/develop/docs/sections/images/logo.svg\" width=\"500\" style=\"max-width: 500px;\">\n</div>\n\n<div align=\"center\">\n\n<a href=\"https://pypi.org/project/outflow/\">\n <img src=\"https://img.shields.io/pypi/pyversions/outflow.svg\" alt=\"python\">\n</a>\n\n<a href=\"https://pypi.org/project/outflow/\">\n <img alt=\"PyPI\" src=\"https://img.shields.io/pypi/v/outflow\">\n</a>\n\n<a href=\"https://gitlab.com/outflow-project/outflow/-/pipelines/master/latest\">\n <img alt=\"pipeline status\" src=\"https://gitlab.com/outflow-project/outflow/badges/master/pipeline.svg\" />\n</a>\n\n<a href=\"https://gitlab.com/outflow-project/outflow/-/pipelines/master/latest\">\n <img alt=\"coverage report\" src=\"https://gitlab.com/outflow-project/outflow/badges/master/coverage.svg\" />\n</a>\n\n<a href=https://github.com/ambv/black>\n <img src=\"https://img.shields.io/badge/code%20style-black-000000.svg\">\n</a>\n\n<a href='https://docs.outflow.dev'>\n <img src='https://readthedocs.org/projects/outflow/badge/?version=latest' alt='Documentation Status' />\n</a>\n\n<a href=\"https://pypi.python.org/pypi/outflow\">\n <img src=\"https://img.shields.io/pypi/l/outflow.svg\" alt=\"license\" />\n</a>\n\n<a href=\"https://discord.outflow.dev/\">\n <img src=\"https://img.shields.io/badge/discord-support-7389D8?logo=discord&style=flat&logoColor=fff\" alt=\"chat-discord-support\" />\n</a>\n\n</div>\n\nOutflow is a framework that helps you build and run task workflows.\n\nThe api is as simple as possible while still giving the user full control over the definition and execution of the\nworkflows.\n\n**Feature highlight :**\n\n- Simple but powerful API\n- Support for **parallelized and distributed execution**\n- Centralized **command line interface** for your pipeline commands\n- Integrated **database** access, sqlalchemy models and alembic migrations\n- Executions and exceptions logging for **tracability**\n- Strict type and input/output checking for a **robust** pipeline\n\nCheck out our [documentation][outflow readthedocs] for more information.\n\n[outflow readthedocs]: https://docs.outflow.dev\n\n# Installing\n\nInstall and update using [pip](https://pip.pypa.io/en/stable/):\n\n```\npip install -U outflow\n```\n\n# Quick start\n\n## One file starter\n\nFirst, create a `pipeline.py` script:\n\n```python\n# -- pipeline.py\n\nfrom outflow.core.commands import Command, RootCommand\nfrom outflow.core.pipeline import Pipeline\nfrom outflow.core.tasks import as_task\n\n# with the as_task decorator, the function will be automatically converted into a Task subclass\n# the signature of the function, including the return type, is used to determine task inputs and outputs\n@as_task\ndef GetValues() -> {\"word1\": str, \"word2\": str}:\n return {\"word1\": \"Hello\", \"word2\": \"world!\"}\n\n# default values can also be used as inputs\n@as_task\ndef Concatenate(word1: str, word2: str) -> {\"result\": str}:\n result = f\"{word1} {word2}\"\n return result # you can return the value directly if your task has only one output\n\n# A task can have side-effects and returns nothing\n@as_task\ndef PrintResult(result: str):\n print(result)\n\n@RootCommand.subcommand()\nclass HelloWorld(Command):\n def setup_tasks(self):\n # instantiate the tasks\n get_values = GetValues()\n concatenate = Concatenate(word2=\"outflow!\") # you can override task inputs value at instantiation\n print_result = PrintResult()\n\n # build the workflow\n get_values >> concatenate >> print_result\n\n\n# instantiate and run the pipeline\nwith Pipeline(\n root_directory=None,\n settings_module=\"outflow.core.pipeline.default_settings\",\n force_dry_run=True,\n) as pipeline:\n result = pipeline.run()\n\n```\n\nand run your first Outflow pipeline:\n\n```\n$ python pipeline.py hello_world\n```\n\n## A robust, configurable and well-organized pipeline\n\nYou had a brief overview of Outflow's features and you want to go further. Outflow offers command line tools to help you to start your pipeline project.\n\nFirst, we will need to auto-generate the pipeline structure -- a collection of files including the pipeline settings, the database and the cluster configuration, etc.\n\n```\n$ python -m outflow management create pipeline my_pipeline\n```\n\nThen, we have to create a plugin -- a dedicated folder regrouping the commands, the tasks as well as the description of the database (the models)\n\n```\n$ python -m outflow management create plugin my_namespace.my_plugin --plugin_dir my_pipeline/plugins/my_plugin\n```\n\nIn the my_pipeline/settings.py file, add your new plugin to the plugin list:\n\n```python\nPLUGINS = [\n 'outflow.management',\n 'my_namespace.my_plugin',\n]\n```\n\nand run the following command:\n\n```\n$ python ./my_pipeline/manage.py my_plugin\n```\n\nYou'll see the following output on the command line:\n\n```\n * outflow.core.pipeline.pipeline - pipeline.py:325 - INFO - No cluster config found in configuration file, running in a local cluster\n * my_namespace.my_plugin.commands - commands.py:49 - INFO - Hello from my_plugin\n```\n\nYour pipeline is up and running. You can now start adding new tasks and commands.\n\n# Contributing\n\nFor guidance on setting up a development environment and how to make a contribution to Outflow, see the [contributing guidelines](https://gitlab.lam.fr/CONCERTO/outflow/-/blob/master/CONTRIBUTING.md).\n",
"bugtrack_url": null,
"license": "CECILL-2.1",
"summary": "Outflow is a framework that helps you create and execute sequential, parallel as well as distributed task workflows.",
"version": "0.9.1",
"project_urls": {
"Documentation": "https://docs.outflow.dev",
"Homepage": "https://outflow.dev",
"Repository": "https://gitlab.com/outflow-project/outflow",
"Tracker": "https://gitlab.com/outflow-project/outflow/-/issues"
},
"split_keywords": [
"outflow",
" pipeline",
" distributed",
" parallel",
" workflow",
" data science",
" slurm"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c5c108dd6ff3487f06c356686d31433b1f0ec8ada4edbace56299eea125e25ed",
"md5": "39e32adcb8bc08c3bfca0c78588efc72",
"sha256": "8f4745a10488dc93774bc48eccc10d23bc85dea5bff78ec28358fea85846e32d"
},
"downloads": -1,
"filename": "outflow-0.9.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "39e32adcb8bc08c3bfca0c78588efc72",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4,>=3.9",
"size": 133977,
"upload_time": "2025-08-28T13:24:37",
"upload_time_iso_8601": "2025-08-28T13:24:37.388993Z",
"url": "https://files.pythonhosted.org/packages/c5/c1/08dd6ff3487f06c356686d31433b1f0ec8ada4edbace56299eea125e25ed/outflow-0.9.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ff532fc37b558d53f0e5f1544b6bdefe9ebf7181356a6fb4dd634edddd9a942c",
"md5": "7ca508b9f2585ce66e9ae08bba0f084a",
"sha256": "5c319a21bb11654a4af971bb306a4f24ff469be849c6d2eea18936415aedbb04"
},
"downloads": -1,
"filename": "outflow-0.9.1.tar.gz",
"has_sig": false,
"md5_digest": "7ca508b9f2585ce66e9ae08bba0f084a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4,>=3.9",
"size": 89507,
"upload_time": "2025-08-28T13:24:38",
"upload_time_iso_8601": "2025-08-28T13:24:38.806635Z",
"url": "https://files.pythonhosted.org/packages/ff/53/2fc37b558d53f0e5f1544b6bdefe9ebf7181356a6fb4dd634edddd9a942c/outflow-0.9.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-28 13:24:38",
"github": false,
"gitlab": true,
"bitbucket": false,
"codeberg": false,
"gitlab_user": "outflow-project",
"gitlab_project": "outflow",
"lcname": "outflow"
}
JSON