| Name | FlowerPower JSON |
| Version |
0.21.0
JSON |
| download |
| home_page | None |
| Summary | A simple workflow framework for building and managing data processing pipelines |
| upload_time | 2025-09-01 21:36:33 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.11 |
| license | None |
| keywords |
hamilton
workflow
pipeline
scheduler
dask
ray
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<div align="center">
<h1>FlowerPower 🌸 - Build & Orchestrate Data Pipelines</h1>
<h3>Simple Workflow Framework - Hamilton = FlowerPower</h3>
<img src="./image.png" alt="FlowerPower Logo" width="400" height="300">
</div>
[](https://pypi.org/project/flowerpower/) <!-- Placeholder -->
[](https://github.com/legout/flowerpower/blob/main/LICENSE)
[](https://deepwiki.com/legout/flowerpower)
[](https://legout.github.io/flowerpower/)
**FlowerPower** is a Python framework designed for building, configuring, and executing data processing pipelines with ease and flexibility. It promotes a modular, configuration-driven approach, allowing you to focus on your pipeline logic while FlowerPower handles the orchestration.
It leverages the [Hamilton](https://github.com/apache/hamilton) library for defining dataflows in a clean, functional way within your Python pipeline scripts. Pipelines are defined in Python modules and configured using YAML files, making it easy to manage and understand your data workflows.
FlowerPower provides a unified project interface that makes it easy to work with pipeline execution. It also provides a web UI (Hamilton UI) for monitoring and managing your pipelines.
FlowerPower is designed to be extensible, allowing you to easily add custom I/O plugins and adapt to different deployment scenarios. This flexibility makes it suitable for a wide range of data processing tasks, from simple ETL jobs to complex data workflows.
## ✨ Key Features
* **Modular Pipeline Design:** Thanks to [Hamilton](https://github.com/apache/hamilton), you can define your data processing logic in Python modules, using functions as nodes in a directed acyclic graph (DAG).
* **Configuration-Driven:** Define pipeline parameters, execution logic, and scheduling declaratively using simple YAML files.
* **Extensible I/O Plugins:** Connect to various data sources and destinations (CSV, JSON, Parquet, DeltaTable, DuckDB, PostgreSQL, MySQL, MSSQL, Oracle, MQTT, SQLite, and more).
* **Unified Project Interface:** Interact with your pipelines via:
* **FlowerPowerProject API:** A unified interface for pipeline execution, supporting both `RunConfig` objects and flexible `**kwargs` overrides.
* **Command Line Interface (CLI):** For running, managing, and inspecting pipelines, with enhanced `run` command capabilities.
* **Web UI:** A graphical interface for monitoring and managing pipelines and schedules. ([Hamilton UI](https://hamilton.dagworks.io/en/latest/hamilton-ui/ui/))
* **Filesystem Abstraction:** Simplified file handling with support for local and remote filesystems (e.g., S3, GCS).
## 📦 Installation
We recommend using [uv](https://github.com/astral-sh/uv) for installing FlowerPower and managing your project environments. `uv` is an extremely fast Python package installer and resolver.
```bash
# Create and activate a virtual environment (recommended)
uv venv
source .venv/bin/activate # Or .\.venv\Scripts\activate on Windows
# Install FlowerPower
uv pip install flowerpower
# Optional: Install additional dependencies for specific features
uv pip install flowerpower[io] # For I/O plugins (CSV, JSON, Parquet, DeltaTable, DuckDB, PostgreSQL, MySQL, MSSQL, Oracle, SQLite)
uv pip install flowerpower[ui] # For Hamilton UI
uv pip install flowerpower[all] # Install all optional dependencies
```
*(Note: Specify required Python versions if known, e.g., Python 3.8+)*
## 🚀 Getting Started
Let's build a simple "Hello World" pipeline.
### 1. Initialize Your Project:
You can quickly set up the standard FlowerPower project structure using the CLI or Python.
**Using the CLI:**
Navigate to your desired parent directory and run:
```bash
flowerpower init --name hello-flowerpower-project
```
**Using Python:**
Alternatively, you can initialize programmatically:
```python
from flowerpower import FlowerPowerProject
# Initialize a new project
project = FlowerPowerProject.init(
name='hello-flowerpower-project',
)
```
This will create a `hello-flowerpower-project` directory with the necessary `conf/` and `pipelines/` subdirectories and default configuration files.
```
hello-flowerpower-project/
├── conf/
│ ├── project.yml
│ └── pipelines/
└── pipelines/
```
Now, navigate into your new project directory:
```bash
cd hello-flowerpower-project
```
**Configure Project (`conf/project.yml`):**
Open `conf/project.yml` and define your project name:
```yaml
name: hello-flowerpower
# adapter: ... # Optional adapter configurations (e.g., Hamilton Tracker, MLflow), see `conf/project.yml` for details
```
### 2. Create Your Pipeline
You can create a new pipeline using the CLI or programmatically.
**Using the CLI:**
```bash
flowerpower pipeline new hello_world
```
**Using Python:**
You can create pipelines programmatically using the FlowerPowerProject interface:
```python
from flowerpower import FlowerPowerProject
# Load the project
project = FlowerPowerProject.load('.')
# Create a new pipeline
project.pipeline_manager.new(name='hello_world')
```
This will create a new file `hello_world.py` in the `pipelines/` directory and a corresponding configuration file `hello_world.yml` in `conf/pipelines/`.
**Implement Pipeline (`pipelines/hello_world.py`):**
Open `pipelines/hello_world.py` and write your pipeline logic using Python and Hamilton. FlowerPower makes configuration easily accessible.
```python
# FlowerPower pipeline hello_world.py
# Created on 2025-05-03 22:34:09
####################################################################################################
# Import necessary libraries
# NOTE: Remove or comment out imports that are not used in the pipeline
from hamilton.function_modifiers import parameterize
from pathlib import Path
from flowerpower.cfg import Config
####################################################################################################
# Load pipeline parameters. Do not modify this section.
PARAMS = Config.load(
Path(__file__).parents[1], pipeline_name="hello_world"
).pipeline.h_params
####################################################################################################
# Helper functions.
# This functions have to start with an underscore (_).
####################################################################################################
# Pipeline functions
@parameterize(**PARAMS.greeting_message) # Inject 'message' from params
def greeting_message(message: str) -> str:
"""Provides the greeting part."""
return f"{message},"
@parameterize(**PARAMS.target_name) # Inject 'name' from params
def target_name(name: str) -> str:
"""Provides the target name."""
return f"{name}!"
def full_greeting(greeting_message: str, target_name: str) -> str:
"""Combines the greeting and target."""
print(f"Generating greeting: {greeting_message} {target_name}")
return f"{greeting_message} {target_name}"
# You can add more complex Hamilton functions here...
```
**Configure Pipeline (`conf/pipelines/hello_world.yml`):**
Open `conf/pipelines/hello_world.yml` and specify parameters, run configurations, and scheduling for your pipeline.
```yaml
# adapter: ... # Pipeline-specific adapter overrides
params: # Parameters accessible in your Python code
greeting_message:
message: "Hello"
target_name:
name: "World"
run: # How to execute the pipeline
final_vars: # Specify the desired output(s) from your Hamilton DAG
- full_greeting
# inputs: # Optional: Specify input variables to the pipeline
# message: "Hello"
# config: ... # Runtime configuration overrides for Hamilton
# executor: ... # Execution backend (e.g., threadpool, multiprocessing)
```
### 3. Run Your Pipeline 🏃♀️
FlowerPower allows you to execute your pipelines synchronously, with flexible configuration options.
#### Synchronous Execution:
For quick testing or local runs, you can execute your pipeline synchronously. This is useful for debugging or running pipelines in a local environment.
* **Via CLI:**
The `flowerpower pipeline run` command now supports `RunConfig` objects (via file path or JSON string) and direct `**kwargs` for overriding.
```bash
# Basic pipeline execution
flowerpower pipeline run hello_world
# Run with individual parameters (kwargs)
flowerpower pipeline run hello_world --inputs '{"greeting_message": "Hi", "target_name": "FlowerPower"}' --final-vars '["full_greeting"]' --log-level DEBUG
# Run using a RunConfig from a YAML file
# Assuming you have a run_config.yaml like:
# inputs:
# greeting_message: "Hola"
# target_name: "Amigo"
# log_level: "INFO"
flowerpower pipeline run hello_world --run-config ./run_config.yaml
# Run using a RunConfig provided as a JSON string
flowerpower pipeline run hello_world --run-config '{"inputs": {"greeting_message": "Bonjour", "target_name": "Monde"}, "log_level": "INFO"}'
# Mixing RunConfig with individual parameters (kwargs overrides RunConfig)
# This will run with log_level="DEBUG" and inputs={"greeting_message": "Howdy", "target_name": "Partner"}
flowerpower pipeline run hello_world --run-config '{"inputs": {"greeting_message": "Original", "target_name": "Value"}, "log_level": "INFO"}' --inputs '{"greeting_message": "Howdy", "target_name": "Partner"}' --log-level DEBUG
```
* **Via Python:**
The `run` methods (`FlowerPowerProject.run`, `PipelineManager.run`) now primarily accept a `RunConfig` object, but also allow individual parameters to be passed via `**kwargs` which override `RunConfig` attributes.
```python
from flowerpower import FlowerPowerProject
from flowerpower.cfg.pipeline.run import RunConfig
from flowerpower.cfg.pipeline.builder import RunConfigBuilder
# Load the project
project = FlowerPowerProject.load('.')
# Basic execution
result = project.run('hello_world')
print(result)
# Using individual parameters (kwargs)
result = project.run(
'hello_world',
inputs={"greeting_message": "Hi", "target_name": "FlowerPower"},
final_vars=["full_greeting"],
log_level="DEBUG"
)
print(result)
# Using RunConfig directly
config = RunConfig(
inputs={"greeting_message": "Aloha", "target_name": "World"},
final_vars=["full_greeting"],
log_level="INFO"
)
result = project.run('hello_world', run_config=config)
print(result)
# Using RunConfigBuilder (recommended)
config = (
RunConfigBuilder(pipeline_name='hello_world')
.with_inputs({"greeting_message": "Greetings", "target_name": "Earth"})
.with_final_vars(["full_greeting"])
.with_log_level("DEBUG")
.with_retries(max_attempts=3, delay=1.0)
.build()
)
result = project.run('hello_world', run_config=config)
print(result)
# Mixing RunConfig with individual parameters (kwargs overrides RunConfig)
base_config = RunConfigBuilder().with_log_level("INFO").build()
result = project.run(
'hello_world',
run_config=base_config,
inputs={"greeting_message": "Howdy", "target_name": "Partner"}, # Overrides inputs in base_config
log_level="DEBUG" # Overrides log_level in base_config
)
print(result)
```
## ⚙️ Configuration Overview
FlowerPower uses a layered configuration system:
* **`conf/project.yml`:** Defines global settings for your project, including integrated `adapter`s (like Hamilton Tracker, MLflow, etc.).
* **`conf/pipelines/*.yml`:** Each file defines a specific pipeline. It contains:
* `params`: Input parameters for your Hamilton functions.
* `run`: Execution details like target outputs (`final_vars`), Hamilton runtime `config`, and `executor` settings.
* `adapter`: Pipeline-specific overrides for adapter settings.
## 🛠️ Basic Usage
You can interact with FlowerPower pipelines through multiple interfaces:
**Python API (Recommended):**
```python
from flowerpower import FlowerPowerProject
from flowerpower.cfg.pipeline.run import RunConfig
from flowerpower.cfg.pipeline.builder import RunConfigBuilder
# Load the project
project = FlowerPowerProject.load('.')
# Run a pipeline using RunConfig
config = RunConfig(inputs={"greeting_message": "Hello", "target_name": "API"})
result = project.run('hello_world', run_config=config)
print(result)
# Run a pipeline using kwargs
result = project.run('hello_world', inputs={"greeting_message": "Hi", "target_name": "Kwargs"})
print(result)
```
**CLI:**
```bash
# Run a pipeline using RunConfig from a file
# flowerpower pipeline run hello_world --run-config ./path/to/run_config.yaml
# Run a pipeline using kwargs
flowerpower pipeline run hello_world --inputs '{"greeting_message": "CLI", "target_name": "Kwargs"}'
# List all available commands
flowerpower --help
```
## 🔧 Direct Module Usage
While the unified `FlowerPowerProject` interface is recommended for most use cases, you can also use the pipeline module directly for more granular control or when you only need specific functionality.
### Pipeline-Only Usage
If you only need pipeline execution, you can use the `PipelineManager` directly:
```python
from flowerpower.pipeline import PipelineManager
from flowerpower.cfg.pipeline.run import RunConfig
from flowerpower.cfg.pipeline.builder import RunConfigBuilder
# Initialize pipeline manager
pm = PipelineManager(base_dir='.')
# Create a new pipeline
pm.new(name='my_pipeline')
# Run a pipeline synchronously using RunConfig
config = RunConfig(inputs={'param': 'value'}, final_vars=['output_var'])
result = pm.run(name='my_pipeline', run_config=config)
print(result)
# Run a pipeline synchronously using kwargs
result = pm.run(name='my_pipeline', inputs={'param': 'new_value'}, final_vars=['output_var'])
print(result)
# List available pipelines
pipelines = pm.list()
print(f"Available pipelines: {pipelines}")
# Get pipeline information
info = pm.get('my_pipeline')
print(f"Pipeline config: {info}")
# Delete a pipeline
pm.delete('old_pipeline')
```
**When to use Pipeline-only approach:**
- Simple synchronous workflows
- Testing and development
- Lightweight applications with minimal dependencies
**Benefits of FlowerPowerProject vs Direct Usage:**
| Approach | Benefits | Use Cases |
|----------|----------|-----------|
| **FlowerPowerProject** | - Unified interface<br>- Automatic dependency injection<br>- Simplified configuration<br>- Best practices built-in | - Most applications<br>- Rapid development<br>- Full feature integration |
| **Pipeline-only** | - Lightweight<br>- Simple synchronous execution | - Testing<br>- Simple workflows |
## 🖥️ UI
The FlowerPower web UI (Hamilton UI) provides a graphical interface for monitoring and managing your pipelines. It allows you to visualize pipeline runs, schedules, and potentially manage configurations.
```bash
# Start the web UI
flowerpower ui
```
## 📖 Documentation
You can find the full documentation for FlowerPower, including installation instructions, usage examples, and API references, at [https://legout.github.io/flowerpower/](https://legout.github.io/flowerpower/).
## 📜 License
This project is licensed under the MIT License - see the `LICENSE` file for details. (Placeholder - update with actual license)
Raw data
{
"_id": null,
"home_page": null,
"name": "FlowerPower",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.11",
"maintainer_email": null,
"keywords": "hamilton, workflow, pipeline, scheduler, dask, ray",
"author": null,
"author_email": "\"Volker L.\" <ligno.blades@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/ba/e2/eed9f122f3e69e08a37484ab2c274ddb012704d62e792ab4a5271896ffae/flowerpower-0.21.0.tar.gz",
"platform": null,
"description": "<div align=\"center\">\n <h1>FlowerPower \ud83c\udf38 - Build & Orchestrate Data Pipelines</h1>\n <h3>Simple Workflow Framework - Hamilton = FlowerPower</h3>\n <img src=\"./image.png\" alt=\"FlowerPower Logo\" width=\"400\" height=\"300\">\n</div>\n\n\n\n[](https://pypi.org/project/flowerpower/) <!-- Placeholder -->\n[](https://github.com/legout/flowerpower/blob/main/LICENSE)\n[](https://deepwiki.com/legout/flowerpower)\n[](https://legout.github.io/flowerpower/)\n\n\n**FlowerPower** is a Python framework designed for building, configuring, and executing data processing pipelines with ease and flexibility. It promotes a modular, configuration-driven approach, allowing you to focus on your pipeline logic while FlowerPower handles the orchestration.\n\nIt leverages the [Hamilton](https://github.com/apache/hamilton) library for defining dataflows in a clean, functional way within your Python pipeline scripts. Pipelines are defined in Python modules and configured using YAML files, making it easy to manage and understand your data workflows.\nFlowerPower provides a unified project interface that makes it easy to work with pipeline execution. It also provides a web UI (Hamilton UI) for monitoring and managing your pipelines.\nFlowerPower is designed to be extensible, allowing you to easily add custom I/O plugins and adapt to different deployment scenarios. This flexibility makes it suitable for a wide range of data processing tasks, from simple ETL jobs to complex data workflows.\n\n\n## \u2728 Key Features\n\n* **Modular Pipeline Design:** Thanks to [Hamilton](https://github.com/apache/hamilton), you can define your data processing logic in Python modules, using functions as nodes in a directed acyclic graph (DAG).\n* **Configuration-Driven:** Define pipeline parameters, execution logic, and scheduling declaratively using simple YAML files.\n* **Extensible I/O Plugins:** Connect to various data sources and destinations (CSV, JSON, Parquet, DeltaTable, DuckDB, PostgreSQL, MySQL, MSSQL, Oracle, MQTT, SQLite, and more).\n* **Unified Project Interface:** Interact with your pipelines via:\n * **FlowerPowerProject API:** A unified interface for pipeline execution, supporting both `RunConfig` objects and flexible `**kwargs` overrides.\n * **Command Line Interface (CLI):** For running, managing, and inspecting pipelines, with enhanced `run` command capabilities.\n * **Web UI:** A graphical interface for monitoring and managing pipelines and schedules. ([Hamilton UI](https://hamilton.dagworks.io/en/latest/hamilton-ui/ui/))\n* **Filesystem Abstraction:** Simplified file handling with support for local and remote filesystems (e.g., S3, GCS).\n\n## \ud83d\udce6 Installation\n\nWe recommend using [uv](https://github.com/astral-sh/uv) for installing FlowerPower and managing your project environments. `uv` is an extremely fast Python package installer and resolver.\n\n```bash\n# Create and activate a virtual environment (recommended)\nuv venv\nsource .venv/bin/activate # Or .\\.venv\\Scripts\\activate on Windows\n\n# Install FlowerPower\nuv pip install flowerpower\n\n# Optional: Install additional dependencies for specific features\nuv pip install flowerpower[io] # For I/O plugins (CSV, JSON, Parquet, DeltaTable, DuckDB, PostgreSQL, MySQL, MSSQL, Oracle, SQLite)\nuv pip install flowerpower[ui] # For Hamilton UI\nuv pip install flowerpower[all] # Install all optional dependencies\n```\n\n*(Note: Specify required Python versions if known, e.g., Python 3.8+)*\n\n## \ud83d\ude80 Getting Started\n\nLet's build a simple \"Hello World\" pipeline.\n\n### 1. Initialize Your Project:\n\nYou can quickly set up the standard FlowerPower project structure using the CLI or Python.\n\n**Using the CLI:**\n\nNavigate to your desired parent directory and run:\n```bash\nflowerpower init --name hello-flowerpower-project\n```\n\n\n**Using Python:**\n\nAlternatively, you can initialize programmatically:\n```python\nfrom flowerpower import FlowerPowerProject\n\n# Initialize a new project\nproject = FlowerPowerProject.init(\n name='hello-flowerpower-project',\n)\n```\n\nThis will create a `hello-flowerpower-project` directory with the necessary `conf/` and `pipelines/` subdirectories and default configuration files.\n\n```\nhello-flowerpower-project/\n\u251c\u2500\u2500 conf/\n\u2502 \u251c\u2500\u2500 project.yml\n\u2502 \u2514\u2500\u2500 pipelines/\n\u2514\u2500\u2500 pipelines/\n```\n\nNow, navigate into your new project directory:\n\n```bash\ncd hello-flowerpower-project\n```\n\n**Configure Project (`conf/project.yml`):**\n\nOpen `conf/project.yml` and define your project name:\n\n```yaml\nname: hello-flowerpower\n# adapter: ... # Optional adapter configurations (e.g., Hamilton Tracker, MLflow), see `conf/project.yml` for details\n```\n\n### 2. Create Your Pipeline\n\nYou can create a new pipeline using the CLI or programmatically.\n\n**Using the CLI:**\n\n```bash\nflowerpower pipeline new hello_world\n```\n\n**Using Python:**\n\nYou can create pipelines programmatically using the FlowerPowerProject interface:\n\n```python\nfrom flowerpower import FlowerPowerProject\n\n# Load the project\nproject = FlowerPowerProject.load('.')\n\n# Create a new pipeline\nproject.pipeline_manager.new(name='hello_world')\n```\n\nThis will create a new file `hello_world.py` in the `pipelines/` directory and a corresponding configuration file `hello_world.yml` in `conf/pipelines/`.\n\n**Implement Pipeline (`pipelines/hello_world.py`):**\n\nOpen `pipelines/hello_world.py` and write your pipeline logic using Python and Hamilton. FlowerPower makes configuration easily accessible.\n\n```python\n# FlowerPower pipeline hello_world.py\n# Created on 2025-05-03 22:34:09\n\n####################################################################################################\n# Import necessary libraries\n# NOTE: Remove or comment out imports that are not used in the pipeline\n\nfrom hamilton.function_modifiers import parameterize\n\nfrom pathlib import Path\n\nfrom flowerpower.cfg import Config\n\n####################################################################################################\n# Load pipeline parameters. Do not modify this section.\n\nPARAMS = Config.load(\n Path(__file__).parents[1], pipeline_name=\"hello_world\"\n).pipeline.h_params\n\n\n####################################################################################################\n# Helper functions.\n# This functions have to start with an underscore (_).\n\n\n####################################################################################################\n# Pipeline functions\n\n@parameterize(**PARAMS.greeting_message) # Inject 'message' from params\ndef greeting_message(message: str) -> str:\n \"\"\"Provides the greeting part.\"\"\"\n return f\"{message},\"\n\n@parameterize(**PARAMS.target_name) # Inject 'name' from params\ndef target_name(name: str) -> str:\n \"\"\"Provides the target name.\"\"\"\n return f\"{name}!\"\n\ndef full_greeting(greeting_message: str, target_name: str) -> str:\n \"\"\"Combines the greeting and target.\"\"\"\n print(f\"Generating greeting: {greeting_message} {target_name}\")\n return f\"{greeting_message} {target_name}\"\n\n# You can add more complex Hamilton functions here...\n```\n\n**Configure Pipeline (`conf/pipelines/hello_world.yml`):**\n\nOpen `conf/pipelines/hello_world.yml` and specify parameters, run configurations, and scheduling for your pipeline.\n\n```yaml\n# adapter: ... # Pipeline-specific adapter overrides\n\nparams: # Parameters accessible in your Python code\n greeting_message:\n message: \"Hello\"\n target_name:\n name: \"World\"\n\nrun: # How to execute the pipeline\n final_vars: # Specify the desired output(s) from your Hamilton DAG\n - full_greeting\n # inputs: # Optional: Specify input variables to the pipeline\n # message: \"Hello\"\n # config: ... # Runtime configuration overrides for Hamilton\n # executor: ... # Execution backend (e.g., threadpool, multiprocessing)\n\n```\n### 3. Run Your Pipeline \ud83c\udfc3\u200d\u2640\ufe0f\n\nFlowerPower allows you to execute your pipelines synchronously, with flexible configuration options.\n\n#### Synchronous Execution:\n\nFor quick testing or local runs, you can execute your pipeline synchronously. This is useful for debugging or running pipelines in a local environment.\n\n* **Via CLI:**\n\n The `flowerpower pipeline run` command now supports `RunConfig` objects (via file path or JSON string) and direct `**kwargs` for overriding.\n\n ```bash\n # Basic pipeline execution\n flowerpower pipeline run hello_world\n\n # Run with individual parameters (kwargs)\n flowerpower pipeline run hello_world --inputs '{\"greeting_message\": \"Hi\", \"target_name\": \"FlowerPower\"}' --final-vars '[\"full_greeting\"]' --log-level DEBUG\n\n # Run using a RunConfig from a YAML file\n # Assuming you have a run_config.yaml like:\n # inputs:\n # greeting_message: \"Hola\"\n # target_name: \"Amigo\"\n # log_level: \"INFO\"\n flowerpower pipeline run hello_world --run-config ./run_config.yaml\n\n # Run using a RunConfig provided as a JSON string\n flowerpower pipeline run hello_world --run-config '{\"inputs\": {\"greeting_message\": \"Bonjour\", \"target_name\": \"Monde\"}, \"log_level\": \"INFO\"}'\n\n # Mixing RunConfig with individual parameters (kwargs overrides RunConfig)\n # This will run with log_level=\"DEBUG\" and inputs={\"greeting_message\": \"Howdy\", \"target_name\": \"Partner\"}\n flowerpower pipeline run hello_world --run-config '{\"inputs\": {\"greeting_message\": \"Original\", \"target_name\": \"Value\"}, \"log_level\": \"INFO\"}' --inputs '{\"greeting_message\": \"Howdy\", \"target_name\": \"Partner\"}' --log-level DEBUG\n ```\n\n* **Via Python:**\n\n The `run` methods (`FlowerPowerProject.run`, `PipelineManager.run`) now primarily accept a `RunConfig` object, but also allow individual parameters to be passed via `**kwargs` which override `RunConfig` attributes.\n\n ```python\n from flowerpower import FlowerPowerProject\n from flowerpower.cfg.pipeline.run import RunConfig\n from flowerpower.cfg.pipeline.builder import RunConfigBuilder\n\n # Load the project\n project = FlowerPowerProject.load('.')\n\n # Basic execution\n result = project.run('hello_world')\n print(result)\n\n # Using individual parameters (kwargs)\n result = project.run(\n 'hello_world',\n inputs={\"greeting_message\": \"Hi\", \"target_name\": \"FlowerPower\"},\n final_vars=[\"full_greeting\"],\n log_level=\"DEBUG\"\n )\n print(result)\n\n # Using RunConfig directly\n config = RunConfig(\n inputs={\"greeting_message\": \"Aloha\", \"target_name\": \"World\"},\n final_vars=[\"full_greeting\"],\n log_level=\"INFO\"\n )\n result = project.run('hello_world', run_config=config)\n print(result)\n\n # Using RunConfigBuilder (recommended)\n config = (\n RunConfigBuilder(pipeline_name='hello_world')\n .with_inputs({\"greeting_message\": \"Greetings\", \"target_name\": \"Earth\"})\n .with_final_vars([\"full_greeting\"])\n .with_log_level(\"DEBUG\")\n .with_retries(max_attempts=3, delay=1.0)\n .build()\n )\n result = project.run('hello_world', run_config=config)\n print(result)\n\n # Mixing RunConfig with individual parameters (kwargs overrides RunConfig)\n base_config = RunConfigBuilder().with_log_level(\"INFO\").build()\n result = project.run(\n 'hello_world',\n run_config=base_config,\n inputs={\"greeting_message\": \"Howdy\", \"target_name\": \"Partner\"}, # Overrides inputs in base_config\n log_level=\"DEBUG\" # Overrides log_level in base_config\n )\n print(result)\n ```\n\n\n## \u2699\ufe0f Configuration Overview\n\nFlowerPower uses a layered configuration system:\n\n* **`conf/project.yml`:** Defines global settings for your project, including integrated `adapter`s (like Hamilton Tracker, MLflow, etc.).\n* **`conf/pipelines/*.yml`:** Each file defines a specific pipeline. It contains:\n * `params`: Input parameters for your Hamilton functions.\n * `run`: Execution details like target outputs (`final_vars`), Hamilton runtime `config`, and `executor` settings.\n * `adapter`: Pipeline-specific overrides for adapter settings.\n\n## \ud83d\udee0\ufe0f Basic Usage\n\nYou can interact with FlowerPower pipelines through multiple interfaces:\n\n**Python API (Recommended):**\n```python\nfrom flowerpower import FlowerPowerProject\nfrom flowerpower.cfg.pipeline.run import RunConfig\nfrom flowerpower.cfg.pipeline.builder import RunConfigBuilder\n\n# Load the project\nproject = FlowerPowerProject.load('.')\n\n# Run a pipeline using RunConfig\nconfig = RunConfig(inputs={\"greeting_message\": \"Hello\", \"target_name\": \"API\"})\nresult = project.run('hello_world', run_config=config)\nprint(result)\n\n# Run a pipeline using kwargs\nresult = project.run('hello_world', inputs={\"greeting_message\": \"Hi\", \"target_name\": \"Kwargs\"})\nprint(result)\n```\n\n**CLI:**\n```bash\n# Run a pipeline using RunConfig from a file\n# flowerpower pipeline run hello_world --run-config ./path/to/run_config.yaml\n\n# Run a pipeline using kwargs\nflowerpower pipeline run hello_world --inputs '{\"greeting_message\": \"CLI\", \"target_name\": \"Kwargs\"}'\n\n# List all available commands\nflowerpower --help\n```\n\n## \ud83d\udd27 Direct Module Usage\n\nWhile the unified `FlowerPowerProject` interface is recommended for most use cases, you can also use the pipeline module directly for more granular control or when you only need specific functionality.\n\n### Pipeline-Only Usage\n\nIf you only need pipeline execution, you can use the `PipelineManager` directly:\n\n```python\nfrom flowerpower.pipeline import PipelineManager\nfrom flowerpower.cfg.pipeline.run import RunConfig\nfrom flowerpower.cfg.pipeline.builder import RunConfigBuilder\n\n# Initialize pipeline manager\npm = PipelineManager(base_dir='.')\n\n# Create a new pipeline\npm.new(name='my_pipeline')\n\n# Run a pipeline synchronously using RunConfig\nconfig = RunConfig(inputs={'param': 'value'}, final_vars=['output_var'])\nresult = pm.run(name='my_pipeline', run_config=config)\nprint(result)\n\n# Run a pipeline synchronously using kwargs\nresult = pm.run(name='my_pipeline', inputs={'param': 'new_value'}, final_vars=['output_var'])\nprint(result)\n\n# List available pipelines\npipelines = pm.list()\nprint(f\"Available pipelines: {pipelines}\")\n\n# Get pipeline information\ninfo = pm.get('my_pipeline')\nprint(f\"Pipeline config: {info}\")\n\n# Delete a pipeline\npm.delete('old_pipeline')\n```\n\n**When to use Pipeline-only approach:**\n- Simple synchronous workflows\n- Testing and development\n- Lightweight applications with minimal dependencies\n\n**Benefits of FlowerPowerProject vs Direct Usage:**\n\n| Approach | Benefits | Use Cases |\n|----------|----------|-----------|\n| **FlowerPowerProject** | - Unified interface<br>- Automatic dependency injection<br>- Simplified configuration<br>- Best practices built-in | - Most applications<br>- Rapid development<br>- Full feature integration |\n| **Pipeline-only** | - Lightweight<br>- Simple synchronous execution | - Testing<br>- Simple workflows |\n\n## \ud83d\udda5\ufe0f UI\n\nThe FlowerPower web UI (Hamilton UI) provides a graphical interface for monitoring and managing your pipelines. It allows you to visualize pipeline runs, schedules, and potentially manage configurations.\n\n```bash\n# Start the web UI\nflowerpower ui\n```\n\n## \ud83d\udcd6 Documentation\n\nYou can find the full documentation for FlowerPower, including installation instructions, usage examples, and API references, at [https://legout.github.io/flowerpower/](https://legout.github.io/flowerpower/).\n\n\n## \ud83d\udcdc License\n\nThis project is licensed under the MIT License - see the `LICENSE` file for details. (Placeholder - update with actual license)\n",
"bugtrack_url": null,
"license": null,
"summary": "A simple workflow framework for building and managing data processing pipelines",
"version": "0.21.0",
"project_urls": {
"Bug Tracker": "https://github.com/legout/flowerpower/issues",
"Homepage": "https://github.com/legout/flowerpower"
},
"split_keywords": [
"hamilton",
" workflow",
" pipeline",
" scheduler",
" dask",
" ray"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "58ff73b1400609f50b0a869501330561f708c65a005170dfc9d0d5cdb167c9d1",
"md5": "e612f1a16b1dcbeac1fa839126bcfa57",
"sha256": "acc17f92e74fc0122eb2c1c4fbeb4a5a095505509757c60a2bb67551586add9e"
},
"downloads": -1,
"filename": "flowerpower-0.21.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "e612f1a16b1dcbeac1fa839126bcfa57",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.11",
"size": 76226,
"upload_time": "2025-09-01T21:36:31",
"upload_time_iso_8601": "2025-09-01T21:36:31.557524Z",
"url": "https://files.pythonhosted.org/packages/58/ff/73b1400609f50b0a869501330561f708c65a005170dfc9d0d5cdb167c9d1/flowerpower-0.21.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "bae2eed9f122f3e69e08a37484ab2c274ddb012704d62e792ab4a5271896ffae",
"md5": "4827b8c0ac56c9bd251837ced1a7179a",
"sha256": "1ccbebd36e6fc6a154dcdf949f4ed07a44883bf146d57723a97400ba1a96d799"
},
"downloads": -1,
"filename": "flowerpower-0.21.0.tar.gz",
"has_sig": false,
"md5_digest": "4827b8c0ac56c9bd251837ced1a7179a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.11",
"size": 66073,
"upload_time": "2025-09-01T21:36:33",
"upload_time_iso_8601": "2025-09-01T21:36:33.031807Z",
"url": "https://files.pythonhosted.org/packages/ba/e2/eed9f122f3e69e08a37484ab2c274ddb012704d62e792ab4a5271896ffae/flowerpower-0.21.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-09-01 21:36:33",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "legout",
"github_project": "flowerpower",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "flowerpower"
}
