hydraflow


Namehydraflow JSON
Version 0.17.0 PyPI version JSON
download
home_pageNone
SummaryHydraFlow seamlessly integrates Hydra and MLflow to streamline ML experiment management, combining Hydra's configuration management with MLflow's tracking capabilities.
upload_time2025-04-25 23:13:16
maintainerNone
docs_urlNone
authorNone
requires_python>=3.13
licenseMIT License Copyright (c) 2024 Daizu Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
keywords ai data-science deep-learning experiment-tracking hydra machine-learning mlflow mlops research
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # HydraFlow

[![PyPI Version][pypi-v-image]][pypi-v-link]
[![Build Status][GHAction-image]][GHAction-link]
[![Coverage Status][codecov-image]][codecov-link]
[![Documentation Status][docs-image]][docs-link]
[![Python Version][python-v-image]][python-v-link]

<!-- Badges -->

[pypi-v-image]: https://img.shields.io/pypi/v/hydraflow.svg
[pypi-v-link]: https://pypi.org/project/hydraflow/
[GHAction-image]: https://github.com/daizutabi/hydraflow/actions/workflows/ci.yaml/badge.svg?branch=main&event=push
[GHAction-link]: https://github.com/daizutabi/hydraflow/actions?query=event%3Apush+branch%3Amain
[codecov-image]: https://codecov.io/github/daizutabi/hydraflow/coverage.svg?branch=main
[codecov-link]: https://codecov.io/github/daizutabi/hydraflow?branch=main
[docs-image]: https://img.shields.io/badge/docs-latest-blue.svg
[docs-link]: https://daizutabi.github.io/hydraflow/
[python-v-image]: https://img.shields.io/pypi/pyversions/hydraflow.svg
[python-v-link]: https://pypi.org/project/hydraflow

## Overview

HydraFlow seamlessly integrates [Hydra](https://hydra.cc/) and [MLflow](https://mlflow.org/) to streamline machine learning experiment workflows. By combining Hydra's powerful configuration management with MLflow's robust experiment tracking, HydraFlow provides a comprehensive solution for defining, executing, and analyzing machine learning experiments.

## Design Principles

HydraFlow is built on the following design principles:

1. **Type Safety** - Utilizing Python dataclasses for configuration type checking and IDE support
2. **Reproducibility** - Automatically tracking all experiment configurations for fully reproducible experiments
3. **Analysis Capabilities** - Providing powerful APIs for easily analyzing experiment results
4. **Workflow Integration** - Creating a cohesive workflow by integrating Hydra's configuration management with MLflow's experiment tracking

## Key Features

- **Type-safe Configuration Management** - Define experiment parameters using Python dataclasses with full IDE support and validation
- **Seamless Hydra-MLflow Integration** - Automatically register configurations with Hydra and track experiments with MLflow
- **Advanced Parameter Sweeps** - Define complex parameter spaces using extended sweep syntax for numerical ranges, combinations, and SI prefixes
- **Workflow Automation** - Create reusable experiment workflows with YAML-based job definitions
- **Powerful Analysis Tools** - Filter, group, and analyze experiment results with type-aware APIs
- **Custom Implementation Support** - Extend experiment analysis with domain-specific functionality

## Installation

```bash
pip install hydraflow
```

**Requirements:** Python 3.13+

## Quick Example

```python
from dataclasses import dataclass
from mlflow.entities import Run
import hydraflow

@dataclass
class Config:
    width: int = 1024
    height: int = 768

@hydraflow.main(Config)
def app(run: Run, cfg: Config) -> None:
    # Your experiment code here
    print(f"Running with width={cfg.width}, height={cfg.height}")

    # Log metrics
    hydraflow.log_metric("area", cfg.width * cfg.height)

if __name__ == "__main__":
    app()
```

Execute a parameter sweep with:

```bash
python app.py -m width=800,1200 height=600,900
```

## Core Components

HydraFlow consists of the following key components:

### Configuration Management

Define type-safe configurations using Python dataclasses:

```python
@dataclass
class Config:
    learning_rate: float = 0.001
    batch_size: int = 32
    epochs: int = 10
```

### Main Decorator

The `@hydraflow.main` decorator integrates Hydra and MLflow:

```python
@hydraflow.main(Config)
def train(run: Run, cfg: Config) -> None:
    # Your experiment code
```

### Workflow Automation

Define reusable experiment workflows in YAML:

```yaml
jobs:
  train_models:
    run: python train.py
    sets:
      - each: model=small,medium,large
        all: learning_rate=0.001,0.01,0.1
```

### Analysis Tools

Analyze experiment results with powerful APIs:

```python
from hydraflow import Run, iter_run_dirs

# Load runs
runs = Run.load(iter_run_dirs("mlruns"))

# Filter and analyze
best_runs = runs.filter(model_type="transformer").to_frame("learning_rate", "accuracy")
```

## Documentation

For detailed documentation, visit our [documentation site](https://daizutabi.github.io/hydraflow/):

- [Getting Started](https://daizutabi.github.io/hydraflow/getting-started/) - Installation and core concepts
- [Practical Tutorials](https://daizutabi.github.io/hydraflow/practical-tutorials/) - Learn through hands-on examples
- [User Guide](https://daizutabi.github.io/hydraflow/part1-applications/) - Detailed documentation of HydraFlow's capabilities
- [API Reference](https://daizutabi.github.io/hydraflow/api/hydraflow/) - Complete API documentation

## License

This project is licensed under the MIT License.

            

Raw data

            {
    "_id": null,
    "home_page": null,
    "name": "hydraflow",
    "maintainer": null,
    "docs_url": null,
    "requires_python": ">=3.13",
    "maintainer_email": null,
    "keywords": "ai, data-science, deep-learning, experiment-tracking, hydra, machine-learning, mlflow, mlops, research",
    "author": null,
    "author_email": "daizutabi <daizutabi@gmail.com>",
    "download_url": "https://files.pythonhosted.org/packages/94/3b/23a5d63a72539d06ea415e02c190c20c666bd927f5c494606aa2e09c7dda/hydraflow-0.17.0.tar.gz",
    "platform": null,
    "description": "# HydraFlow\n\n[![PyPI Version][pypi-v-image]][pypi-v-link]\n[![Build Status][GHAction-image]][GHAction-link]\n[![Coverage Status][codecov-image]][codecov-link]\n[![Documentation Status][docs-image]][docs-link]\n[![Python Version][python-v-image]][python-v-link]\n\n<!-- Badges -->\n\n[pypi-v-image]: https://img.shields.io/pypi/v/hydraflow.svg\n[pypi-v-link]: https://pypi.org/project/hydraflow/\n[GHAction-image]: https://github.com/daizutabi/hydraflow/actions/workflows/ci.yaml/badge.svg?branch=main&event=push\n[GHAction-link]: https://github.com/daizutabi/hydraflow/actions?query=event%3Apush+branch%3Amain\n[codecov-image]: https://codecov.io/github/daizutabi/hydraflow/coverage.svg?branch=main\n[codecov-link]: https://codecov.io/github/daizutabi/hydraflow?branch=main\n[docs-image]: https://img.shields.io/badge/docs-latest-blue.svg\n[docs-link]: https://daizutabi.github.io/hydraflow/\n[python-v-image]: https://img.shields.io/pypi/pyversions/hydraflow.svg\n[python-v-link]: https://pypi.org/project/hydraflow\n\n## Overview\n\nHydraFlow seamlessly integrates [Hydra](https://hydra.cc/) and [MLflow](https://mlflow.org/) to streamline machine learning experiment workflows. By combining Hydra's powerful configuration management with MLflow's robust experiment tracking, HydraFlow provides a comprehensive solution for defining, executing, and analyzing machine learning experiments.\n\n## Design Principles\n\nHydraFlow is built on the following design principles:\n\n1. **Type Safety** - Utilizing Python dataclasses for configuration type checking and IDE support\n2. **Reproducibility** - Automatically tracking all experiment configurations for fully reproducible experiments\n3. **Analysis Capabilities** - Providing powerful APIs for easily analyzing experiment results\n4. **Workflow Integration** - Creating a cohesive workflow by integrating Hydra's configuration management with MLflow's experiment tracking\n\n## Key Features\n\n- **Type-safe Configuration Management** - Define experiment parameters using Python dataclasses with full IDE support and validation\n- **Seamless Hydra-MLflow Integration** - Automatically register configurations with Hydra and track experiments with MLflow\n- **Advanced Parameter Sweeps** - Define complex parameter spaces using extended sweep syntax for numerical ranges, combinations, and SI prefixes\n- **Workflow Automation** - Create reusable experiment workflows with YAML-based job definitions\n- **Powerful Analysis Tools** - Filter, group, and analyze experiment results with type-aware APIs\n- **Custom Implementation Support** - Extend experiment analysis with domain-specific functionality\n\n## Installation\n\n```bash\npip install hydraflow\n```\n\n**Requirements:** Python 3.13+\n\n## Quick Example\n\n```python\nfrom dataclasses import dataclass\nfrom mlflow.entities import Run\nimport hydraflow\n\n@dataclass\nclass Config:\n    width: int = 1024\n    height: int = 768\n\n@hydraflow.main(Config)\ndef app(run: Run, cfg: Config) -> None:\n    # Your experiment code here\n    print(f\"Running with width={cfg.width}, height={cfg.height}\")\n\n    # Log metrics\n    hydraflow.log_metric(\"area\", cfg.width * cfg.height)\n\nif __name__ == \"__main__\":\n    app()\n```\n\nExecute a parameter sweep with:\n\n```bash\npython app.py -m width=800,1200 height=600,900\n```\n\n## Core Components\n\nHydraFlow consists of the following key components:\n\n### Configuration Management\n\nDefine type-safe configurations using Python dataclasses:\n\n```python\n@dataclass\nclass Config:\n    learning_rate: float = 0.001\n    batch_size: int = 32\n    epochs: int = 10\n```\n\n### Main Decorator\n\nThe `@hydraflow.main` decorator integrates Hydra and MLflow:\n\n```python\n@hydraflow.main(Config)\ndef train(run: Run, cfg: Config) -> None:\n    # Your experiment code\n```\n\n### Workflow Automation\n\nDefine reusable experiment workflows in YAML:\n\n```yaml\njobs:\n  train_models:\n    run: python train.py\n    sets:\n      - each: model=small,medium,large\n        all: learning_rate=0.001,0.01,0.1\n```\n\n### Analysis Tools\n\nAnalyze experiment results with powerful APIs:\n\n```python\nfrom hydraflow import Run, iter_run_dirs\n\n# Load runs\nruns = Run.load(iter_run_dirs(\"mlruns\"))\n\n# Filter and analyze\nbest_runs = runs.filter(model_type=\"transformer\").to_frame(\"learning_rate\", \"accuracy\")\n```\n\n## Documentation\n\nFor detailed documentation, visit our [documentation site](https://daizutabi.github.io/hydraflow/):\n\n- [Getting Started](https://daizutabi.github.io/hydraflow/getting-started/) - Installation and core concepts\n- [Practical Tutorials](https://daizutabi.github.io/hydraflow/practical-tutorials/) - Learn through hands-on examples\n- [User Guide](https://daizutabi.github.io/hydraflow/part1-applications/) - Detailed documentation of HydraFlow's capabilities\n- [API Reference](https://daizutabi.github.io/hydraflow/api/hydraflow/) - Complete API documentation\n\n## License\n\nThis project is licensed under the MIT License.\n",
    "bugtrack_url": null,
    "license": "MIT License  Copyright (c) 2024 Daizu  Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:  The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.  THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.",
    "summary": "HydraFlow seamlessly integrates Hydra and MLflow to streamline ML experiment management, combining Hydra's configuration management with MLflow's tracking capabilities.",
    "version": "0.17.0",
    "project_urls": {
        "Documentation": "https://daizutabi.github.io/hydraflow/",
        "Issues": "https://github.com/daizutabi/hydraflow/issues",
        "Source": "https://github.com/daizutabi/hydraflow"
    },
    "split_keywords": [
        "ai",
        " data-science",
        " deep-learning",
        " experiment-tracking",
        " hydra",
        " machine-learning",
        " mlflow",
        " mlops",
        " research"
    ],
    "urls": [
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "e95e978611ab0403566970b2f962d27fd3e668a92ddb2f99f605f8f9046c3183",
                "md5": "2a8a005f28aa979adc1af31d56bd5da1",
                "sha256": "97e1316142f4153740103efdba7332c7b1c05d464fbb3e7f4c0285ac0e3565a6"
            },
            "downloads": -1,
            "filename": "hydraflow-0.17.0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "2a8a005f28aa979adc1af31d56bd5da1",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.13",
            "size": 32183,
            "upload_time": "2025-04-25T23:13:15",
            "upload_time_iso_8601": "2025-04-25T23:13:15.325728Z",
            "url": "https://files.pythonhosted.org/packages/e9/5e/978611ab0403566970b2f962d27fd3e668a92ddb2f99f605f8f9046c3183/hydraflow-0.17.0-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "943b23a5d63a72539d06ea415e02c190c20c666bd927f5c494606aa2e09c7dda",
                "md5": "76960c5be3444945dbfa75f033bfc51c",
                "sha256": "089582efe4dde72f0f89b62ba9cfaee930251a0af653a1dec23a62557a202b70"
            },
            "downloads": -1,
            "filename": "hydraflow-0.17.0.tar.gz",
            "has_sig": false,
            "md5_digest": "76960c5be3444945dbfa75f033bfc51c",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.13",
            "size": 73049,
            "upload_time": "2025-04-25T23:13:16",
            "upload_time_iso_8601": "2025-04-25T23:13:16.380535Z",
            "url": "https://files.pythonhosted.org/packages/94/3b/23a5d63a72539d06ea415e02c190c20c666bd927f5c494606aa2e09c7dda/hydraflow-0.17.0.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2025-04-25 23:13:16",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "daizutabi",
    "github_project": "hydraflow",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "lcname": "hydraflow"
}
        
Elapsed time: 0.42974s