# Curifactory
[](https://github.com/psf/black)
[](https://badge.fury.io/py/curifactory)
[](https://anaconda.org/conda-forge/curifactory)
[](https://joss.theoj.org/papers/e6ace365c4f632391a289ddea5bbfd1c)
[](https://github.com/ORNL/curifactory/actions/workflows/tests.yml)
[](https://github.com/ORNL/curifactory)
[](https://github.com/ORNL/curifactory/blob/main/LICENSE)
Curifactory is a library and CLI tool designed to help organize and manage
research experiments in python.
Experiment management must fulfill several tasks, including experiment orchestration,
parameterization, caching, reproducibility, reporting, and parallelization.
Existing projects such as MLFlow, MetaFlow, Luigi, and Pachyderm
support these tasks in several different ways and to various degrees.
Curifactory provides a different opinion, with a heavier focus on supporting general
research experiment workflows for individuals or small teams working primarily
in python.
You can read more about these design principles in our paper in the [SciPy 2022
proceedings](https://conference.scipy.org/proceedings/scipy2022/nathan_martindale.html).
## Features
* Adds a CLI layer on top of your codebase, a single entrypoint for running experiments
* Automatic caching of intermediate data and lazy loading of stored objects
* Jupyter notebook output for further exploration of an experiment run
* Docker container output with copy of codebase, conda environment, full experiment run cache, and jupyter run notebook
* HTML report output from each run with graphviz-rendered diagram of experiment
* Easily report plots and values to HTML report
* Configuration files are python scripts, allowing programmatic definition, parameter composition, and parameter inheritance
* Output logs from every run
* Run experiments directly from CLI or other python code, notebooks, etc.
## Installation
```python
pip install curifactory
```
Graphviz is required for certain features and can be installed through conda
via:
```python
conda install python-graphviz
```
### Tab completion
For tab-completion in bash/zsh, install the `argcomplete` package (if using curifactory inside
a conda environment, you'll need to install this in your system python.)
```python
pip install argcomplete
```
To enable, you can either use argcomplete's global hook `activate-global-python-argcomplete`, which
will enable tab complete on all argcomplete-enabled python packages (e.g. pytest), or you can add
`eval "$(register-python-argcomplete experiment)"` to your shell's rc file. Curifactory can add
this line for you automatically with:
```bash
curifactory completion [--bash|--zsh] # use the shell flag appropriate
```
Once enabled, the `experiment` command will provide tab complete for experiment names, parameter names, and flags.
## Requirements
**OS**: We primarily develop and test Curifactory on Linux, but it runs on Windows and MacOS as well.
**Python**: 3.9-3.11
**Optional**:
* Conda/Mamba
* Graphviz
* Docker
* Jupyter notebook/lab
## Documentation
The documentation for the latest version of Curifactory can be found at:
[https://ornl.github.io/curifactory/stable/index.html](https://ornl.github.io/curifactory/stable/index.html).
## Examples
Several small example can be found in the `examples` folder.
`examples/notebooks` includes walkthroughs demonstrating usage of curifactory
solely in Jupyter.
## Citation
Please use the following BibTeX if citing this project:
```bibtex
@article{Martindale2023,
doi = {10.21105/joss.05793},
url = {https://doi.org/10.21105/joss.05793},
year = {2023},
publisher = {The Open Journal},
volume = {8},
number = {90},
pages = {5793},
author = {Nathan Martindale and Scott L. Stewart and Jason Hite and Mark B. Adams},
title = {Curifactory: A research experiment manager},
journal = {Journal of Open Source Software}
}
```
## Similar Projects
Curifactory is one tool and one opinion among many, other projects that have similar goals and/or approaches:
* [Kedro](https://github.com/kedro-org/kedro)
* [Tango](https://github.com/allenai/tango)
* [Sacred](https://github.com/IDSIA/sacred)
* [MLFlow](https://github.com/mlflow/mlflow)
Raw data
{
"_id": null,
"home_page": "https://github.com/ORNL/curifactory",
"name": "curifactory",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "research, experiment, workflow",
"author": "Nathan Martindale, Jason Hite, Scott L. Stewart, Mark Adams",
"author_email": "curifactory-help@ornl.gov",
"download_url": "https://files.pythonhosted.org/packages/f8/87/a4aaadd1eeab1ebe533ec2037d05d20e8b5ddeae3c97c1b630c4a1ec1902/curifactory-0.18.0.tar.gz",
"platform": null,
"description": "# Curifactory\n\n[](https://github.com/psf/black)\n[](https://badge.fury.io/py/curifactory)\n[](https://anaconda.org/conda-forge/curifactory)\n[](https://joss.theoj.org/papers/e6ace365c4f632391a289ddea5bbfd1c)\n[](https://github.com/ORNL/curifactory/actions/workflows/tests.yml)\n[](https://github.com/ORNL/curifactory)\n[](https://github.com/ORNL/curifactory/blob/main/LICENSE)\n\nCurifactory is a library and CLI tool designed to help organize and manage\nresearch experiments in python.\n\n\n\nExperiment management must fulfill several tasks, including experiment orchestration,\nparameterization, caching, reproducibility, reporting, and parallelization.\nExisting projects such as MLFlow, MetaFlow, Luigi, and Pachyderm\nsupport these tasks in several different ways and to various degrees.\nCurifactory provides a different opinion, with a heavier focus on supporting general\nresearch experiment workflows for individuals or small teams working primarily\nin python.\n\nYou can read more about these design principles in our paper in the [SciPy 2022\nproceedings](https://conference.scipy.org/proceedings/scipy2022/nathan_martindale.html).\n\n## Features\n\n* Adds a CLI layer on top of your codebase, a single entrypoint for running experiments\n* Automatic caching of intermediate data and lazy loading of stored objects\n* Jupyter notebook output for further exploration of an experiment run\n* Docker container output with copy of codebase, conda environment, full experiment run cache, and jupyter run notebook\n* HTML report output from each run with graphviz-rendered diagram of experiment\n* Easily report plots and values to HTML report\n* Configuration files are python scripts, allowing programmatic definition, parameter composition, and parameter inheritance\n* Output logs from every run\n* Run experiments directly from CLI or other python code, notebooks, etc.\n\n\n## Installation\n\n```python\npip install curifactory\n```\n\nGraphviz is required for certain features and can be installed through conda\nvia:\n\n```python\nconda install python-graphviz\n```\n\n### Tab completion\n\nFor tab-completion in bash/zsh, install the `argcomplete` package (if using curifactory inside\na conda environment, you'll need to install this in your system python.)\n\n```python\npip install argcomplete\n```\n\nTo enable, you can either use argcomplete's global hook `activate-global-python-argcomplete`, which\nwill enable tab complete on all argcomplete-enabled python packages (e.g. pytest), or you can add\n`eval \"$(register-python-argcomplete experiment)\"` to your shell's rc file. Curifactory can add\nthis line for you automatically with:\n\n```bash\ncurifactory completion [--bash|--zsh] # use the shell flag appropriate\n```\n\nOnce enabled, the `experiment` command will provide tab complete for experiment names, parameter names, and flags.\n\n## Requirements\n\n**OS**: We primarily develop and test Curifactory on Linux, but it runs on Windows and MacOS as well.\n\n**Python**: 3.9-3.11\n\n**Optional**:\n\n* Conda/Mamba\n* Graphviz\n* Docker\n* Jupyter notebook/lab\n\n\n## Documentation\n\nThe documentation for the latest version of Curifactory can be found at:\n[https://ornl.github.io/curifactory/stable/index.html](https://ornl.github.io/curifactory/stable/index.html).\n\n\n## Examples\n\nSeveral small example can be found in the `examples` folder.\n`examples/notebooks` includes walkthroughs demonstrating usage of curifactory\nsolely in Jupyter.\n\n\n## Citation\n\nPlease use the following BibTeX if citing this project:\n```bibtex\n@article{Martindale2023,\n doi = {10.21105/joss.05793},\n url = {https://doi.org/10.21105/joss.05793},\n year = {2023},\n publisher = {The Open Journal},\n volume = {8},\n number = {90},\n pages = {5793},\n author = {Nathan Martindale and Scott L. Stewart and Jason Hite and Mark B. Adams},\n title = {Curifactory: A research experiment manager},\n journal = {Journal of Open Source Software}\n}\n```\n\n\n## Similar Projects\n\nCurifactory is one tool and one opinion among many, other projects that have similar goals and/or approaches:\n\n* [Kedro](https://github.com/kedro-org/kedro)\n* [Tango](https://github.com/allenai/tango)\n* [Sacred](https://github.com/IDSIA/sacred)\n* [MLFlow](https://github.com/mlflow/mlflow)\n",
"bugtrack_url": null,
"license": null,
"summary": "An experiment workflow and organization tool",
"version": "0.18.0",
"project_urls": {
"Documentation": "https://ornl.github.io/curifactory/stable/index.html",
"Homepage": "https://github.com/ORNL/curifactory"
},
"split_keywords": [
"research",
" experiment",
" workflow"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "1713c8b2ca10941086ab7f68bcb4a943712e9f51578676412d879690f78f1325",
"md5": "6c952d4780b23120b3bac141490414d1",
"sha256": "b31c4af4cc53a511e3fcbed5f52e0069247602b2063e6c1c0bfefd62f99d8fc2"
},
"downloads": -1,
"filename": "curifactory-0.18.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "6c952d4780b23120b3bac141490414d1",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 99300,
"upload_time": "2024-10-09T21:44:01",
"upload_time_iso_8601": "2024-10-09T21:44:01.815832Z",
"url": "https://files.pythonhosted.org/packages/17/13/c8b2ca10941086ab7f68bcb4a943712e9f51578676412d879690f78f1325/curifactory-0.18.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f887a4aaadd1eeab1ebe533ec2037d05d20e8b5ddeae3c97c1b630c4a1ec1902",
"md5": "4d9653b0b93d4af8a57f8a48986e9bcf",
"sha256": "4c0580a3f1741296d2794473a0b2da84c13f603f5fb94bef36ed62f0b8d79f48"
},
"downloads": -1,
"filename": "curifactory-0.18.0.tar.gz",
"has_sig": false,
"md5_digest": "4d9653b0b93d4af8a57f8a48986e9bcf",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 124374,
"upload_time": "2024-10-09T21:44:03",
"upload_time_iso_8601": "2024-10-09T21:44:03.701744Z",
"url": "https://files.pythonhosted.org/packages/f8/87/a4aaadd1eeab1ebe533ec2037d05d20e8b5ddeae3c97c1b630c4a1ec1902/curifactory-0.18.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-09 21:44:03",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ORNL",
"github_project": "curifactory",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "numpy",
"specs": []
},
{
"name": "pandas",
"specs": []
},
{
"name": "sphinx",
"specs": [
[
"~=",
"3.5.4"
]
]
},
{
"name": "jinja2",
"specs": []
},
{
"name": "autodocsumm",
"specs": []
},
{
"name": "sphinx-rtd-theme",
"specs": []
},
{
"name": "ipynb-py-convert",
"specs": []
},
{
"name": "graphviz",
"specs": []
},
{
"name": "matplotlib",
"specs": []
},
{
"name": "pytest",
"specs": []
},
{
"name": "pytest-mock",
"specs": []
},
{
"name": "pytest-cov",
"specs": []
},
{
"name": "psutil",
"specs": []
},
{
"name": "twine",
"specs": []
},
{
"name": "build",
"specs": []
},
{
"name": "rich",
"specs": []
},
{
"name": "pre-commit",
"specs": []
},
{
"name": "argcomplete",
"specs": []
},
{
"name": "pyarrow",
"specs": [
[
">=",
"7"
]
]
},
{
"name": "lxml",
"specs": []
},
{
"name": "openpyxl",
"specs": []
},
{
"name": "tables",
"specs": []
}
],
"lcname": "curifactory"
}
JSON
