| Name | jobflow JSON |
| Version |
0.1.19
JSON |
| download |
| home_page | None |
| Summary | jobflow is a library for writing computational workflows |
| upload_time | 2024-11-22 14:27:11 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.9 |
| license | modified BSD |
| keywords |
high-throughput
workflow
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<div align="center">
# 
[](https://github.com/materialsproject/jobflow/actions?query=workflow%3Atesting)
[](https://codecov.io/gh/materialsproject/jobflow/)
[](https://pypi.org/project/jobflow/)
[](https://doi.org/10.21105/joss.05995)
</div>
[Documentation](https://materialsproject.github.io/jobflow/) | [PyPI](https://pypi.org/project/jobflow/) | [GitHub](https://github.com/materialsproject/jobflow) | [Paper](https://doi.org/10.21105/joss.05995)
Jobflow is a free, open-source library for writing and executing workflows. Complex
workflows can be defined using simple python functions and executed locally or on
arbitrary computing resources using the [jobflow-remote][jfr] or [FireWorks][fireworks]
workflow managers.
Some features that distinguish jobflow are dynamic workflows, easy compositing and
connecting of workflows, and the ability to store workflow outputs across multiple
databases.
## Is jobflow for me
jobflow is intended to be a friendly workflow software that is easy to get started with,
but flexible enough to handle complicated use cases.
Some of its features include:
- A clean and flexible Python API.
- A powerful approach to compositing and connecting workflows — information passing
between jobs is a key goal of jobflow. Workflows can be nested allowing for a natural
way to build complex workflows.
- Integration with multiple databases (MongoDB, S3, GridFS, and more) through the
[Maggma][maggma] package.
- Support for the [jobflow-remote][jfr] and [FireWorks][fireworks] workflow management systems, allowing workflow
execution on multicore machines or through a queue, on a single machine or multiple
machines.
- Support for dynamic workflows — workflows that modify themselves or create new ones
based on what happens during execution.
## Workflow model
Workflows in jobflows are made up of two main components:
- A `Job` is an atomic computing job. Essentially any python function can be `Job`,
provided its input and return values can be serialized to json. Anything returned by the job is
considered an "output" and is stored in the jobflow database.
- A `Flow` is a collection of `Job` objects or other `Flow` objects. The connectivity
between jobs is determined automatically from the job inputs. The execution order
of jobs is automatically determined based on their connectivity.
Python functions can be easily converted in to `Job` objects using the `@job` decorator.
In the example below, we define a job to add two numbers.
```python
from jobflow import job, Flow
@job
def add(a, b):
return a + b
add_first = add(1, 5)
add_second = add(add_first.output, 5)
flow = Flow([add_first, add_second])
flow.draw_graph().show()
```
The output of the job is accessed using the `output` attribute. As the job has not
yet been run, `output` contains be a reference to a future output. Outputs can be used
as inputs to other jobs and will be automatically "resolved" before the job is
executed.
Finally, we created a flow using the two `Job` objects. The connectivity between
the jobs is determined automatically and can be visualised using the flow graph.
<p align="center">
<img alt="simple flow graph" src="https://raw.githubusercontent.com/materialsproject/jobflow/main/docs/_static/img/simple_flow.png" width="30%" height="30%">
</p>
## Installation
`jobflow` is a Python 3.9+ library and can be installed using `pip`.
```bash
pip install jobflow
```
## Quickstart and tutorials
To get a first glimpse of jobflow, we suggest that you follow our quickstart tutorial.
Later tutorials delve into the advanced features of jobflow.
- [Five-minute quickstart tutorial][quickstart]
- [Introduction to jobflow][introduction]
- [Defining Jobs using jobflow][defining-jobs]
## Need help?
Ask questions about jobflow on the [jobflow support forum][help-forum].
If you've found an issue with jobflow, please submit a bug report on [GitHub Issues][issues].
## What’s new?
Track changes to jobflow through the [changelog][changelog].
## Contributing
We greatly appreciate any contributions in the form of a pull request.
Additional information on contributing to jobflow can be found [here][contributing].
We maintain a list of all contributors [here][contributors].
## License
jobflow is released under a modified BSD license; the full text can be found [here][license].
## Citation
If you use Jobflow in your work, please cite it as follows:
- "Jobflow: Computational Workflows Made Simple", A.S. Rosen, M. Gallant, J. George, J. Riebesell, H. Sahasrabuddhe, J.X. Shen, M. Wen, M.L. Evans, G. Petretto, D. Waroquiers, G.‑M. Rignanese, K.A. Persson, A. Jain, A.M. Ganose, _Journal of Open Source Software_, 9(93), 5995 (2024) DOI: [10.21105/joss.05995](https://doi.org/10.21105/joss.05995)
## Acknowledgements
Jobflow was designed by Alex Ganose, Anubhav Jain, Gian-Marco Rignanese, David Waroquiers, and Guido Petretto. Alex Ganose implemented the first version of the package. Later versions have benefited from the contributions of several research groups. A full list of contributors is available [here](https://materialsproject.github.io/jobflow/contributors.html).
[maggma]: https://materialsproject.github.io/maggma/
[fireworks]: https://materialsproject.github.io/fireworks/
[jfr]: https://matgenix.github.io/jobflow-remote
[help-forum]: https://matsci.org/c/fireworks
[issues]: https://github.com/materialsproject/jobflow/issues
[changelog]: https://materialsproject.github.io/jobflow/changelog.html
[contributing]: https://materialsproject.github.io/jobflow/contributing.html
[contributors]: https://materialsproject.github.io/jobflow/contributors.html
[license]: https://raw.githubusercontent.com/materialsproject/jobflow/main/LICENSE
[quickstart]: https://materialsproject.github.io/jobflow/tutorials/1-quickstart.html
[introduction]: https://materialsproject.github.io/jobflow/tutorials/2-introduction.html
[defining-jobs]: https://materialsproject.github.io/jobflow/tutorials/3-defining-jobs.html
[creating-flows]: https://materialsproject.github.io/jobflow/tutorials/4-creating-flows.html
[dynamic-flows]: https://materialsproject.github.io/jobflow/tutorials/5-dynamic-flows.html
[makers]: https://materialsproject.github.io/jobflow/tutorials/6-makers.html
[generalized-makers]: https://materialsproject.github.io/jobflow/tutorials/7-generalized-makers.html
[jobflow-fireworks]: https://materialsproject.github.io/jobflow/tutorials/8-fireworks.html
Raw data
{
"_id": null,
"home_page": null,
"name": "jobflow",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "high-throughput, workflow",
"author": null,
"author_email": "Alex Ganose <a.ganose@imperial.ac.uk>",
"download_url": "https://files.pythonhosted.org/packages/75/06/e7a8e363eb759ec45bf283a2305f05ebb0af1eeea753e9a5e2d67a6e6343/jobflow-0.1.19.tar.gz",
"platform": null,
"description": "<div align=\"center\">\n\n# \n\n[](https://github.com/materialsproject/jobflow/actions?query=workflow%3Atesting)\n[](https://codecov.io/gh/materialsproject/jobflow/)\n[](https://pypi.org/project/jobflow/)\n\n[](https://doi.org/10.21105/joss.05995)\n\n</div>\n\n[Documentation](https://materialsproject.github.io/jobflow/) | [PyPI](https://pypi.org/project/jobflow/) | [GitHub](https://github.com/materialsproject/jobflow) | [Paper](https://doi.org/10.21105/joss.05995)\n\nJobflow is a free, open-source library for writing and executing workflows. Complex\nworkflows can be defined using simple python functions and executed locally or on\narbitrary computing resources using the [jobflow-remote][jfr] or [FireWorks][fireworks]\nworkflow managers.\n\nSome features that distinguish jobflow are dynamic workflows, easy compositing and\nconnecting of workflows, and the ability to store workflow outputs across multiple\ndatabases.\n\n## Is jobflow for me\n\njobflow is intended to be a friendly workflow software that is easy to get started with,\nbut flexible enough to handle complicated use cases.\n\nSome of its features include:\n\n- A clean and flexible Python API.\n- A powerful approach to compositing and connecting workflows \u2014 information passing\n between jobs is a key goal of jobflow. Workflows can be nested allowing for a natural\n way to build complex workflows.\n- Integration with multiple databases (MongoDB, S3, GridFS, and more) through the\n [Maggma][maggma] package.\n- Support for the [jobflow-remote][jfr] and [FireWorks][fireworks] workflow management systems, allowing workflow\n execution on multicore machines or through a queue, on a single machine or multiple\n machines.\n- Support for dynamic workflows \u2014 workflows that modify themselves or create new ones\n based on what happens during execution.\n\n## Workflow model\n\nWorkflows in jobflows are made up of two main components:\n\n- A `Job` is an atomic computing job. Essentially any python function can be `Job`,\n provided its input and return values can be serialized to json. Anything returned by the job is\n considered an \"output\" and is stored in the jobflow database.\n- A `Flow` is a collection of `Job` objects or other `Flow` objects. The connectivity\n between jobs is determined automatically from the job inputs. The execution order\n of jobs is automatically determined based on their connectivity.\n\nPython functions can be easily converted in to `Job` objects using the `@job` decorator.\nIn the example below, we define a job to add two numbers.\n\n```python\nfrom jobflow import job, Flow\n\n@job\ndef add(a, b):\n return a + b\n\nadd_first = add(1, 5)\nadd_second = add(add_first.output, 5)\n\nflow = Flow([add_first, add_second])\nflow.draw_graph().show()\n```\n\nThe output of the job is accessed using the `output` attribute. As the job has not\nyet been run, `output` contains be a reference to a future output. Outputs can be used\nas inputs to other jobs and will be automatically \"resolved\" before the job is\nexecuted.\n\nFinally, we created a flow using the two `Job` objects. The connectivity between\nthe jobs is determined automatically and can be visualised using the flow graph.\n\n<p align=\"center\">\n<img alt=\"simple flow graph\" src=\"https://raw.githubusercontent.com/materialsproject/jobflow/main/docs/_static/img/simple_flow.png\" width=\"30%\" height=\"30%\">\n</p>\n\n## Installation\n\n`jobflow` is a Python 3.9+ library and can be installed using `pip`.\n\n```bash\npip install jobflow\n```\n\n## Quickstart and tutorials\n\nTo get a first glimpse of jobflow, we suggest that you follow our quickstart tutorial.\nLater tutorials delve into the advanced features of jobflow.\n\n- [Five-minute quickstart tutorial][quickstart]\n- [Introduction to jobflow][introduction]\n- [Defining Jobs using jobflow][defining-jobs]\n\n## Need help?\n\nAsk questions about jobflow on the [jobflow support forum][help-forum].\nIf you've found an issue with jobflow, please submit a bug report on [GitHub Issues][issues].\n\n## What\u2019s new?\n\nTrack changes to jobflow through the [changelog][changelog].\n\n## Contributing\n\nWe greatly appreciate any contributions in the form of a pull request.\nAdditional information on contributing to jobflow can be found [here][contributing].\nWe maintain a list of all contributors [here][contributors].\n\n## License\n\njobflow is released under a modified BSD license; the full text can be found [here][license].\n\n## Citation\n\nIf you use Jobflow in your work, please cite it as follows:\n\n- \"Jobflow: Computational Workflows Made Simple\", A.S. Rosen, M. Gallant, J. George, J. Riebesell, H. Sahasrabuddhe, J.X. Shen, M. Wen, M.L. Evans, G. Petretto, D. Waroquiers, G.\u2011M. Rignanese, K.A. Persson, A. Jain, A.M. Ganose, _Journal of Open Source Software_, 9(93), 5995 (2024) DOI: [10.21105/joss.05995](https://doi.org/10.21105/joss.05995)\n\n## Acknowledgements\n\nJobflow was designed by Alex Ganose, Anubhav Jain, Gian-Marco Rignanese, David Waroquiers, and Guido Petretto. Alex Ganose implemented the first version of the package. Later versions have benefited from the contributions of several research groups. A full list of contributors is available [here](https://materialsproject.github.io/jobflow/contributors.html).\n\n[maggma]: https://materialsproject.github.io/maggma/\n[fireworks]: https://materialsproject.github.io/fireworks/\n[jfr]: https://matgenix.github.io/jobflow-remote\n[help-forum]: https://matsci.org/c/fireworks\n[issues]: https://github.com/materialsproject/jobflow/issues\n[changelog]: https://materialsproject.github.io/jobflow/changelog.html\n[contributing]: https://materialsproject.github.io/jobflow/contributing.html\n[contributors]: https://materialsproject.github.io/jobflow/contributors.html\n[license]: https://raw.githubusercontent.com/materialsproject/jobflow/main/LICENSE\n[quickstart]: https://materialsproject.github.io/jobflow/tutorials/1-quickstart.html\n[introduction]: https://materialsproject.github.io/jobflow/tutorials/2-introduction.html\n[defining-jobs]: https://materialsproject.github.io/jobflow/tutorials/3-defining-jobs.html\n[creating-flows]: https://materialsproject.github.io/jobflow/tutorials/4-creating-flows.html\n[dynamic-flows]: https://materialsproject.github.io/jobflow/tutorials/5-dynamic-flows.html\n[makers]: https://materialsproject.github.io/jobflow/tutorials/6-makers.html\n[generalized-makers]: https://materialsproject.github.io/jobflow/tutorials/7-generalized-makers.html\n[jobflow-fireworks]: https://materialsproject.github.io/jobflow/tutorials/8-fireworks.html\n",
"bugtrack_url": null,
"license": "modified BSD",
"summary": "jobflow is a library for writing computational workflows",
"version": "0.1.19",
"project_urls": {
"changelog": "https://github.com/materialsproject/jobflow/blob/main/CHANGELOG.md",
"documentation": "https://materialsproject.github.io/jobflow/",
"homepage": "https://materialsproject.github.io/jobflow/",
"repository": "https://github.com/materialsproject/jobflow"
},
"split_keywords": [
"high-throughput",
" workflow"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e096ba6a6531c61d55ce4ff0ade1a97eceb4aeaf21304f5921c34e0b0b95c883",
"md5": "f9ceaa52e03321b549c1ec512ae2fb08",
"sha256": "4d836ef1eaab3699aba29d1dbbc44d34a8c9294018acf21cb030f7dec1653c57"
},
"downloads": -1,
"filename": "jobflow-0.1.19-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f9ceaa52e03321b549c1ec512ae2fb08",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 58671,
"upload_time": "2024-11-22T14:27:10",
"upload_time_iso_8601": "2024-11-22T14:27:10.135166Z",
"url": "https://files.pythonhosted.org/packages/e0/96/ba6a6531c61d55ce4ff0ade1a97eceb4aeaf21304f5921c34e0b0b95c883/jobflow-0.1.19-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "7506e7a8e363eb759ec45bf283a2305f05ebb0af1eeea753e9a5e2d67a6e6343",
"md5": "df4aea6aae344b6db7d87ae11c848de0",
"sha256": "1d0420daf50f7a240f2e33e1a14f1701144aa1a24297f4c2a18825410617fd64"
},
"downloads": -1,
"filename": "jobflow-0.1.19.tar.gz",
"has_sig": false,
"md5_digest": "df4aea6aae344b6db7d87ae11c848de0",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 55318,
"upload_time": "2024-11-22T14:27:11",
"upload_time_iso_8601": "2024-11-22T14:27:11.250444Z",
"url": "https://files.pythonhosted.org/packages/75/06/e7a8e363eb759ec45bf283a2305f05ebb0af1eeea753e9a5e2d67a6e6343/jobflow-0.1.19.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-22 14:27:11",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "materialsproject",
"github_project": "jobflow",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "jobflow"
}
