HyperShell v2: Distributed Task Execution for HPC
=================================================
.. image:: https://img.shields.io/badge/license-Apache-blue.svg?style=flat
:target: https://www.apache.org/licenses/LICENSE-2.0
:alt: License
.. image:: https://img.shields.io/github/v/release/hypershell/hypershell?sort=semver
:target: https://github.com/hypershell/hypershell/releases
:alt: Github Release
.. image:: https://img.shields.io/badge/Python-3.9+-blue.svg
:target: https://www.python.org/downloads
:alt: Python Versions
.. image:: https://img.shields.io/badge/Contributor%20Covenant-2.1-4baaaa.svg
:target: https://www.contributor-covenant.org/version/2/1/code_of_conduct/
:alt: Code of Conduct
.. image:: https://readthedocs.org/projects/hypershell/badge/?version=latest
:target: https://hypershell.readthedocs.io/en/latest/?badge=latest
:alt: Documentation Status
|
*HyperShell* is an elegant, cross-platform, high-throughput computing utility for
processing shell commands over a distributed, asynchronous queue. It is a highly
scalable workflow automation tool for *many-task* scenarios.
Built on Python and tested on Linux, macOS, and Windows.
Several tools offer similar functionality but not all together in a single tool with
the user ergonomics we provide. Novel design elements include but are not limited to
* **Cross-platform:** run on any platform where Python runs. In fact, the server and
client can run on different platforms in the same cluster.
* **Client-server:** workloads do not need to be monolithic. Run the server as a
stand-alone service with SQLite or Postgres as a persistent database and dynamically
scale clients as needed.
* **Staggered launch:** At the largest scales (1000s of nodes, 100k+ of workers),
the launch process can be challenging. Come up gradually to balance the workload.
* **Database in-the-loop:** run in-memory for quick, ad-hoc workloads. Otherwise,
include a database for persistence, recovery when restarting, and search.
Documentation
-------------
Documentation is available at `hypershell.readthedocs.io <https://hypershell.readthedocs.io>`_.
For basic usage information on the command line use: ``hs --help``. For a more
comprehensive usage guide on the command line you can view the manual page with
``man hs``.
Contributions
-------------
Contributions are welcome. If you find bugs or have questions, open an *Issue* here.
We've added a Code of Conduct recently, adapted from the
`Contributor Covenant <https://www.contributor-covenant.org/>`_, version 2.0.
Citation
--------
If *HyperShell* has helped in your research please consider citing us.
.. code-block:: bibtex
@inproceedings{lentner_2022,
author = {Lentner, Geoffrey and Gorenstein, Lev},
title = {HyperShell v2: Distributed Task Execution for HPC},
year = {2022},
isbn = {9781450391610},
publisher = {Association for Computing Machinery},
url = {https://doi.org/10.1145/3491418.3535138},
doi = {10.1145/3491418.3535138},
booktitle = {Practice and Experience in Advanced Research Computing},
articleno = {80},
numpages = {3},
series = {PEARC '22}
}
Raw data
{
"_id": null,
"home_page": "https://hypershell.org",
"name": "hypershell",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.9",
"maintainer_email": null,
"keywords": "distributed-computing, command-line-tool, shell-scripting, high-performance-computing, high-throughput-computing",
"author": "Geoffrey Lentner",
"author_email": "glentner@purdue.edu",
"download_url": "https://files.pythonhosted.org/packages/4e/03/e060166a066cd8d4ec74c7eed67d351b841014dd0c11ace053207e9dac57/hypershell-2.6.4.tar.gz",
"platform": null,
"description": "HyperShell v2: Distributed Task Execution for HPC\n=================================================\n\n.. image:: https://img.shields.io/badge/license-Apache-blue.svg?style=flat\n :target: https://www.apache.org/licenses/LICENSE-2.0\n :alt: License\n\n.. image:: https://img.shields.io/github/v/release/hypershell/hypershell?sort=semver\n :target: https://github.com/hypershell/hypershell/releases\n :alt: Github Release\n\n.. image:: https://img.shields.io/badge/Python-3.9+-blue.svg\n :target: https://www.python.org/downloads\n :alt: Python Versions\n\n.. image:: https://img.shields.io/badge/Contributor%20Covenant-2.1-4baaaa.svg\n :target: https://www.contributor-covenant.org/version/2/1/code_of_conduct/\n :alt: Code of Conduct\n\n.. image:: https://readthedocs.org/projects/hypershell/badge/?version=latest\n :target: https://hypershell.readthedocs.io/en/latest/?badge=latest\n :alt: Documentation Status\n\n|\n\n*HyperShell* is an elegant, cross-platform, high-throughput computing utility for\nprocessing shell commands over a distributed, asynchronous queue. It is a highly\nscalable workflow automation tool for *many-task* scenarios.\n\nBuilt on Python and tested on Linux, macOS, and Windows.\n\nSeveral tools offer similar functionality but not all together in a single tool with\nthe user ergonomics we provide. Novel design elements include but are not limited to\n\n* **Cross-platform:** run on any platform where Python runs. In fact, the server and\n client can run on different platforms in the same cluster.\n* **Client-server:** workloads do not need to be monolithic. Run the server as a\n stand-alone service with SQLite or Postgres as a persistent database and dynamically\n scale clients as needed.\n* **Staggered launch:** At the largest scales (1000s of nodes, 100k+ of workers),\n the launch process can be challenging. Come up gradually to balance the workload.\n* **Database in-the-loop:** run in-memory for quick, ad-hoc workloads. Otherwise,\n include a database for persistence, recovery when restarting, and search.\n\n\nDocumentation\n-------------\n\nDocumentation is available at `hypershell.readthedocs.io <https://hypershell.readthedocs.io>`_.\nFor basic usage information on the command line use: ``hs --help``. For a more\ncomprehensive usage guide on the command line you can view the manual page with \n``man hs``.\n\n\nContributions\n-------------\n\nContributions are welcome. If you find bugs or have questions, open an *Issue* here.\nWe've added a Code of Conduct recently, adapted from the\n`Contributor Covenant <https://www.contributor-covenant.org/>`_, version 2.0.\n\n\nCitation\n--------\n\nIf *HyperShell* has helped in your research please consider citing us.\n\n.. code-block:: bibtex\n\n @inproceedings{lentner_2022,\n author = {Lentner, Geoffrey and Gorenstein, Lev},\n title = {HyperShell v2: Distributed Task Execution for HPC},\n year = {2022},\n isbn = {9781450391610},\n publisher = {Association for Computing Machinery},\n url = {https://doi.org/10.1145/3491418.3535138},\n doi = {10.1145/3491418.3535138},\n booktitle = {Practice and Experience in Advanced Research Computing},\n articleno = {80},\n numpages = {3},\n series = {PEARC '22}\n }\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "A cross-platform, high-throughput computing utility for processing shell commands over a distributed, asynchronous queue.",
"version": "2.6.4",
"project_urls": {
"Documentation": "https://hypershell.readthedocs.io",
"Homepage": "https://hypershell.org",
"Repository": "https://github.com/hypershell/hypershell"
},
"split_keywords": [
"distributed-computing",
" command-line-tool",
" shell-scripting",
" high-performance-computing",
" high-throughput-computing"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "316cd35d78d0ad509124ef1433270f454c9b4faafe580a6e789b551607a9f29c",
"md5": "de6625b574b642c07b2fa0b9aa1b1e2b",
"sha256": "3e46c94284a3348d16152a53a150b5e6d5705300db0219830551e2de60f55ad0"
},
"downloads": -1,
"filename": "hypershell-2.6.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "de6625b574b642c07b2fa0b9aa1b1e2b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.9",
"size": 98654,
"upload_time": "2025-01-17T02:11:34",
"upload_time_iso_8601": "2025-01-17T02:11:34.175576Z",
"url": "https://files.pythonhosted.org/packages/31/6c/d35d78d0ad509124ef1433270f454c9b4faafe580a6e789b551607a9f29c/hypershell-2.6.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "4e03e060166a066cd8d4ec74c7eed67d351b841014dd0c11ace053207e9dac57",
"md5": "45ee64087fbd953a04c1455abf695fa5",
"sha256": "69c04bd83ab467f9c88613598f3b757710f974b7fca03c22bb7521b0851c5f9e"
},
"downloads": -1,
"filename": "hypershell-2.6.4.tar.gz",
"has_sig": false,
"md5_digest": "45ee64087fbd953a04c1455abf695fa5",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.9",
"size": 78882,
"upload_time": "2025-01-17T02:11:36",
"upload_time_iso_8601": "2025-01-17T02:11:36.592867Z",
"url": "https://files.pythonhosted.org/packages/4e/03/e060166a066cd8d4ec74c7eed67d351b841014dd0c11ace053207e9dac57/hypershell-2.6.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-17 02:11:36",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "hypershell",
"github_project": "hypershell",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "hypershell"
}
JSON
