BEE: Build and Execution Environment
************************************
BEE is a workflow orchestration system designed to build containerized HPC applications and orchestrate workflows across HPC and cloud systems. BEE has adopted the Common Workflow Language (`CWL <https://www.commonwl.org/>`_) for specifying workflows. Complex scientific workflows specified by CWL are managed and visualized through a graph database, giving the user the ability to monitor the state of each task in the workflow. BEE runs jobs using the workload scheduler (i.e. Slurm or LSF) on the HPC system that tasks are specified to run on.
BEE workflows can be archived for provenance and reproducibility. BEE can orchestrate workflows with containerized applications or those built locally on a system. However, there are advantages to containerizing an application.
A container is a package of code (usually binaries) and all of that code's dependencies (libraries, etc.). Once built, this container can be run on many different platforms.
Containers provide many benefits:
* Users can choose their own software stack (libraries, compilers, etc.) and not be bound by the currently installed environment on any one machine.
* Codes can be run portably across numerous platforms--all dependencies will be downloaded and installed at run time.
* Entire **workflow** environments can be built into one or more containers. A user can include visualization and analysis tools along with the application. They will all work together as the application runs.
* Provenance and history can be tracked by storing containers in a historical repository. At any time, an older container can be rerun (all of its dependencies are stored with it). Execution is repeatable and interactions between software components can be tracked.
* Functional testing can be performed on smaller, dissimilar machines--there is no real need to test on the actual HPC platform (performance testing obviously requires target hardware).
BEE Sites
=========
* Documentation: `https://lanl.github.io/BEE/ <https://lanl.github.io/BEE/>`_
* Github: `https://github.com/lanl/BEE <https://github.com/lanl/BEE>`_
Contact
=======
For bugs and problems report, suggestions and other general questions regarding the BEE project, email questions to `bee-dev@lanl.gov <bee-dev@lanl.gov>`_.
Contributors:
==========================
* Steven Anaya - `Boogie3D <https://github.com/Boogie3D>`_
* Paul Bryant - `paulbry <https://github.com/paulbry>`_
* Rusty Davis - `rstyd <https://github.com/rstyd>`_
* Jieyang Chen - `JieyangChen7 <https://github.com/JieyangChen7>`_
* Krishna Chilleri - `Krishna Chilleri <https://github.com/kchilleri>`_
* Patricia Grubel - `pagrubel <https://github.com/pagrubel>`_
* Qiang Guan - `guanxyz <https://github.com/guanxyz>`_
* Ragini Gupta - `raginigupta6 <https://github.com/raginigupta6>`_
* Andres Quan - `aquan9 <https://github.com/aquan9>`_
* Quincy Wofford - `qwofford <https://github.com/qwofford>`_
* Tim Randles - `trandles-lanl <https://github.com/trandles-lanl>`_
* Jacob Tronge - `jtronge <https://github.com/jtronge>`_
Concept and Design Contributors
* James Ahrens
* Allen McPherson
* Li-Ta Lo
* Louis Vernon
Contributing
==========================
The BEE project adheres to style guidelines specified in `setup.cfg <https://github.com/lanl/BEE/blob/master/setup\.cfg>`_. Before attempting to commit and push changes, please install our pre-commit githooks by running the following command in project root:
If using `git --version` >= 2.9:
git config core.hooksPath .githooks
Otherwise:
cp .githooks/* .git/hooks/
Using these git hooks will ensure your contributions adhere to style guidelines required for contribution. You will need to repeat these steps for every `BEE` repo you clone.
Release
==========================
This software has been approved for open source release and has been assigned **BEE C17056**.
Copyright
==========================
License can be found `here <https://github.com/lanl/BEE/blob/master/LICENSE>`_
Publications
==========================
- An HPC-Container Based Continuous Integration Tool for Detecting Scaling and Performance Issues in HPC Applications, IEEE Transactions on Services Computing, 2024, `DOI: 10.1109/TSC.2023.3337662 <https://doi.ieeecomputersociety.org/10.1109/TSC.2023.3337662>`_
- BEE Orchestrator: Running Complex Scientific Workflows on Multiple Systems, HiPC, 2021, `DOI: 10.1109/HiPC53243.2021.00052 <https://doi.org/10.1109/HiPC53243.2021.00052>`_
- "BeeSwarm: Enabling Parallel Scaling Performance Measurement in Continuous Integration for HPC Applications", ASE, 2021, `DOI: 10.1109/ASE51524.2021.9678805 <https://www.computer.org/csdl/proceedings-article/ase/2021/033700b136/1AjTjgnW2pa#:~:text=10.1109/ASE51524.2021.9678805>`_
- "BeeFlow: A Workflow Management System for In Situ Processing across HPC and Cloud Systems", ICDCS, 2018, `DOI: 10.1109/ICDCS.2018.00103 <https://ieeexplore.ieee.org/abstract/document/8416366>`_
- "Build and execution environment (BEE): an encapsulated environment enabling HPC applications running everywhere", IEEE BigData, 2018, `DOI: 10.1109/BigData.2018.8622572 <https://ieeexplore.ieee.org/document/8622572>`_
Raw data
{
"_id": null,
"home_page": "https://github.com/lanl/BEE",
"name": "hpc-beeflow",
"maintainer": null,
"docs_url": null,
"requires_python": "<=3.12.2,>=3.8.3",
"maintainer_email": null,
"keywords": "bee, hpc, workflow, cluster, computing",
"author": "BEE-LANL Dev Team",
"author_email": "bee-dev@lanl.gov",
"download_url": "https://files.pythonhosted.org/packages/49/ca/095d38d5f9768f5c33766a2e51f4571a201ea0c4281c2aa581796be3e8fa/hpc-beeflow-0.1.7.tar.gz",
"platform": null,
"description": "BEE: Build and Execution Environment\n************************************\n\nBEE is a workflow orchestration system designed to build containerized HPC applications and orchestrate workflows across HPC and cloud systems. BEE has adopted the Common Workflow Language (`CWL <https://www.commonwl.org/>`_) for specifying workflows. Complex scientific workflows specified by CWL are managed and visualized through a graph database, giving the user the ability to monitor the state of each task in the workflow. BEE runs jobs using the workload scheduler (i.e. Slurm or LSF) on the HPC system that tasks are specified to run on.\n\nBEE workflows can be archived for provenance and reproducibility. BEE can orchestrate workflows with containerized applications or those built locally on a system. However, there are advantages to containerizing an application.\n\nA container is a package of code (usually binaries) and all of that code's dependencies (libraries, etc.). Once built, this container can be run on many different platforms.\n\nContainers provide many benefits:\n\n* Users can choose their own software stack (libraries, compilers, etc.) and not be bound by the currently installed environment on any one machine.\n\n* Codes can be run portably across numerous platforms--all dependencies will be downloaded and installed at run time.\n\n* Entire **workflow** environments can be built into one or more containers. A user can include visualization and analysis tools along with the application. They will all work together as the application runs.\n\n* Provenance and history can be tracked by storing containers in a historical repository. At any time, an older container can be rerun (all of its dependencies are stored with it). Execution is repeatable and interactions between software components can be tracked.\n\n* Functional testing can be performed on smaller, dissimilar machines--there is no real need to test on the actual HPC platform (performance testing obviously requires target hardware).\n\n\nBEE Sites\n=========\n\n* Documentation: `https://lanl.github.io/BEE/ <https://lanl.github.io/BEE/>`_\n\n* Github: `https://github.com/lanl/BEE <https://github.com/lanl/BEE>`_\n\n\nContact\n=======\n\n\nFor bugs and problems report, suggestions and other general questions regarding the BEE project, email questions to `bee-dev@lanl.gov <bee-dev@lanl.gov>`_.\n\n\nContributors:\n==========================\n\n* Steven Anaya - `Boogie3D <https://github.com/Boogie3D>`_\n* Paul Bryant - `paulbry <https://github.com/paulbry>`_\n* Rusty Davis - `rstyd <https://github.com/rstyd>`_\n* Jieyang Chen - `JieyangChen7 <https://github.com/JieyangChen7>`_\n* Krishna Chilleri - `Krishna Chilleri <https://github.com/kchilleri>`_\n* Patricia Grubel - `pagrubel <https://github.com/pagrubel>`_\n* Qiang Guan - `guanxyz <https://github.com/guanxyz>`_\n* Ragini Gupta - `raginigupta6 <https://github.com/raginigupta6>`_\n* Andres Quan - `aquan9 <https://github.com/aquan9>`_\n* Quincy Wofford - `qwofford <https://github.com/qwofford>`_\n* Tim Randles - `trandles-lanl <https://github.com/trandles-lanl>`_\n* Jacob Tronge - `jtronge <https://github.com/jtronge>`_\n\nConcept and Design Contributors\n\n* James Ahrens\n* Allen McPherson\n* Li-Ta Lo\n* Louis Vernon\n\n\nContributing\n==========================\n\nThe BEE project adheres to style guidelines specified in `setup.cfg <https://github.com/lanl/BEE/blob/master/setup\\.cfg>`_. Before attempting to commit and push changes, please install our pre-commit githooks by running the following command in project root:\n\nIf using `git --version` >= 2.9:\n git config core.hooksPath .githooks\n\nOtherwise:\n cp .githooks/* .git/hooks/\n\nUsing these git hooks will ensure your contributions adhere to style guidelines required for contribution. You will need to repeat these steps for every `BEE` repo you clone.\n\n\nRelease\n==========================\n\nThis software has been approved for open source release and has been assigned **BEE C17056**.\n\nCopyright\n==========================\nLicense can be found `here <https://github.com/lanl/BEE/blob/master/LICENSE>`_\n\n\nPublications\n==========================\n\n- An HPC-Container Based Continuous Integration Tool for Detecting Scaling and Performance Issues in HPC Applications, IEEE Transactions on Services Computing, 2024, `DOI: 10.1109/TSC.2023.3337662 <https://doi.ieeecomputersociety.org/10.1109/TSC.2023.3337662>`_\n- BEE Orchestrator: Running Complex Scientific Workflows on Multiple Systems, HiPC, 2021, `DOI: 10.1109/HiPC53243.2021.00052 <https://doi.org/10.1109/HiPC53243.2021.00052>`_\n- \"BeeSwarm: Enabling Parallel Scaling Performance Measurement in Continuous Integration for HPC Applications\", ASE, 2021, `DOI: 10.1109/ASE51524.2021.9678805 <https://www.computer.org/csdl/proceedings-article/ase/2021/033700b136/1AjTjgnW2pa#:~:text=10.1109/ASE51524.2021.9678805>`_\n- \"BeeFlow: A Workflow Management System for In Situ Processing across HPC and Cloud Systems\", ICDCS, 2018, `DOI: 10.1109/ICDCS.2018.00103 <https://ieeexplore.ieee.org/abstract/document/8416366>`_\n- \"Build and execution environment (BEE): an encapsulated environment enabling HPC applications running everywhere\", IEEE BigData, 2018, `DOI: 10.1109/BigData.2018.8622572 <https://ieeexplore.ieee.org/document/8622572>`_\n",
"bugtrack_url": null,
"license": null,
"summary": "A software package for containerizing HPC applications and managing job workflows",
"version": "0.1.7",
"project_urls": {
"Homepage": "https://github.com/lanl/BEE",
"Repository": "https://github.com/lanl/BEE"
},
"split_keywords": [
"bee",
" hpc",
" workflow",
" cluster",
" computing"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "887729f82166a41df2b452d7f72125ad8e33e74dc6d5721a1416d5e7e718b064",
"md5": "944b17393905ff30e36ab054e133aa36",
"sha256": "16c29a4b68ae2e0180460ce3a498e24ef1e9b34f8db9c6a3828e6eec760d7d05"
},
"downloads": -1,
"filename": "hpc_beeflow-0.1.7-py3-none-any.whl",
"has_sig": false,
"md5_digest": "944b17393905ff30e36ab054e133aa36",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<=3.12.2,>=3.8.3",
"size": 759810,
"upload_time": "2024-04-01T17:18:10",
"upload_time_iso_8601": "2024-04-01T17:18:10.009597Z",
"url": "https://files.pythonhosted.org/packages/88/77/29f82166a41df2b452d7f72125ad8e33e74dc6d5721a1416d5e7e718b064/hpc_beeflow-0.1.7-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "49ca095d38d5f9768f5c33766a2e51f4571a201ea0c4281c2aa581796be3e8fa",
"md5": "64407bc5a1fe945625c17bf20bcdb669",
"sha256": "ed37480d463f149187c4d3ea9fdb2ac3d2376efa9e79afa009c3fa66c04d09de"
},
"downloads": -1,
"filename": "hpc-beeflow-0.1.7.tar.gz",
"has_sig": false,
"md5_digest": "64407bc5a1fe945625c17bf20bcdb669",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<=3.12.2,>=3.8.3",
"size": 634413,
"upload_time": "2024-04-01T17:18:06",
"upload_time_iso_8601": "2024-04-01T17:18:06.977440Z",
"url": "https://files.pythonhosted.org/packages/49/ca/095d38d5f9768f5c33766a2e51f4571a201ea0c4281c2aa581796be3e8fa/hpc-beeflow-0.1.7.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-01 17:18:06",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "lanl",
"github_project": "BEE",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "hpc-beeflow"
}
JSON
