| Name | lemmings-hpc JSON |
| Version |
0.9.1
JSON |
| download |
| home_page | None |
| Summary | Easy creation of workflows for recursive and farming HPC jobs |
| upload_time | 2025-10-07 10:53:05 |
| maintainer | None |
| docs_url | None |
| author | Thibault Gioud, Jimmy-John Hoste, CoopTeam-CERFACS |
| requires_python | >=3.7 |
| license | Copyright (c) 2021 Cerfacs/COOP Team
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE. |
| keywords |
hpc
job chaining
workflows
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
*Lemmings is a [1991 video game](https://en.wikipedia.org/wiki/Lemmings_(video_game)) where the player try to herd small animals, the "lemmings" out of a a 2D puzzle. Lemmings are clueless about their surroundings, walk blindly, and will eventually fall, burn, be crushed, ... well die, unless the player personally take care of them. The "Lemmings Jobs ", introduced here, are the same : by nature, these unsupervised job submission often end up in dramatic failures. Human oversight is compulsory when you are dealing with chained runs.*
## Lemmings
### Idea
Lemmings is an open-source code designed to simplify the submission of multiple inter-dependent jobs on the schedulers of HPC clusters.
While originally developed within the context of Computational Fluid Dynamics (CFD) applications, it is adapted to many recursive jobs. A farming mode is present to help the replication of these recursive jobs for a parametric study.
### Installation
Lemmings is open-source and can be pip-installed :
```bash
pip install lemmings-hpc
```
### End user POV
The end-user of lemmings is someone making a lot of simulations with a repetitive pattern.
This repetition (eg. resubmit the job until simulated time reaches 1ms) is automated by a lemmings "workflow", a python file gathering all the logic of the application. This "workflow" was created by a super user using lemmings.
Here The end-user (John Doe) adds the workflow (sandcastle) file where he usually launches the run, then run the `lemmings run` command:
```
>lemmings run --machine-file sandbox.yml --job-prefix funtask sandcastle
INFO -
##############################
Starting Lemmings 0.8.0...
##############################
INFO - Job name :funtask_PAJI77
INFO - Loop :1
INFO - Status :start
INFO - Worflow path :/Users/johndoe/productionpath/sandcastle.py
INFO - Imput path :/Users/johndoe/productionpath/sandcastle.yml
INFO - Machine path :/Users/johndoe/productionpath/sandbox.yml
INFO - Farming mode :False
INFO - Lemmings START (1/3)
INFO - Check on startTrue (False -> Exit)
INFO - Prior to job
INFO - Lemmings SPAWN (2/3)
INFO - Prepare run
INFO - Submit batch 74148
INFO - Submit batch post job 74149
```
This execution will be called `funtask_PAJI77` and will automatically submit runs through the job schedulers. On the job scheduler, he will find something like
```>qstat -u johndoe
+----------------+---------------+-------+----------+-------------------+---------+
| job name | queue | pid | state | last update | after |
+----------------+---------------+-------+----------+-------------------+---------+
| funtask_PAJI77 | long00:00:30 | 74148 | done | 06/13/22 15:22:52 | - |
| funtask_PAJI77 | short00:00:10 | 74149 | running | 06/13/22 15:22:53 | 74148 |
+----------------+---------------+-------+----------+-------------------+---------+
```
Here jobs `funtask_PAJI77_74148` and `funtask_PAJI77_74149` are the two first dependent jobs of the workflow, but more will come.
The decision to re-submit and the creation of the next job will be handled by `funtask_PAJI77_74149` after completion. *Therefore Lemmings does not "book" consecutive PID on start, only the next jobs are queued*.
Finally lemmings is not moving/hiding log files automatically. By actively limiting such "black magic", it enforces an experience similar to manual re-submission
### Creating a workflow
A super-user creates a workflow by injecting code into some parts of a baseline Loop.
The default, simplified, lemmings job follows this algorithm:
```
+-----------+ +------------+True
Start---------->|Prepare Run+--->Job submission--->Check on end+----------->Happy
^ +-----------+ +------+-----+ End
| |
| |False
| |
| |
+--------------------------------------------+
```
By adding code to **Prepare Run** phase (updates of input file) and to **Check on end** (when to stop the job), the super-user can customize it to his needs. Follows the HowTos for an extended explanation.
### Resources
Lemmings documentation can be found following this link : [lemmings documentation](https://lemmings.readthedocs.io/en/latest/)
### Acknowledgements
Lemmings is a service created in the [EXCELLERAT Center Of Excellence](https://www.excellerat.eu/wp/) and is continued as part of the [COEC Center Of Excellence](https://coec-project.eu/). Both projects are funded by the European community.
Raw data
{
"_id": null,
"home_page": null,
"name": "lemmings-hpc",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": null,
"keywords": "hpc, job chaining, workflows",
"author": "Thibault Gioud, Jimmy-John Hoste, CoopTeam-CERFACS",
"author_email": "coop@cerfacs.fr",
"download_url": "https://files.pythonhosted.org/packages/58/87/f9d236b29a560ed7af9f795955440d4dbbe765410fac62073b8e811139eb/lemmings_hpc-0.9.1.tar.gz",
"platform": null,
"description": "\n\n*Lemmings is a [1991 video game](https://en.wikipedia.org/wiki/Lemmings_(video_game)) where the player try to herd small animals, the \"lemmings\" out of a a 2D puzzle. Lemmings are clueless about their surroundings, walk blindly, and will eventually fall, burn, be crushed, ... well die, unless the player personally take care of them. The \"Lemmings Jobs \", introduced here, are the same : by nature, these unsupervised job submission often end up in dramatic failures. Human oversight is compulsory when you are dealing with chained runs.*\n\n## Lemmings\n\n### Idea\n\nLemmings is an open-source code designed to simplify the submission of multiple inter-dependent jobs on the schedulers of HPC clusters.\nWhile originally developed within the context of Computational Fluid Dynamics (CFD) applications, it is adapted to many recursive jobs. A farming mode is present to help the replication of these recursive jobs for a parametric study.\n\n### Installation\n\nLemmings is open-source and can be pip-installed :\n\n```bash\npip install lemmings-hpc\n```\n\n### End user POV\n\n\nThe end-user of lemmings is someone making a lot of simulations with a repetitive pattern.\nThis repetition (eg. resubmit the job until simulated time reaches 1ms) is automated by a lemmings \"workflow\", a python file gathering all the logic of the application. This \"workflow\" was created by a super user using lemmings.\n\nHere The end-user (John Doe) adds the workflow (sandcastle) file where he usually launches the run, then run the `lemmings run` command:\n\n\n```\n>lemmings run --machine-file sandbox.yml --job-prefix funtask sandcastle\nINFO - \n##############################\nStarting Lemmings 0.8.0...\n##############################\n\nINFO - Job name :funtask_PAJI77\nINFO - Loop :1\nINFO - Status :start\nINFO - Worflow path :/Users/johndoe/productionpath/sandcastle.py\nINFO - Imput path :/Users/johndoe/productionpath/sandcastle.yml\nINFO - Machine path :/Users/johndoe/productionpath/sandbox.yml\nINFO - Farming mode :False\nINFO - Lemmings START (1/3)\nINFO - Check on startTrue (False -> Exit)\nINFO - Prior to job\nINFO - Lemmings SPAWN (2/3)\nINFO - Prepare run\nINFO - Submit batch 74148 \nINFO - Submit batch post job 74149\n```\n\nThis execution will be called `funtask_PAJI77` and will automatically submit runs through the job schedulers. On the job scheduler, he will find something like\n\n```>qstat -u johndoe\n+----------------+---------------+-------+----------+-------------------+---------+\n| job name | queue | pid | state | last update | after |\n+----------------+---------------+-------+----------+-------------------+---------+\n| funtask_PAJI77 | long00:00:30 | 74148 | done | 06/13/22 15:22:52 | - |\n| funtask_PAJI77 | short00:00:10 | 74149 | running | 06/13/22 15:22:53 | 74148 |\n+----------------+---------------+-------+----------+-------------------+---------+\n```\n\nHere jobs `funtask_PAJI77_74148` and `funtask_PAJI77_74149` are the two first dependent jobs of the workflow, but more will come.\nThe decision to re-submit and the creation of the next job will be handled by `funtask_PAJI77_74149` after completion. *Therefore Lemmings does not \"book\" consecutive PID on start, only the next jobs are queued*. \n\nFinally lemmings is not moving/hiding log files automatically. By actively limiting such \"black magic\", it enforces an experience similar to manual re-submission\n\n### Creating a workflow\n\nA super-user creates a workflow by injecting code into some parts of a baseline Loop.\nThe default, simplified, lemmings job follows this algorithm:\n\n```\n +-----------+ +------------+True \nStart---------->|Prepare Run+--->Job submission--->Check on end+----------->Happy\n ^ +-----------+ +------+-----+ End\n | |\n | |False\n | |\n | |\n +--------------------------------------------+ \n```\n\nBy adding code to **Prepare Run** phase (updates of input file) and to **Check on end** (when to stop the job), the super-user can customize it to his needs. Follows the HowTos for an extended explanation.\n\n\n### Resources\n\nLemmings documentation can be found following this link : [lemmings documentation](https://lemmings.readthedocs.io/en/latest/)\n\n### Acknowledgements\n\nLemmings is a service created in the [EXCELLERAT Center Of Excellence](https://www.excellerat.eu/wp/) and is continued as part of the [COEC Center Of Excellence](https://coec-project.eu/). Both projects are funded by the European community.\n\n\n\n\n\n",
"bugtrack_url": null,
"license": "Copyright (c) 2021 Cerfacs/COOP Team\n \n Permission is hereby granted, free of charge, to any person obtaining a copy\n of this software and associated documentation files (the \"Software\"), to deal\n in the Software without restriction, including without limitation the rights\n to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n copies of the Software, and to permit persons to whom the Software is\n furnished to do so, subject to the following conditions:\n \n The above copyright notice and this permission notice shall be included in all\n copies or substantial portions of the Software.\n \n THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n SOFTWARE.",
"summary": "Easy creation of workflows for recursive and farming HPC jobs",
"version": "0.9.1",
"project_urls": {
"Documentation": "https://lemmings.readthedocs.io/en/latest/",
"Homepage": "https://gitlab.com/cerfacs/lemmings",
"Issues": "https://gitlab.com/cerfacs/lemmings/-/issues"
},
"split_keywords": [
"hpc",
" job chaining",
" workflows"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "cade21481fb4c6eaf2d39b5287c33bcdac0a940f9905da39b9be617d787a576a",
"md5": "20647de4a630dd0058cf8ae14a4eccb3",
"sha256": "8e4d9c9061e68add83c5fb3c6a336a1a9a4bdb7a3a147098fa8ee34ee134bf69"
},
"downloads": -1,
"filename": "lemmings_hpc-0.9.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "20647de4a630dd0058cf8ae14a4eccb3",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 208496,
"upload_time": "2025-10-07T10:53:03",
"upload_time_iso_8601": "2025-10-07T10:53:03.359957Z",
"url": "https://files.pythonhosted.org/packages/ca/de/21481fb4c6eaf2d39b5287c33bcdac0a940f9905da39b9be617d787a576a/lemmings_hpc-0.9.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "5887f9d236b29a560ed7af9f795955440d4dbbe765410fac62073b8e811139eb",
"md5": "2e60bfd980dab5ad250ad176a55b9dc0",
"sha256": "d8a184aab91d1ad7bd683705f94166ba8c33d9117d863f47976ec89a39a04fa7"
},
"downloads": -1,
"filename": "lemmings_hpc-0.9.1.tar.gz",
"has_sig": false,
"md5_digest": "2e60bfd980dab5ad250ad176a55b9dc0",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 140185,
"upload_time": "2025-10-07T10:53:05",
"upload_time_iso_8601": "2025-10-07T10:53:05.134985Z",
"url": "https://files.pythonhosted.org/packages/58/87/f9d236b29a560ed7af9f795955440d4dbbe765410fac62073b8e811139eb/lemmings_hpc-0.9.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-10-07 10:53:05",
"github": false,
"gitlab": true,
"bitbucket": false,
"codeberg": false,
"gitlab_user": "cerfacs",
"gitlab_project": "lemmings",
"lcname": "lemmings-hpc"
}
