*Lemmings is a [1991 video game](https://en.wikipedia.org/wiki/Lemmings_(video_game)) where the player try to herd small animals, the "lemmings" out of a a 2D puzzle. Lemmings are clueless about their surroundings, walk blindly, and will eventually fall, burn, be crushed, ... well die, unless the player personally take care of them. The "Lemmings Jobs ", introduced here, are the same : by nature, these unsupervised job submission often end up in dramatic failures. Human oversight is compulsory when you are dealing with chained runs.*
## Lemmings
### Idea
Lemmings is an open-source code designed to simplify the submission of multiple inter-dependent jobs on the schedulers of HPC clusters.
While originally developed within the context of Computational Fluid Dynamics (CFD) applications, it is adapted to many recursive jobs. A farming mode is present to help the replication of these recursive jobs for a parametric study.
### Installation
Lemmings is open-source and can be pip-installed :
```bash
pip install lemmings-hpc
```
### End user POV
The end-user of lemmings is someone making a lot of simulations with a repetitive pattern.
This repetition (eg. resubmit the job until simulated time reaches 1ms) is automated by a lemmings "workflow", a python file gathering all the logic of the application. This "workflow" was created by a super user using lemmings.
Here The end-user (John Doe) adds the workflow (sandcastle) file where he usually launches the run, then run the `lemmings run` command:
```
>lemmings run --machine-file sandbox.yml --job-prefix funtask sandcastle
INFO -
##############################
Starting Lemmings 0.8.0...
##############################
INFO - Job name :funtask_PAJI77
INFO - Loop :1
INFO - Status :start
INFO - Worflow path :/Users/johndoe/productionpath/sandcastle.py
INFO - Imput path :/Users/johndoe/productionpath/sandcastle.yml
INFO - Machine path :/Users/johndoe/productionpath/sandbox.yml
INFO - Farming mode :False
INFO - Lemmings START (1/3)
INFO - Check on startTrue (False -> Exit)
INFO - Prior to job
INFO - Lemmings SPAWN (2/3)
INFO - Prepare run
INFO - Submit batch 74148
INFO - Submit batch post job 74149
```
This execution will be called `funtask_PAJI77` and will automatically submit runs through the job schedulers. On the job scheduler, he will find something like
```>qstat -u johndoe
+----------------+---------------+-------+----------+-------------------+---------+
| job name | queue | pid | state | last update | after |
+----------------+---------------+-------+----------+-------------------+---------+
| funtask_PAJI77 | long00:00:30 | 74148 | done | 06/13/22 15:22:52 | - |
| funtask_PAJI77 | short00:00:10 | 74149 | running | 06/13/22 15:22:53 | 74148 |
+----------------+---------------+-------+----------+-------------------+---------+
```
Here jobs `funtask_PAJI77_74148` and `funtask_PAJI77_74149` are the two first dependent jobs of the workflow, but more will come.
The decision to re-submit and the creation of the next job will be handled by `funtask_PAJI77_74149` after completion. *Therefore Lemmings does not "book" consecutive PID on start, only the next jobs are queued*.
Finally lemmings is not moving/hiding log files automatically. By actively limiting such "black magic", it enforces an experience similar to manual re-submission
### Creating a workflow
A super-user creates a workflow by injecting code into some parts of a baseline Loop.
The default, simplified, lemmings job follows this algorithm:
```
+-----------+ +------------+True
Start---------->|Prepare Run+--->Job submission--->Check on end+----------->Happy
^ +-----------+ +------+-----+ End
| |
| |False
| |
| |
+--------------------------------------------+
```
By adding code to **Prepare Run** phase (updates of input file) and to **Check on end** (when to stop the job), the super-user can customize it to his needs. Follows the HowTos for an extended explanation.
### Resources
Lemmings documentation can be found following this link : [lemmings documentation](https://lemmings.readthedocs.io/en/latest/)
### Acknowledgements
Lemmings is a service created in the [EXCELLERAT Center Of Excellence](https://www.excellerat.eu/wp/) and is continued as part of the [COEC Center Of Excellence](https://coec-project.eu/). Both projects are funded by the European community.
Raw data
{
"_id": null,
"home_page": "https://gitlab.com/cerfacs/lemmings",
"name": "lemmings-hpc",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "hpc,job chaining,workflows",
"author": "Thibault Gioud, Jimmy-John Hoste, CoopTeam-CERFACS",
"author_email": "coop@cerfacs.fr",
"download_url": "https://files.pythonhosted.org/packages/94/1f/fbba8523f77579827529989056c60b805a9749ff58e3d2d0e2e1f91ec0a8/lemmings-hpc-0.8.1.tar.gz",
"platform": null,
"description": "\n\n*Lemmings is a [1991 video game](https://en.wikipedia.org/wiki/Lemmings_(video_game)) where the player try to herd small animals, the \"lemmings\" out of a a 2D puzzle. Lemmings are clueless about their surroundings, walk blindly, and will eventually fall, burn, be crushed, ... well die, unless the player personally take care of them. The \"Lemmings Jobs \", introduced here, are the same : by nature, these unsupervised job submission often end up in dramatic failures. Human oversight is compulsory when you are dealing with chained runs.*\n\n## Lemmings\n\n### Idea\n\nLemmings is an open-source code designed to simplify the submission of multiple inter-dependent jobs on the schedulers of HPC clusters.\nWhile originally developed within the context of Computational Fluid Dynamics (CFD) applications, it is adapted to many recursive jobs. A farming mode is present to help the replication of these recursive jobs for a parametric study.\n\n### Installation\n\nLemmings is open-source and can be pip-installed :\n\n```bash\npip install lemmings-hpc\n```\n\n### End user POV\n\n\nThe end-user of lemmings is someone making a lot of simulations with a repetitive pattern.\nThis repetition (eg. resubmit the job until simulated time reaches 1ms) is automated by a lemmings \"workflow\", a python file gathering all the logic of the application. This \"workflow\" was created by a super user using lemmings.\n\nHere The end-user (John Doe) adds the workflow (sandcastle) file where he usually launches the run, then run the `lemmings run` command:\n\n\n```\n>lemmings run --machine-file sandbox.yml --job-prefix funtask sandcastle\nINFO - \n##############################\nStarting Lemmings 0.8.0...\n##############################\n\nINFO - Job name :funtask_PAJI77\nINFO - Loop :1\nINFO - Status :start\nINFO - Worflow path :/Users/johndoe/productionpath/sandcastle.py\nINFO - Imput path :/Users/johndoe/productionpath/sandcastle.yml\nINFO - Machine path :/Users/johndoe/productionpath/sandbox.yml\nINFO - Farming mode :False\nINFO - Lemmings START (1/3)\nINFO - Check on startTrue (False -> Exit)\nINFO - Prior to job\nINFO - Lemmings SPAWN (2/3)\nINFO - Prepare run\nINFO - Submit batch 74148 \nINFO - Submit batch post job 74149\n```\n\nThis execution will be called `funtask_PAJI77` and will automatically submit runs through the job schedulers. On the job scheduler, he will find something like\n\n```>qstat -u johndoe\n+----------------+---------------+-------+----------+-------------------+---------+\n| job name | queue | pid | state | last update | after |\n+----------------+---------------+-------+----------+-------------------+---------+\n| funtask_PAJI77 | long00:00:30 | 74148 | done | 06/13/22 15:22:52 | - |\n| funtask_PAJI77 | short00:00:10 | 74149 | running | 06/13/22 15:22:53 | 74148 |\n+----------------+---------------+-------+----------+-------------------+---------+\n```\n\nHere jobs `funtask_PAJI77_74148` and `funtask_PAJI77_74149` are the two first dependent jobs of the workflow, but more will come.\nThe decision to re-submit and the creation of the next job will be handled by `funtask_PAJI77_74149` after completion. *Therefore Lemmings does not \"book\" consecutive PID on start, only the next jobs are queued*. \n\nFinally lemmings is not moving/hiding log files automatically. By actively limiting such \"black magic\", it enforces an experience similar to manual re-submission\n\n### Creating a workflow\n\nA super-user creates a workflow by injecting code into some parts of a baseline Loop.\nThe default, simplified, lemmings job follows this algorithm:\n\n```\n +-----------+ +------------+True \nStart---------->|Prepare Run+--->Job submission--->Check on end+----------->Happy\n ^ +-----------+ +------+-----+ End\n | |\n | |False\n | |\n | |\n +--------------------------------------------+ \n```\n\nBy adding code to **Prepare Run** phase (updates of input file) and to **Check on end** (when to stop the job), the super-user can customize it to his needs. Follows the HowTos for an extended explanation.\n\n\n### Resources\n\nLemmings documentation can be found following this link : [lemmings documentation](https://lemmings.readthedocs.io/en/latest/)\n\n### Acknowledgements\n\nLemmings is a service created in the [EXCELLERAT Center Of Excellence](https://www.excellerat.eu/wp/) and is continued as part of the [COEC Center Of Excellence](https://coec-project.eu/). Both projects are funded by the European community.\n\n\n\n\n\n",
"bugtrack_url": null,
"license": "",
"summary": "Easy creation of workflows for recursive and farming HPC jobs",
"version": "0.8.1",
"split_keywords": [
"hpc",
"job chaining",
"workflows"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c0c0fd005f52b62f1c96f64665dfac826d2939868a25e8f81ec4215175bc09a8",
"md5": "54fca9071ed48b2494601ae20cb258b3",
"sha256": "14691f5969bf3c722c3f8907326279569d74316948c3002e12bde1f898c9cb20"
},
"downloads": -1,
"filename": "lemmings_hpc-0.8.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "54fca9071ed48b2494601ae20cb258b3",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 33921,
"upload_time": "2023-02-03T09:34:12",
"upload_time_iso_8601": "2023-02-03T09:34:12.092312Z",
"url": "https://files.pythonhosted.org/packages/c0/c0/fd005f52b62f1c96f64665dfac826d2939868a25e8f81ec4215175bc09a8/lemmings_hpc-0.8.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "941ffbba8523f77579827529989056c60b805a9749ff58e3d2d0e2e1f91ec0a8",
"md5": "c4f1a8661ee7087c442ea90ed454c86b",
"sha256": "83bacb6a0a1cbe18004f8b35058a2df9e9104a29d70f0a04b854a760ec68aa56"
},
"downloads": -1,
"filename": "lemmings-hpc-0.8.1.tar.gz",
"has_sig": false,
"md5_digest": "c4f1a8661ee7087c442ea90ed454c86b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 60194,
"upload_time": "2023-02-03T09:34:13",
"upload_time_iso_8601": "2023-02-03T09:34:13.898100Z",
"url": "https://files.pythonhosted.org/packages/94/1f/fbba8523f77579827529989056c60b805a9749ff58e3d2d0e2e1f91ec0a8/lemmings-hpc-0.8.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-02-03 09:34:13",
"github": false,
"gitlab": true,
"bitbucket": false,
"gitlab_user": "cerfacs",
"gitlab_project": "lemmings",
"lcname": "lemmings-hpc"
}
JSON