| Name | ictasks JSON |
| Version |
0.1.0
JSON |
| download |
| home_page | None |
| Summary | A utility to run multiple small tasks in a single HPC job submission. |
| upload_time | 2024-10-08 09:50:45 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.8 |
| license | None |
| keywords |
slurm
hpc
tasks
mpi
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# `ictasks`
This is a set of utilities to support running groups of small tasks as part of a single HPC batch submission at ICHEC.
The repo includes:
1) the `ictasks` package with tooling building blocks
2) Snapshots of the ICHEC [Taskfarm](https://www.ichec.ie/academic/national-hpc/documentation/tutorials/task-farming) tool available on our HPC systems in the `applications` directory.
# Installing #
The package can be installed from PyPI:
``` shell
pip install ictasks
```
# Features #
## Taskfarm ##
The `taskfarm` feature will launch the tasks in the pointed to `tasklist` file:
``` shell
ictasks taskfarm --tasklist $PATH_TO_TASKLIST
```
See the `test/data/tasklist.dat` file for an example input with two small tasks.
If you run this on your local machine it will create a directory per task, launch the task in that directory and output a status file `task.json`. By default all processors on the machine (or compute node) will be assigned tasks. There are two ways to control this, and many other, settings:
1. With environment variables using a `TASKFARM_` prefix, consistent with the original ICHEC `taskfarm` tool and documented here: https://www.ichec.ie/academic/national-hpc/documentation/tutorials/task-farming
2. With a config file in `yaml` format - passed in with a `--config` command line argument. Config values take precedence over environment variables if both are specified for the same setting.
### Using a config file ###
The config file below shows an example constraining tasks to run on only a single core per compute node. We can also specify tasks in the config instead of in a tasklist file.
``` yaml
environment:
job_id: my_job
workers:
cores_per_node: 1
tasks:
items:
- id: 0
launch_cmd: "echo 'hello from task 0'"
- id: 1
launch_cmd: "echo 'hello from task 1'"
```
we can run this with:
``` shell
ictasks taskfarm --config my_config.yaml
```
# License #
This package is Coypright of the Irish Centre for High End Computing. It can be used under the terms of the GNU Public License (GPL). See the included `LICENSE.txt` file for details.
Raw data
{
"_id": null,
"home_page": null,
"name": "ictasks",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "Slurm, HPC, tasks, MPI",
"author": null,
"author_email": "\"James Grogan, Irish Centre for High End Computing\" <james.grogan@ichec.ie>",
"download_url": "https://files.pythonhosted.org/packages/60/36/fdcf1df747f936be201ea1f85aef7afff3b35d69ee56bae35aa1a235d55d/ictasks-0.1.0.tar.gz",
"platform": null,
"description": "# `ictasks`\n\nThis is a set of utilities to support running groups of small tasks as part of a single HPC batch submission at ICHEC.\n\nThe repo includes:\n\n1) the `ictasks` package with tooling building blocks\n2) Snapshots of the ICHEC [Taskfarm](https://www.ichec.ie/academic/national-hpc/documentation/tutorials/task-farming) tool available on our HPC systems in the `applications` directory.\n\n# Installing #\n\nThe package can be installed from PyPI:\n\n``` shell\npip install ictasks\n```\n\n# Features #\n\n## Taskfarm ##\n\nThe `taskfarm` feature will launch the tasks in the pointed to `tasklist` file:\n\n``` shell\nictasks taskfarm --tasklist $PATH_TO_TASKLIST\n```\n\nSee the `test/data/tasklist.dat` file for an example input with two small tasks.\n\nIf you run this on your local machine it will create a directory per task, launch the task in that directory and output a status file `task.json`. By default all processors on the machine (or compute node) will be assigned tasks. There are two ways to control this, and many other, settings:\n\n1. With environment variables using a `TASKFARM_` prefix, consistent with the original ICHEC `taskfarm` tool and documented here: https://www.ichec.ie/academic/national-hpc/documentation/tutorials/task-farming\n\n2. With a config file in `yaml` format - passed in with a `--config` command line argument. Config values take precedence over environment variables if both are specified for the same setting.\n\n### Using a config file ###\n\nThe config file below shows an example constraining tasks to run on only a single core per compute node. We can also specify tasks in the config instead of in a tasklist file.\n\n\n``` yaml\nenvironment:\n\tjob_id: my_job\n\tworkers:\n\t\tcores_per_node: 1\n\ntasks:\n\titems:\n\t - id: 0\n\t launch_cmd: \"echo 'hello from task 0'\"\n\t - id: 1\n\t launch_cmd: \"echo 'hello from task 1'\"\n```\n\nwe can run this with:\n\n``` shell\nictasks taskfarm --config my_config.yaml\n```\n\n# License #\n\nThis package is Coypright of the Irish Centre for High End Computing. It can be used under the terms of the GNU Public License (GPL). See the included `LICENSE.txt` file for details.\n\n\n\n\n\n",
"bugtrack_url": null,
"license": null,
"summary": "A utility to run multiple small tasks in a single HPC job submission.",
"version": "0.1.0",
"project_urls": {
"Homepage": "https://git.ichec.ie/performance/toolshed/ictasks",
"Repository": "https://git.ichec.ie/performance/toolshed/ictasks"
},
"split_keywords": [
"slurm",
" hpc",
" tasks",
" mpi"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "90a80d1184e9d79a915832f601b03a41162c1a9ebd29d5694c5a2a979700863e",
"md5": "a94262d4fa84fe710b0262265842f30a",
"sha256": "ff06f8d29305714e74c63f31c82a7df7031740aced3ba64ad1b4bfac51136138"
},
"downloads": -1,
"filename": "ictasks-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a94262d4fa84fe710b0262265842f30a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 27619,
"upload_time": "2024-10-08T09:50:43",
"upload_time_iso_8601": "2024-10-08T09:50:43.876111Z",
"url": "https://files.pythonhosted.org/packages/90/a8/0d1184e9d79a915832f601b03a41162c1a9ebd29d5694c5a2a979700863e/ictasks-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6036fdcf1df747f936be201ea1f85aef7afff3b35d69ee56bae35aa1a235d55d",
"md5": "132b532bca27d2fe4f2abf06b9070d30",
"sha256": "95426446573ee24be38bd90bdb37d10ed7159e786f159f124af72fc91ce70838"
},
"downloads": -1,
"filename": "ictasks-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "132b532bca27d2fe4f2abf06b9070d30",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 24833,
"upload_time": "2024-10-08T09:50:45",
"upload_time_iso_8601": "2024-10-08T09:50:45.951846Z",
"url": "https://files.pythonhosted.org/packages/60/36/fdcf1df747f936be201ea1f85aef7afff3b35d69ee56bae35aa1a235d55d/ictasks-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-08 09:50:45",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "ictasks"
}
