| Name | ictasks JSON |
| Version |
0.1.3
JSON |
| download |
| home_page | None |
| Summary | A utility to run multiple small tasks in a single HPC job submission. |
| upload_time | 2024-12-11 16:15:46 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.8 |
| license | None |
| keywords |
slurm
hpc
tasks
mpi
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# ictasks
`ictasks` is a library to support running collections of small tasks in parallel, from a laptop through to a HPC cluster.
It is being developed to support projects and workflow standarisation at the Irish Centre for High End Computing (ICHEC).
There are many libraries like this, this one focuses on:
* being small and simple - easy for ICHEC researchers to understand all elements of it
* having low dependencies - can be installed with `pip`
* having an easy to use configuration format with `yaml`
* having verbose and structured reporting of task related metadata for analysis
# Features #
The library allows a few different ways to specify tasks.
1) As a Python package
You can create a collection of `Task` instances and launch them with the `session.run()` function.
2) Using a `tasklist.dat` file and the command line interface (CLI):
``` shell
ictasks taskfarm --tasklist $PATH_TO_TASKLIST
```
See the `test/data/tasklist.dat` file for an example input with two small tasks.
3) Using a `yaml` config file, giving task and environment info.
``` yaml
environment:
job_id: my_job
workers:
cores_per_node: 1
tasks:
items:
- id: 0
launch_cmd: "echo 'hello from task 0'"
- id: 1
launch_cmd: "echo 'hello from task 1'"
```
we can run this with:
``` shell
ictasks taskfarm --config my_config.yaml
```
Launching the run will launch the task in that directory and output a status file `task.json`. By default all processors on the machine (or compute node) will be assigned tasks.
# Installing #
The package can be installed from PyPI:
``` shell
pip install ictasks
```
# Contact #
Our Gitlab hosting arrangement doesn't allow us to easily accept external contributions or feedback, however they are still welcome.
In future if there is interest
If you have any feedback on this library
# License #
This package is Coypright of the Irish Centre for High End Computing. It can be used under the terms of the GNU Public License (GPL v3+). See the included `LICENSE.txt` file for details.
If you are an ICHEC collaborator or user of the National Service different licensing terms can be offered - please get in touch to discuss.
Raw data
{
"_id": null,
"home_page": null,
"name": "ictasks",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "Slurm, HPC, tasks, MPI",
"author": null,
"author_email": "\"James Grogan, Irish Centre for High End Computing\" <james.grogan@ichec.ie>",
"download_url": "https://files.pythonhosted.org/packages/13/dd/14f2f0188ab039bf095b51aaf2acc87fc76d8b1a2cb9b2568e90ccfc7356/ictasks-0.1.3.tar.gz",
"platform": null,
"description": "# ictasks\n\n`ictasks` is a library to support running collections of small tasks in parallel, from a laptop through to a HPC cluster.\n\nIt is being developed to support projects and workflow standarisation at the Irish Centre for High End Computing (ICHEC).\n\nThere are many libraries like this, this one focuses on:\n\n* being small and simple - easy for ICHEC researchers to understand all elements of it\n* having low dependencies - can be installed with `pip`\n* having an easy to use configuration format with `yaml`\n* having verbose and structured reporting of task related metadata for analysis\n\n# Features #\n\nThe library allows a few different ways to specify tasks.\n\n1) As a Python package\n\nYou can create a collection of `Task` instances and launch them with the `session.run()` function. \n\n2) Using a `tasklist.dat` file and the command line interface (CLI):\n\n``` shell\nictasks taskfarm --tasklist $PATH_TO_TASKLIST\n```\n\nSee the `test/data/tasklist.dat` file for an example input with two small tasks.\n\n3) Using a `yaml` config file, giving task and environment info.\n\n``` yaml\nenvironment:\n job_id: my_job\n workers:\n cores_per_node: 1\n\ntasks:\n items:\n - id: 0\n launch_cmd: \"echo 'hello from task 0'\"\n - id: 1\n launch_cmd: \"echo 'hello from task 1'\"\n```\n\nwe can run this with:\n\n``` shell\nictasks taskfarm --config my_config.yaml\n```\n\nLaunching the run will launch the task in that directory and output a status file `task.json`. By default all processors on the machine (or compute node) will be assigned tasks.\n\n\n# Installing #\n\nThe package can be installed from PyPI:\n\n``` shell\npip install ictasks\n```\n\n# Contact #\n\nOur Gitlab hosting arrangement doesn't allow us to easily accept external contributions or feedback, however they are still welcome.\n\nIn future if there is interest \n\n\nIf you have any feedback on this library \n\n# License #\n\nThis package is Coypright of the Irish Centre for High End Computing. It can be used under the terms of the GNU Public License (GPL v3+). See the included `LICENSE.txt` file for details.\n\nIf you are an ICHEC collaborator or user of the National Service different licensing terms can be offered - please get in touch to discuss.\n\n\n\n\n\n",
"bugtrack_url": null,
"license": null,
"summary": "A utility to run multiple small tasks in a single HPC job submission.",
"version": "0.1.3",
"project_urls": {
"Homepage": "https://git.ichec.ie/performance/toolshed/ictasks",
"Repository": "https://git.ichec.ie/performance/toolshed/ictasks"
},
"split_keywords": [
"slurm",
" hpc",
" tasks",
" mpi"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "1102c363b04f902bcb2d0bd338e55f442bfd1f7a484167a50b5df72891655c3a",
"md5": "9bbfbcfa51324495858903669bf236e4",
"sha256": "5c207863448a77be3b008e748852be4eaa3e5aca61e6427e634a92071f32dbe0"
},
"downloads": -1,
"filename": "ictasks-0.1.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "9bbfbcfa51324495858903669bf236e4",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 29091,
"upload_time": "2024-12-11T16:15:44",
"upload_time_iso_8601": "2024-12-11T16:15:44.202017Z",
"url": "https://files.pythonhosted.org/packages/11/02/c363b04f902bcb2d0bd338e55f442bfd1f7a484167a50b5df72891655c3a/ictasks-0.1.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "13dd14f2f0188ab039bf095b51aaf2acc87fc76d8b1a2cb9b2568e90ccfc7356",
"md5": "cbe43749d6cbb6c0b6fffc3143292540",
"sha256": "b3c1c17f9897a38c7a14dc316d03ad0cf42a55f629b3333d6ce806047d8e0cab"
},
"downloads": -1,
"filename": "ictasks-0.1.3.tar.gz",
"has_sig": false,
"md5_digest": "cbe43749d6cbb6c0b6fffc3143292540",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 26073,
"upload_time": "2024-12-11T16:15:46",
"upload_time_iso_8601": "2024-12-11T16:15:46.234425Z",
"url": "https://files.pythonhosted.org/packages/13/dd/14f2f0188ab039bf095b51aaf2acc87fc76d8b1a2cb9b2568e90ccfc7356/ictasks-0.1.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-11 16:15:46",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "ictasks"
}
