| Name | nshrunner JSON |
| Version |
0.17.0
JSON |
| download |
| home_page | None |
| Summary | None |
| upload_time | 2024-08-21 04:08:33 |
| maintainer | None |
| docs_url | None |
| author | Nima Shoghi |
| requires_python | <4.0,>=3.10 |
| license | None |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# nshrunner
nshrunner is a Python library that provides a unified way to run functions in various environments, such as local dev machines, cloud VMs, SLURM clusters, and LSF clusters. It was created to simplify the process of running ML training jobs across multiple machines and environments.
## Motivation
When running ML training jobs on different machines and environments, it can be challenging to manage the specifics of each environment. nshrunner was developed to address this issue by providing a single function that can be used to run jobs on any supported environment without having to worry about the details of each environment.
## Features
- Supports running functions locally, on SLURM clusters, and on LSF clusters
- Provides a unified interface for running functions across different environments
- Allows for easy configuration of job options, such as resource requirements and environment variables
- Supports snapshotting the environment to ensure reproducibility, using the [`nshsnap`](https://www.github.com/nimashoghi/nshsnap) library
- Provides utilities for logging, seeding, and signal handling
## Installation
nshrunner can be installed using pip:
```bash
pip install nshrunner
```
## Usage
Here's a simple example of how to use nshrunner to run a function locally:
```python
import nshrunner as R
def run_fn(x: int):
return x + 5
runs = [(1,)]
runner = R.Runner(run_fn)
list(runner.local(runs))
```
To run the same function on a SLURM cluster:
```python
runner.submit_slurm(
runs,
{
"partition": "learnaccel",
"nodes": 4,
"ntasks_per_node": 8, # Change this to limit # of GPUs
"gpus_per_task": 1,
"cpus_per_task": 1,
},
snapshot=True,
)
```
And on an LSF cluster:
```python
runner.submit_lsf(
runs,
{
"summit": True,
"queue": "learnaccel",
"nodes": 4,
"rs_per_node": 8, # Change this to limit # of GPUs
},
snapshot=True,
)
```
For more detailed usage examples, please refer to the documentation.
## Acknowledgements
`nshrunner` is heavily inspired by [`submitit`](https://github.com/facebookincubator/submitit). It builds on `submitit`'s design and adds support for LSF clusters, snapshotting, and other features.
## Contributing
Contributions are welcome! For feature requests, bug reports, or questions, please open an issue on GitHub. If you'd like to contribute code, please submit a pull request with your changes.
## License
nshrunner is released under the MIT License. See `LICENSE` for more information.
Raw data
{
"_id": null,
"home_page": null,
"name": "nshrunner",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.10",
"maintainer_email": null,
"keywords": null,
"author": "Nima Shoghi",
"author_email": "nimashoghi@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/a4/2e/399b17a24b7c0a833aebf6f8f8ad197134ddaa3860bf8b9cfda97132ef1e/nshrunner-0.17.0.tar.gz",
"platform": null,
"description": "# nshrunner\n\nnshrunner is a Python library that provides a unified way to run functions in various environments, such as local dev machines, cloud VMs, SLURM clusters, and LSF clusters. It was created to simplify the process of running ML training jobs across multiple machines and environments.\n\n## Motivation\n\nWhen running ML training jobs on different machines and environments, it can be challenging to manage the specifics of each environment. nshrunner was developed to address this issue by providing a single function that can be used to run jobs on any supported environment without having to worry about the details of each environment.\n\n## Features\n\n- Supports running functions locally, on SLURM clusters, and on LSF clusters\n- Provides a unified interface for running functions across different environments\n- Allows for easy configuration of job options, such as resource requirements and environment variables\n- Supports snapshotting the environment to ensure reproducibility, using the [`nshsnap`](https://www.github.com/nimashoghi/nshsnap) library\n- Provides utilities for logging, seeding, and signal handling\n\n## Installation\n\nnshrunner can be installed using pip:\n\n```bash\npip install nshrunner\n```\n\n## Usage\n\nHere's a simple example of how to use nshrunner to run a function locally:\n\n```python\nimport nshrunner as R\n\ndef run_fn(x: int):\n return x + 5\n\nruns = [(1,)]\n\nrunner = R.Runner(run_fn)\nlist(runner.local(runs))\n```\n\nTo run the same function on a SLURM cluster:\n\n```python\nrunner.submit_slurm(\n runs,\n {\n \"partition\": \"learnaccel\",\n \"nodes\": 4,\n \"ntasks_per_node\": 8, # Change this to limit # of GPUs\n \"gpus_per_task\": 1,\n \"cpus_per_task\": 1,\n },\n snapshot=True,\n)\n```\n\nAnd on an LSF cluster:\n\n```python\nrunner.submit_lsf(\n runs,\n {\n \"summit\": True,\n \"queue\": \"learnaccel\",\n \"nodes\": 4,\n \"rs_per_node\": 8, # Change this to limit # of GPUs\n },\n snapshot=True,\n)\n```\n\nFor more detailed usage examples, please refer to the documentation.\n\n## Acknowledgements\n\n`nshrunner` is heavily inspired by [`submitit`](https://github.com/facebookincubator/submitit). It builds on `submitit`'s design and adds support for LSF clusters, snapshotting, and other features.\n\n## Contributing\n\nContributions are welcome! For feature requests, bug reports, or questions, please open an issue on GitHub. If you'd like to contribute code, please submit a pull request with your changes.\n\n## License\n\nnshrunner is released under the MIT License. See `LICENSE` for more information.\n",
"bugtrack_url": null,
"license": null,
"summary": null,
"version": "0.17.0",
"project_urls": {
"homepage": "https://github.com/nimashoghi/nshrunner"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "eb012ce53a0c59894eb4b15f017338bad047cedcf9825e89dc174211f6006c21",
"md5": "d1278aafba46fceff8c38d4315a074ed",
"sha256": "b81dc2dccddf1b4ec2c715974f8d0139ac25f01f111f31cc648fa0f8fa337a3c"
},
"downloads": -1,
"filename": "nshrunner-0.17.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d1278aafba46fceff8c38d4315a074ed",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.10",
"size": 35457,
"upload_time": "2024-08-21T04:08:32",
"upload_time_iso_8601": "2024-08-21T04:08:32.384048Z",
"url": "https://files.pythonhosted.org/packages/eb/01/2ce53a0c59894eb4b15f017338bad047cedcf9825e89dc174211f6006c21/nshrunner-0.17.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a42e399b17a24b7c0a833aebf6f8f8ad197134ddaa3860bf8b9cfda97132ef1e",
"md5": "a08813267fc15439f89e078eef930b3b",
"sha256": "b1bbaca3182c1687823b0ff4383a9dd208c2cf52eee7dfe3369a3c6e65efd7aa"
},
"downloads": -1,
"filename": "nshrunner-0.17.0.tar.gz",
"has_sig": false,
"md5_digest": "a08813267fc15439f89e078eef930b3b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.10",
"size": 26606,
"upload_time": "2024-08-21T04:08:33",
"upload_time_iso_8601": "2024-08-21T04:08:33.795335Z",
"url": "https://files.pythonhosted.org/packages/a4/2e/399b17a24b7c0a833aebf6f8f8ad197134ddaa3860bf8b9cfda97132ef1e/nshrunner-0.17.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-21 04:08:33",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "nimashoghi",
"github_project": "nshrunner",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "nshrunner"
}
