nshrunner


Namenshrunner JSON
Version 1.0.5 PyPI version JSON
download
home_pageNone
SummaryNone
upload_time2024-12-11 04:20:07
maintainerNone
docs_urlNone
authorNima Shoghi
requires_python<4.0,>=3.10
licenseNone
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # nshrunner

nshrunner is a Python library that provides a unified way to run functions in various environments, such as local dev machines, cloud VMs, and SLURM clusters. It was created to simplify the process of running ML training jobs across multiple machines and environments.

## Motivation

When running ML training jobs on different machines and environments, it can be challenging to manage the specifics of each environment. nshrunner was developed to address this issue by providing a single function that can be used to run jobs on any supported environment without having to worry about the details of each environment.

## Features

- Supports running functions locally, on SLURM clusters, and in GNU Screen sessions
- Provides a unified interface for running functions across different environments
- Allows for easy configuration of job options, such as resource requirements and environment variables
- Supports snapshotting the environment to ensure reproducibility, using the [`nshsnap`](https://www.github.com/nimashoghi/nshsnap) library
- Provides utilities for logging, seeding, and signal handling

## Installation

nshrunner can be installed using pip:

```bash
pip install nshrunner
```

## Usage

Here's a simple example showing the different ways to run a function:

```python
import nshrunner as R

def train_model(batch_size: int, learning_rate: float):
    # Training logic here
    return {"accuracy": 0.95}

# Define runs with different hyperparameters
runs = [
    (32, 0.001),  # (batch_size, learning_rate)
    (64, 0.0005),
]

# Run locally
results = R.run_local(train_model, runs)

# Run in a GNU Screen session
R.submit_screen(
    train_model,
    runs,
    screen={
        "name": "training",
        "logging": {
            "output_file": "logs/output.log",
            "error_file": "logs/error.log"
        },
        "attach": False  # Run detached
    }
)

# Run on SLURM
R.submit_slurm(
    train_model,
    runs,
    slurm={
        "name": "training",
        "partition": "gpu",
        "resources": {
            "nodes": 1,
            "cpus": 4,
            "gpus": 1,
            "memory_gb": 32,
            "time": "12:00:00"
        },
        "output_dir": "logs"
    }
)
```

The library provides a consistent interface across different execution environments while handling the complexities of:

- Job submission and management
- Resource allocation
- Environment setup
- Output logging
- Error handling

For more advanced usage, you can configure additional options like:

```python
# Configure environment snapshot for reproducibility
R.submit_slurm(
    train_model,
    runs,
    runner={
        "working_dir": "experiments",
        "snapshot": True,  # Snapshot code and dependencies
        "seed": {"seed": 42}  # Set random seeds
    },
    slurm={...}
)
```

## Contributing

Contributions are welcome! For feature requests, bug reports, or questions, please open an issue on GitHub. If you'd like to contribute code, please submit a pull request with your changes.

## License

nshrunner is released under the MIT License. See `LICENSE` for more information.

            

Raw data

            {
    "_id": null,
    "home_page": null,
    "name": "nshrunner",
    "maintainer": null,
    "docs_url": null,
    "requires_python": "<4.0,>=3.10",
    "maintainer_email": null,
    "keywords": null,
    "author": "Nima Shoghi",
    "author_email": "nimashoghi@gmail.com",
    "download_url": "https://files.pythonhosted.org/packages/32/8f/58abd4341a985ad2608c9d442eda13dce3e7c7aff9080ac64a009216e69d/nshrunner-1.0.5.tar.gz",
    "platform": null,
    "description": "# nshrunner\n\nnshrunner is a Python library that provides a unified way to run functions in various environments, such as local dev machines, cloud VMs, and SLURM clusters. It was created to simplify the process of running ML training jobs across multiple machines and environments.\n\n## Motivation\n\nWhen running ML training jobs on different machines and environments, it can be challenging to manage the specifics of each environment. nshrunner was developed to address this issue by providing a single function that can be used to run jobs on any supported environment without having to worry about the details of each environment.\n\n## Features\n\n- Supports running functions locally, on SLURM clusters, and in GNU Screen sessions\n- Provides a unified interface for running functions across different environments\n- Allows for easy configuration of job options, such as resource requirements and environment variables\n- Supports snapshotting the environment to ensure reproducibility, using the [`nshsnap`](https://www.github.com/nimashoghi/nshsnap) library\n- Provides utilities for logging, seeding, and signal handling\n\n## Installation\n\nnshrunner can be installed using pip:\n\n```bash\npip install nshrunner\n```\n\n## Usage\n\nHere's a simple example showing the different ways to run a function:\n\n```python\nimport nshrunner as R\n\ndef train_model(batch_size: int, learning_rate: float):\n    # Training logic here\n    return {\"accuracy\": 0.95}\n\n# Define runs with different hyperparameters\nruns = [\n    (32, 0.001),  # (batch_size, learning_rate)\n    (64, 0.0005),\n]\n\n# Run locally\nresults = R.run_local(train_model, runs)\n\n# Run in a GNU Screen session\nR.submit_screen(\n    train_model,\n    runs,\n    screen={\n        \"name\": \"training\",\n        \"logging\": {\n            \"output_file\": \"logs/output.log\",\n            \"error_file\": \"logs/error.log\"\n        },\n        \"attach\": False  # Run detached\n    }\n)\n\n# Run on SLURM\nR.submit_slurm(\n    train_model,\n    runs,\n    slurm={\n        \"name\": \"training\",\n        \"partition\": \"gpu\",\n        \"resources\": {\n            \"nodes\": 1,\n            \"cpus\": 4,\n            \"gpus\": 1,\n            \"memory_gb\": 32,\n            \"time\": \"12:00:00\"\n        },\n        \"output_dir\": \"logs\"\n    }\n)\n```\n\nThe library provides a consistent interface across different execution environments while handling the complexities of:\n\n- Job submission and management\n- Resource allocation\n- Environment setup\n- Output logging\n- Error handling\n\nFor more advanced usage, you can configure additional options like:\n\n```python\n# Configure environment snapshot for reproducibility\nR.submit_slurm(\n    train_model,\n    runs,\n    runner={\n        \"working_dir\": \"experiments\",\n        \"snapshot\": True,  # Snapshot code and dependencies\n        \"seed\": {\"seed\": 42}  # Set random seeds\n    },\n    slurm={...}\n)\n```\n\n## Contributing\n\nContributions are welcome! For feature requests, bug reports, or questions, please open an issue on GitHub. If you'd like to contribute code, please submit a pull request with your changes.\n\n## License\n\nnshrunner is released under the MIT License. See `LICENSE` for more information.\n",
    "bugtrack_url": null,
    "license": null,
    "summary": null,
    "version": "1.0.5",
    "project_urls": {
        "homepage": "https://github.com/nimashoghi/nshrunner"
    },
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "0275caf3c405343ffb444e10ea8ab4439b1337ed6417190b5f616f0ed8ad38cb",
                "md5": "09a93784af5435d8cd133b0eb804b1d5",
                "sha256": "3897ba6be91d17032dd009bc46d81ae943e814f60abaf5ca2fdbe67891838fc3"
            },
            "downloads": -1,
            "filename": "nshrunner-1.0.5-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "09a93784af5435d8cd133b0eb804b1d5",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": "<4.0,>=3.10",
            "size": 43940,
            "upload_time": "2024-12-11T04:20:05",
            "upload_time_iso_8601": "2024-12-11T04:20:05.293183Z",
            "url": "https://files.pythonhosted.org/packages/02/75/caf3c405343ffb444e10ea8ab4439b1337ed6417190b5f616f0ed8ad38cb/nshrunner-1.0.5-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "328f58abd4341a985ad2608c9d442eda13dce3e7c7aff9080ac64a009216e69d",
                "md5": "cf4b08a301b8bb2189a26e204c33adeb",
                "sha256": "59b9533c1f70278bed2fd647ee34dcecfeaaf9b0da1927c9d32a59467437f270"
            },
            "downloads": -1,
            "filename": "nshrunner-1.0.5.tar.gz",
            "has_sig": false,
            "md5_digest": "cf4b08a301b8bb2189a26e204c33adeb",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": "<4.0,>=3.10",
            "size": 29373,
            "upload_time": "2024-12-11T04:20:07",
            "upload_time_iso_8601": "2024-12-11T04:20:07.735774Z",
            "url": "https://files.pythonhosted.org/packages/32/8f/58abd4341a985ad2608c9d442eda13dce3e7c7aff9080ac64a009216e69d/nshrunner-1.0.5.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-12-11 04:20:07",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "nimashoghi",
    "github_project": "nshrunner",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": false,
    "lcname": "nshrunner"
}
        
Elapsed time: 0.41144s