jz-hydra-submitit-launcher


Namejz-hydra-submitit-launcher JSON
Version 0.1.5 PyPI version JSON
download
home_pagehttps://github.com/zaccharieramzi/jz-hydra-submitit-launcher
SummaryJean Zay tailored Hydra submitit launcher.
upload_time2023-04-13 09:31:16
maintainer
docs_urlNone
authorZaccharie Ramzi
requires_python>=3.6
licenseMIT
keywords hydra submitit jean-zay
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # JZ-hydra-submitit-launcher

![GitHub Workflow Build Status](https://github.com/zaccharieramzi/jz-hydra-submitit-launcher/workflows/Continuous%20testing/badge.svg)

A Jean Zay (JZ) tailored Hydra submitit launcher based on [Hydra](https://hydra.cc/docs/intro/) and its [submitit-launcher plugin](https://hydra.cc/docs/plugins/submitit_launcher/).
Basically it extends the submitit-launcher plugin with defaults that make sense for JZ.

## Install

This package can be installed from pypi (using `--user` is required on JZ):
```
pip install --user jz-hydra-submitit-launcher
```

You can also install it from source:
```
git clone https://github.com/zaccharieramzi/jz-hydra-submitit-launcher.git
cd jz-hydra-submitit-launcher
pip install --user .
```

The main command is then installed in your local binaries.
You need to add them to your path in order for the command to be found (and avoid the error `-bash: hydra-submitit-launch: command not found`):

```
echo "export PATH=$PATH:$HOME/.local/bin" >> $HOME/.bashrc
```

Don't forget to `source $HOME/.bashrc` before using the command.

## Use

The primary use is with the `hydra-submitit-launch` command with your script name and the config type:
```
hydra-submitit-launch my_app.py dev
```

### Available configs
6 different configs are available:
- `dev`: with 2 hours, 1 gpu, and qos_gpu-dev.
- `t3`: with 20 hours, 1 gpu, and qos_gpu-t3.
- `t4`: with 100 hours, 1 gpu, and qos_gpu-t4.
- `4gpus_dev`: with 2 hours, 4 gpus, and qos_gpu-dev.
- `4gpus_t3`: with 20 hours, 4 gpus, and qos_gpu-t3.
- `4gpus_t4`: with 100 hours, 4 gpus, and qos_gpu-t4.

By default, all the configs select 32Gb GPUs, use a single node and use the gpu_p1 partition.
### Advanced configs
You can override the SLURM config, the same way you would with any hydra configuration.
The parameters you can override are defined in the `hydra-submitit-launcher` plugin [doc](https://hydra.cc/docs/plugins/submitit_launcher/#usage).

For example, if you want to use the gpu_p2 partition, you would need to do:
```
hydra-submitit-launch my_app.py dev hydra.launcher.setup=null hydra.launcher.partition=gpu_p2 hydra.launcher.cpus_per_task=3
```

In order to change the timeout on the SLURM job to for example 10 hours, you would need to do:
```
hydra-submitit-launch my_app.py base +hours=10
```
This will automatically select the right qos for you (and if you want to force a certain qos, you can always add `hydra.launcher.qos=<your-qos>`).

### Logs
Your SLURM logs are stored in the following path: `launch-dir/multirun/day-of-launch/time-of-launch/.submitit/job_array_task_id/`
This is the default from `hydra` and `submitit` and this can be changed using both the `submitit` plugin parameters (see [here](https://hydra.cc/docs/plugins/submitit_launcher/#usage)) and the `hydra` job configurations (see [here](https://hydra.cc/docs/configure_hydra/job/)). 

## References
- Hydra: https://hydra.cc/docs/intro/
- submitit-launcher: https://hydra.cc/docs/plugins/submitit_launcher/
- submitit: https://github.com/facebookincubator/submitit
- JZ docs: http://www.idris.fr/ or https://jean-zay-doc.readthedocs.io/en/latest/

            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/zaccharieramzi/jz-hydra-submitit-launcher",
    "name": "jz-hydra-submitit-launcher",
    "maintainer": "",
    "docs_url": null,
    "requires_python": ">=3.6",
    "maintainer_email": "",
    "keywords": "hydra,submitit,jean-zay",
    "author": "Zaccharie Ramzi",
    "author_email": "zaccharie.ramzi@gmail.com",
    "download_url": "https://files.pythonhosted.org/packages/89/e8/73ba3299ffb0f989a58e9d5b8ef29097dc572cf49c50ebd7828a13549789/jz-hydra-submitit-launcher-0.1.5.tar.gz",
    "platform": null,
    "description": "# JZ-hydra-submitit-launcher\n\n![GitHub Workflow Build Status](https://github.com/zaccharieramzi/jz-hydra-submitit-launcher/workflows/Continuous%20testing/badge.svg)\n\nA Jean Zay (JZ) tailored Hydra submitit launcher based on [Hydra](https://hydra.cc/docs/intro/) and its [submitit-launcher plugin](https://hydra.cc/docs/plugins/submitit_launcher/).\nBasically it extends the submitit-launcher plugin with defaults that make sense for JZ.\n\n## Install\n\nThis package can be installed from pypi (using `--user` is required on JZ):\n```\npip install --user jz-hydra-submitit-launcher\n```\n\nYou can also install it from source:\n```\ngit clone https://github.com/zaccharieramzi/jz-hydra-submitit-launcher.git\ncd jz-hydra-submitit-launcher\npip install --user .\n```\n\nThe main command is then installed in your local binaries.\nYou need to add them to your path in order for the command to be found (and avoid the error `-bash: hydra-submitit-launch: command not found`):\n\n```\necho \"export PATH=$PATH:$HOME/.local/bin\" >> $HOME/.bashrc\n```\n\nDon't forget to `source $HOME/.bashrc` before using the command.\n\n## Use\n\nThe primary use is with the `hydra-submitit-launch` command with your script name and the config type:\n```\nhydra-submitit-launch my_app.py dev\n```\n\n### Available configs\n6 different configs are available:\n- `dev`: with 2 hours, 1 gpu, and qos_gpu-dev.\n- `t3`: with 20 hours, 1 gpu, and qos_gpu-t3.\n- `t4`: with 100 hours, 1 gpu, and qos_gpu-t4.\n- `4gpus_dev`: with 2 hours, 4 gpus, and qos_gpu-dev.\n- `4gpus_t3`: with 20 hours, 4 gpus, and qos_gpu-t3.\n- `4gpus_t4`: with 100 hours, 4 gpus, and qos_gpu-t4.\n\nBy default, all the configs select 32Gb GPUs, use a single node and use the gpu_p1 partition.\n### Advanced configs\nYou can override the SLURM config, the same way you would with any hydra configuration.\nThe parameters you can override are defined in the `hydra-submitit-launcher` plugin [doc](https://hydra.cc/docs/plugins/submitit_launcher/#usage).\n\nFor example, if you want to use the gpu_p2 partition, you would need to do:\n```\nhydra-submitit-launch my_app.py dev hydra.launcher.setup=null hydra.launcher.partition=gpu_p2 hydra.launcher.cpus_per_task=3\n```\n\nIn order to change the timeout on the SLURM job to for example 10 hours, you would need to do:\n```\nhydra-submitit-launch my_app.py base +hours=10\n```\nThis will automatically select the right qos for you (and if you want to force a certain qos, you can always add `hydra.launcher.qos=<your-qos>`).\n\n### Logs\nYour SLURM logs are stored in the following path: `launch-dir/multirun/day-of-launch/time-of-launch/.submitit/job_array_task_id/`\nThis is the default from `hydra` and `submitit` and this can be changed using both the `submitit` plugin parameters (see [here](https://hydra.cc/docs/plugins/submitit_launcher/#usage)) and the `hydra` job configurations (see [here](https://hydra.cc/docs/configure_hydra/job/)). \n\n## References\n- Hydra: https://hydra.cc/docs/intro/\n- submitit-launcher: https://hydra.cc/docs/plugins/submitit_launcher/\n- submitit: https://github.com/facebookincubator/submitit\n- JZ docs: http://www.idris.fr/ or https://jean-zay-doc.readthedocs.io/en/latest/\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "Jean Zay tailored Hydra submitit launcher.",
    "version": "0.1.5",
    "split_keywords": [
        "hydra",
        "submitit",
        "jean-zay"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "487143b8cdf6eefddaed5b9adbe955f190586d1f449a87065c18a1dc678c25a9",
                "md5": "b0f49b742654639520394f077a4ba93a",
                "sha256": "c6d2a7fcc20e84177b361bff4e5a6806a29366cf36fe62a1bfd2b0552270570a"
            },
            "downloads": -1,
            "filename": "jz_hydra_submitit_launcher-0.1.5-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "b0f49b742654639520394f077a4ba93a",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.6",
            "size": 8371,
            "upload_time": "2023-04-13T09:31:15",
            "upload_time_iso_8601": "2023-04-13T09:31:15.324966Z",
            "url": "https://files.pythonhosted.org/packages/48/71/43b8cdf6eefddaed5b9adbe955f190586d1f449a87065c18a1dc678c25a9/jz_hydra_submitit_launcher-0.1.5-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "89e873ba3299ffb0f989a58e9d5b8ef29097dc572cf49c50ebd7828a13549789",
                "md5": "cb0848180c25519b9feb542a10e50006",
                "sha256": "45153115a5f3e6410c13908b95e2daabd27040beacde5e31dde2476036dea643"
            },
            "downloads": -1,
            "filename": "jz-hydra-submitit-launcher-0.1.5.tar.gz",
            "has_sig": false,
            "md5_digest": "cb0848180c25519b9feb542a10e50006",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.6",
            "size": 6600,
            "upload_time": "2023-04-13T09:31:16",
            "upload_time_iso_8601": "2023-04-13T09:31:16.861542Z",
            "url": "https://files.pythonhosted.org/packages/89/e8/73ba3299ffb0f989a58e9d5b8ef29097dc572cf49c50ebd7828a13549789/jz-hydra-submitit-launcher-0.1.5.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2023-04-13 09:31:16",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "github_user": "zaccharieramzi",
    "github_project": "jz-hydra-submitit-launcher",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "requirements": [],
    "lcname": "jz-hydra-submitit-launcher"
}
        
Elapsed time: 0.06612s