Jupyter slurm provisioner
=========================
Installation
------------
``pip install slurm-provisioner``
Overview
--------
The slurm provisioner is a subclass of
```jupyter_client.KernelProvisionerBase`` <https://github.com/jupyter/jupyter_client/blob/v7.4.2/jupyter_client/provisioning/provisioner_base.py#L24>`__.
Its area of use is any slurm-based HPC system. However, it was written
to be used within the Juelich Supercomputing Centre, which uses a script
called ``jutil`` to create a user-specific environment based on a
project. You may have to update ``scripts/slurmel_allocate`` to use it
on any other HPC system. It allows users to start their jupyter kernel
on any partition, while their notebook server is running on an
interactive login node. This offers a separation between code execution
and user interface. The slurm provisioner will not use its ipykernel
argv configuration but is meant to be a wrapper for any existing kernel.
It is recommended to use this kernel with the option
```--KernelRestarter.restart_limit=0`` <https://github.com/jupyter/jupyter_client/blob/v7.4.2/jupyter_client/restarter.py#L43>`__
to avoid unsought compute time and costs.
Configuration
-------------
Configure a kernel.json file in your ``$HOME``. It is not recommended to
define the kernel globally, since the configuration is user-specific.
Options
~~~~~~~
- project [required]: Used for correct budgeting of compute time.
- partition [required]: specify the slurm partition
- nodes [required]: specify the number of nodes
- runtime [required]: specify the runtime in minutes
- kernel_argv [required]: copy & paste this from the kernel you want to
run
- gpus [optional]: specify the number of GPUs, if the partition
supports GPUs
- reservation [optional]: specify the slurm reservation, if you have
one
- jobid [optional]: specify a pre-existing slurm allocation and start
your kernel there. Without this, a new allocation for each kernel
will be created.
- node [optional]: specify a node in your pre-existing allocation
Example ``kernel.json``
~~~~~~~~~~~~~~~~~~~~~~~
``.local/share/jupyter/kernels/slurmel/kernel.json``
::
{
"display_name": "Slurm Kernel",
"language": "python",
"metadata": {
"debugger": true,
"kernel_provisioner": {
"config": {
"gpus": "0",
"nodes": "2",
"partition": "batch",
"project": "...",
"reservation": "None",
"runtime": 30,
"jobid": "None",
"node": "None",
"kernel_argv": [
"~/.local/share/jupyter/kernels/my_kernel/kernel.sh",
"-f",
"{connection_file}"
]
},
"provisioner_name": "slurm-provisioner"
}
}
}
Raw data
{
"_id": null,
"home_page": "https://github.com/FZJ-JSC/jupyter-slurm-provisioner",
"name": "jupyter-slurm-provisioner",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "",
"author": "Tim Kreuzer",
"author_email": "t.kreuzer@fz-juelich.de",
"download_url": "https://files.pythonhosted.org/packages/1a/f7/1336205f53e79ffb5e7acbac42be2656cb2fe43fe750b561a28150213254/jupyter-slurm-provisioner-0.6.0.tar.gz",
"platform": null,
"description": "Jupyter slurm provisioner\n=========================\n\nInstallation\n------------\n\n``pip install slurm-provisioner``\n\nOverview\n--------\n\nThe slurm provisioner is a subclass of\n```jupyter_client.KernelProvisionerBase`` <https://github.com/jupyter/jupyter_client/blob/v7.4.2/jupyter_client/provisioning/provisioner_base.py#L24>`__.\nIts area of use is any slurm-based HPC system. However, it was written\nto be used within the Juelich Supercomputing Centre, which uses a script\ncalled ``jutil`` to create a user-specific environment based on a\nproject. You may have to update ``scripts/slurmel_allocate`` to use it\non any other HPC system. It allows users to start their jupyter kernel\non any partition, while their notebook server is running on an\ninteractive login node. This offers a separation between code execution\nand user interface. The slurm provisioner will not use its ipykernel\nargv configuration but is meant to be a wrapper for any existing kernel.\nIt is recommended to use this kernel with the option\n```--KernelRestarter.restart_limit=0`` <https://github.com/jupyter/jupyter_client/blob/v7.4.2/jupyter_client/restarter.py#L43>`__\nto avoid unsought compute time and costs.\n\nConfiguration\n-------------\n\nConfigure a kernel.json file in your ``$HOME``. It is not recommended to\ndefine the kernel globally, since the configuration is user-specific.\n\nOptions\n~~~~~~~\n\n- project [required]: Used for correct budgeting of compute time.\n- partition [required]: specify the slurm partition\n- nodes [required]: specify the number of nodes\n- runtime [required]: specify the runtime in minutes\n- kernel_argv [required]: copy & paste this from the kernel you want to\n run\n- gpus [optional]: specify the number of GPUs, if the partition\n supports GPUs\n- reservation [optional]: specify the slurm reservation, if you have\n one\n- jobid [optional]: specify a pre-existing slurm allocation and start\n your kernel there. Without this, a new allocation for each kernel\n will be created.\n- node [optional]: specify a node in your pre-existing allocation\n\nExample ``kernel.json``\n~~~~~~~~~~~~~~~~~~~~~~~\n\n``.local/share/jupyter/kernels/slurmel/kernel.json``\n\n::\n\n {\n \"display_name\": \"Slurm Kernel\",\n \"language\": \"python\",\n \"metadata\": {\n \"debugger\": true,\n \"kernel_provisioner\": {\n \"config\": {\n \"gpus\": \"0\",\n \"nodes\": \"2\",\n \"partition\": \"batch\",\n \"project\": \"...\",\n \"reservation\": \"None\",\n \"runtime\": 30,\n \"jobid\": \"None\",\n \"node\": \"None\",\n \"kernel_argv\": [\n \"~/.local/share/jupyter/kernels/my_kernel/kernel.sh\",\n \"-f\",\n \"{connection_file}\"\n ]\n },\n \"provisioner_name\": \"slurm-provisioner\"\n }\n }\n }\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Jupyter slurm kernel provisioner",
"version": "0.6.0",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "58333444e9a83a2744eb9b7b65a06969d688e1e48ef3823b0988d3de048e3af1",
"md5": "7bfe27f8372086160a1d5d08b6971363",
"sha256": "73d454aa68cbb5e9a018bab5e2b3460901b6e78e47a32fb695c44ab6fd3d6f98"
},
"downloads": -1,
"filename": "jupyter_slurm_provisioner-0.6.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "7bfe27f8372086160a1d5d08b6971363",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 11793,
"upload_time": "2023-01-27T08:33:08",
"upload_time_iso_8601": "2023-01-27T08:33:08.517786Z",
"url": "https://files.pythonhosted.org/packages/58/33/3444e9a83a2744eb9b7b65a06969d688e1e48ef3823b0988d3de048e3af1/jupyter_slurm_provisioner-0.6.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1af71336205f53e79ffb5e7acbac42be2656cb2fe43fe750b561a28150213254",
"md5": "e69fbefe756b0282832a77e00a350681",
"sha256": "24141dc4abfb009d424837a5d78159c2b63e0243d63618cacebcf4f23edc101f"
},
"downloads": -1,
"filename": "jupyter-slurm-provisioner-0.6.0.tar.gz",
"has_sig": false,
"md5_digest": "e69fbefe756b0282832a77e00a350681",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 11722,
"upload_time": "2023-01-27T08:33:11",
"upload_time_iso_8601": "2023-01-27T08:33:11.070128Z",
"url": "https://files.pythonhosted.org/packages/1a/f7/1336205f53e79ffb5e7acbac42be2656cb2fe43fe750b561a28150213254/jupyter-slurm-provisioner-0.6.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-01-27 08:33:11",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "FZJ-JSC",
"github_project": "jupyter-slurm-provisioner",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "jupyter-slurm-provisioner"
}
JSON
