slurmutils


Nameslurmutils JSON
Version 1.1.5 PyPI version JSON
download
home_pageNone
SummaryUtilities and APIs for interfacing with the Slurm workload manager.
upload_time2025-07-16 13:24:29
maintainerNone
docs_urlNone
authorNone
requires_python>=3.10
licenseNone
keywords hpc administration orchestration utility
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # slurmutils

![PyPI - Version](https://img.shields.io/pypi/v/slurmutils)
![PyPI - Downloads](https://img.shields.io/pypi/dm/slurmutils)
![GitHub License](https://img.shields.io/github/license/charmed-hpc/slurmutils)
[![Matrix](https://img.shields.io/matrix/ubuntu-hpc%3Amatrix.org?logo=matrix&label=ubuntu-hpc)](https://matrix.to/#/#hpc:ubuntu.com)

Utilities and APIs for interfacing with the Slurm workload manager.

slurmutils is a collection of various utilities that make it easier 
for you and your friends to interface with the Slurm workload manager, especially if you 
are orchestrating deployments of new and current Slurm clusters. Gone are the days of
seething over incomplete Jinja2 templates. Current utilities shipped in the 
slurmutils package include:

#### `from slurmutils import ...`

* `calculate_rs`: A function for calculating the ranges and strides of an iterable with
  unique elements. This function can be used to help convert arrays of node hostnames,
  device file ids, etc into a Slurm hostname specification.
* `acctgatherconfig`: An editor for _acct_gather.conf_ configuration files.
* `cgroupconfig`: An editor for _cgroup.conf_ configuration files.
* `gresconfig`: An editor for _gres.conf_ configuration files.
* `ociconfig`: An editor for _oci.conf_ configuration files.
* `slurmconfig`: An editor for _slurm.conf_ configuration files.
* `slurmdbdconfig`: An editor for _slurmdbd.conf_ configuration files.

For more information on how to use or contribute to slurmutils, 
check out the [Getting Started](#-getting-started) and [Development](#-development) 
sections below 👇

## ✨ Getting Started

### Installation

#### Option 1: Install from PyPI

```shell
$ python3 -m pip install slurmutils
```

#### Option 2: Install from source

```shell
$ pip install .
```

### Usage

#### `slurmutils`

The top-level provides access to some utilities that streamline common Slurm-related
operations such as calculating the ranges and strides for a Slurm hostname specification
or editing configuration files in-place. Here's some example operations you can perform 
with these utilities:

##### `calculate_rs`

###### Calculate a range and/or stride from a list of node hostnames

```python3
from os.path import commonprefix

from slurmutils import calculate_rs

nodes = ["juju-abc654-1", "juju-abc654-2", "juju-abc654-4"]
prefix = commonprefix(nodes)
nums = [int(n.partition(prefix)[2]) for n in nodes]
slurm_host_spec = prefix + calculate_rs(nums)  # "juju-abc654-[1-2,4]"
```

###### Calculate a device file range for Nvidia GPUs

```python3
from pathlib import Path

from slurmutils import calculate_rs

device_files = [file for file in Path("/dev").iterdir() if "nvidia" in file.name]
prefix = "/dev/nvidia"
nums = [int(n.partition(prefix)[2]) for n in device_files]
file_spec = prefix + calculate_rs(nums)  # "/dev/nvidia[0-4]"
```

##### `acctgatherconfig`

###### Edit a pre-existing _acct_gather.conf_ configuration file

```python
from slurmutils import acctgatherconfig

with acctgatherconfig.edit("/etc/slurm/acct_gather.conf") as config:
    config.profile_influxdb_database = "test_acct_gather_db"
    config.profile_influxdb_default = ["none"]
    config.profile_influxdb_host = "testhostname1"
    config.profile_influxdb_pass = "testpassword1"
    config.profile_influxdb_rt_policy = "testpolicy1"
    config.profile_influxdb_user = "testuser1"
    config.profile_influxdb_timeout = 20
```

##### `cgroupconfig`

###### Edit a pre-existing _cgroup.conf_ configuration file

```python
from slurmutils import cgroupconfig

with cgroupconfig.edit("/etc/slurm/cgroup.conf") as config:
    config.constrain_cores = True
    config.constrain_devices = True
    config.constrain_ram_space = True
    config.constrain_swap_space = True
```

##### `gresconfig`

###### Edit a pre-existing _gres.conf_ configuration file

```python
from slurmutils import Gres, GresList, gresconfig

with gresconfig.edit("/etc/slurm/gres.conf") as config:
    gres1 = Gres(
        name="gpu",
        type="epyc",
        file="/dev/amd4",
        cores=[0, 1],
    )
    gres2 = Gres(
        name="gpu",
        nodename="juju-abc654-[1-20]",
        type="epyc",
        file="/dev/amd[0-3]",
        count="12G",
    )
    config.auto_detect = "rsmi"
    config.gres["gpu"] = GresList(gres1, gres2)
```

##### `ociconfig`

###### Edit a pre-existing _oci.conf_ configuration file

```python
from slurmutils import ociconfig

with ociconfig.edit("/etc/slurm/oci.conf") as config:
    config.ignore_file_config_json = False
    config.env_exclude = "^(SLURM_CONF|SLURM_CONF_SERVER|SLURM_JWT)="
    config.create_env_file = "newline"
    config.std_io_debug = "debug"
    config.syslog_debug = "debug"
```

##### `slurmconfig`

###### Edit a pre-existing _slurm.conf_ configuration file

```python
from slurmutils import slurmconfig

with slurmconfig.edit("/etc/slurm/slurm.conf") as config:
    del config.inactive_limit
    config.max_job_count = 20000
    config.proctrack_type = "proctrack/linuxproc"
```

###### Add a new node to the _slurm.conf_ file

```python
from slurmutils import Node, slurmconfig

with slurmconfig.edit("/etc/slurm/slurm.conf") as config:
    node = Node(
        nodename="batch-[0-25]",
        nodeaddr="12.34.56.78",
        cpus=1,
        realmemory=1000,
        tmpdisk=10000,
    )
    config.nodes[node.node_name] = node
```

##### `slurmdbdconfig`

###### Edit a pre-existing _slurmdbd.conf_ configuration file

```python
from slurmutils import slurmdbdconfig

with slurmdbdconfig.edit("/etc/slurm/slurmdbd.conf") as config:
    config.archive_usage = True
    config.log_file = "/var/spool/slurmdbd.log"
    config.debug_flags = ["db_event", "db_job", "db_usage"]
    del config.auth_alt_types
    del config.auth_alt_parameters
```

## 🤔 What's next?

If you want to learn more about all the things you can do with slurmutils, 
here are some further resources for you to explore:

* [Open an issue](https://github.com/charmed-hpc/slurmutils/issues/new?title=ISSUE+TITLE&body=*Please+describe+your+issue*)
* [Ask a question on GitHub](https://github.com/orgs/charmed-hpc/discussions/categories/q-a)

## 🛠️ Development

The project uses [just](https://github.com/casey/just) and [uv](https://github.com/astral-sh/uv) 
for development, which provides some useful commands that will help you while hacking on slurmutils:

```shell
just fmt          # Apply formatting standards to code
just lint         # Check code against coding style standards
just typecheck    # Run static type checks
just unit         # Run unit tests
```

If you're interested in contributing your work to slurmutils, 
take a look at our [contributing guidelines](./CONTRIBUTING.md) for further details.

## 🤝 Project and community

slurmutils is a project of the [Ubuntu High-Performance Computing community](https://ubuntu.com/community/governance/teams/hpc).
Interested in contributing bug fixes, new editors, documentation, or feedback? Want to join the Ubuntu HPC community? You’ve come to the right place 🤩

Here’s some links to help you get started with joining the community:

* [Ubuntu Code of Conduct](https://ubuntu.com/community/ethos/code-of-conduct)
* [Contributing guidelines](./CONTRIBUTING.md)
* [Join the conversation on Matrix](https://matrix.to/#/#hpc:ubuntu.com)
* [Get the latest news on Discourse](https://discourse.ubuntu.com/c/hpc/151)
* [Ask and answer questions on GitHub](https://github.com/orgs/charmed-hpc/discussions/categories/q-a)

## 📋 License

slurmutils is free software, distributed under the GNU Lesser General Public License, v3.0.
See the [LGPL-3.0 LICENSE](./LICENSE) file for further details.

            

Raw data

            {
    "_id": null,
    "home_page": null,
    "name": "slurmutils",
    "maintainer": null,
    "docs_url": null,
    "requires_python": ">=3.10",
    "maintainer_email": "\"Jason C. Nucciarone\" <nuccitheboss@ubuntu.com>, Ubuntu High-Performance Computing <hpc-ubuntu-group@canonical.com>",
    "keywords": "HPC, administration, orchestration, utility",
    "author": null,
    "author_email": "\"Jason C. Nucciarone\" <nuccitheboss@ubuntu.com>",
    "download_url": "https://files.pythonhosted.org/packages/8f/b6/d205b58796d51253352f8a14fc3742110fffb39475f6bba66633461e2190/slurmutils-1.1.5.tar.gz",
    "platform": null,
    "description": "# slurmutils\n\n![PyPI - Version](https://img.shields.io/pypi/v/slurmutils)\n![PyPI - Downloads](https://img.shields.io/pypi/dm/slurmutils)\n![GitHub License](https://img.shields.io/github/license/charmed-hpc/slurmutils)\n[![Matrix](https://img.shields.io/matrix/ubuntu-hpc%3Amatrix.org?logo=matrix&label=ubuntu-hpc)](https://matrix.to/#/#hpc:ubuntu.com)\n\nUtilities and APIs for interfacing with the Slurm workload manager.\n\nslurmutils is a collection of various utilities that make it easier \nfor you and your friends to interface with the Slurm workload manager, especially if you \nare orchestrating deployments of new and current Slurm clusters. Gone are the days of\nseething over incomplete Jinja2 templates. Current utilities shipped in the \nslurmutils package include:\n\n#### `from slurmutils import ...`\n\n* `calculate_rs`: A function for calculating the ranges and strides of an iterable with\n  unique elements. This function can be used to help convert arrays of node hostnames,\n  device file ids, etc into a Slurm hostname specification.\n* `acctgatherconfig`: An editor for _acct_gather.conf_ configuration files.\n* `cgroupconfig`: An editor for _cgroup.conf_ configuration files.\n* `gresconfig`: An editor for _gres.conf_ configuration files.\n* `ociconfig`: An editor for _oci.conf_ configuration files.\n* `slurmconfig`: An editor for _slurm.conf_ configuration files.\n* `slurmdbdconfig`: An editor for _slurmdbd.conf_ configuration files.\n\nFor more information on how to use or contribute to slurmutils, \ncheck out the [Getting Started](#-getting-started) and [Development](#-development) \nsections below \ud83d\udc47\n\n## \u2728 Getting Started\n\n### Installation\n\n#### Option 1: Install from PyPI\n\n```shell\n$ python3 -m pip install slurmutils\n```\n\n#### Option 2: Install from source\n\n```shell\n$ pip install .\n```\n\n### Usage\n\n#### `slurmutils`\n\nThe top-level provides access to some utilities that streamline common Slurm-related\noperations such as calculating the ranges and strides for a Slurm hostname specification\nor editing configuration files in-place. Here's some example operations you can perform \nwith these utilities:\n\n##### `calculate_rs`\n\n###### Calculate a range and/or stride from a list of node hostnames\n\n```python3\nfrom os.path import commonprefix\n\nfrom slurmutils import calculate_rs\n\nnodes = [\"juju-abc654-1\", \"juju-abc654-2\", \"juju-abc654-4\"]\nprefix = commonprefix(nodes)\nnums = [int(n.partition(prefix)[2]) for n in nodes]\nslurm_host_spec = prefix + calculate_rs(nums)  # \"juju-abc654-[1-2,4]\"\n```\n\n###### Calculate a device file range for Nvidia GPUs\n\n```python3\nfrom pathlib import Path\n\nfrom slurmutils import calculate_rs\n\ndevice_files = [file for file in Path(\"/dev\").iterdir() if \"nvidia\" in file.name]\nprefix = \"/dev/nvidia\"\nnums = [int(n.partition(prefix)[2]) for n in device_files]\nfile_spec = prefix + calculate_rs(nums)  # \"/dev/nvidia[0-4]\"\n```\n\n##### `acctgatherconfig`\n\n###### Edit a pre-existing _acct_gather.conf_ configuration file\n\n```python\nfrom slurmutils import acctgatherconfig\n\nwith acctgatherconfig.edit(\"/etc/slurm/acct_gather.conf\") as config:\n    config.profile_influxdb_database = \"test_acct_gather_db\"\n    config.profile_influxdb_default = [\"none\"]\n    config.profile_influxdb_host = \"testhostname1\"\n    config.profile_influxdb_pass = \"testpassword1\"\n    config.profile_influxdb_rt_policy = \"testpolicy1\"\n    config.profile_influxdb_user = \"testuser1\"\n    config.profile_influxdb_timeout = 20\n```\n\n##### `cgroupconfig`\n\n###### Edit a pre-existing _cgroup.conf_ configuration file\n\n```python\nfrom slurmutils import cgroupconfig\n\nwith cgroupconfig.edit(\"/etc/slurm/cgroup.conf\") as config:\n    config.constrain_cores = True\n    config.constrain_devices = True\n    config.constrain_ram_space = True\n    config.constrain_swap_space = True\n```\n\n##### `gresconfig`\n\n###### Edit a pre-existing _gres.conf_ configuration file\n\n```python\nfrom slurmutils import Gres, GresList, gresconfig\n\nwith gresconfig.edit(\"/etc/slurm/gres.conf\") as config:\n    gres1 = Gres(\n        name=\"gpu\",\n        type=\"epyc\",\n        file=\"/dev/amd4\",\n        cores=[0, 1],\n    )\n    gres2 = Gres(\n        name=\"gpu\",\n        nodename=\"juju-abc654-[1-20]\",\n        type=\"epyc\",\n        file=\"/dev/amd[0-3]\",\n        count=\"12G\",\n    )\n    config.auto_detect = \"rsmi\"\n    config.gres[\"gpu\"] = GresList(gres1, gres2)\n```\n\n##### `ociconfig`\n\n###### Edit a pre-existing _oci.conf_ configuration file\n\n```python\nfrom slurmutils import ociconfig\n\nwith ociconfig.edit(\"/etc/slurm/oci.conf\") as config:\n    config.ignore_file_config_json = False\n    config.env_exclude = \"^(SLURM_CONF|SLURM_CONF_SERVER|SLURM_JWT)=\"\n    config.create_env_file = \"newline\"\n    config.std_io_debug = \"debug\"\n    config.syslog_debug = \"debug\"\n```\n\n##### `slurmconfig`\n\n###### Edit a pre-existing _slurm.conf_ configuration file\n\n```python\nfrom slurmutils import slurmconfig\n\nwith slurmconfig.edit(\"/etc/slurm/slurm.conf\") as config:\n    del config.inactive_limit\n    config.max_job_count = 20000\n    config.proctrack_type = \"proctrack/linuxproc\"\n```\n\n###### Add a new node to the _slurm.conf_ file\n\n```python\nfrom slurmutils import Node, slurmconfig\n\nwith slurmconfig.edit(\"/etc/slurm/slurm.conf\") as config:\n    node = Node(\n        nodename=\"batch-[0-25]\",\n        nodeaddr=\"12.34.56.78\",\n        cpus=1,\n        realmemory=1000,\n        tmpdisk=10000,\n    )\n    config.nodes[node.node_name] = node\n```\n\n##### `slurmdbdconfig`\n\n###### Edit a pre-existing _slurmdbd.conf_ configuration file\n\n```python\nfrom slurmutils import slurmdbdconfig\n\nwith slurmdbdconfig.edit(\"/etc/slurm/slurmdbd.conf\") as config:\n    config.archive_usage = True\n    config.log_file = \"/var/spool/slurmdbd.log\"\n    config.debug_flags = [\"db_event\", \"db_job\", \"db_usage\"]\n    del config.auth_alt_types\n    del config.auth_alt_parameters\n```\n\n## \ud83e\udd14 What's next?\n\nIf you want to learn more about all the things you can do with slurmutils, \nhere are some further resources for you to explore:\n\n* [Open an issue](https://github.com/charmed-hpc/slurmutils/issues/new?title=ISSUE+TITLE&body=*Please+describe+your+issue*)\n* [Ask a question on GitHub](https://github.com/orgs/charmed-hpc/discussions/categories/q-a)\n\n## \ud83d\udee0\ufe0f Development\n\nThe project uses [just](https://github.com/casey/just) and [uv](https://github.com/astral-sh/uv) \nfor development, which provides some useful commands that will help you while hacking on slurmutils:\n\n```shell\njust fmt          # Apply formatting standards to code\njust lint         # Check code against coding style standards\njust typecheck    # Run static type checks\njust unit         # Run unit tests\n```\n\nIf you're interested in contributing your work to slurmutils, \ntake a look at our [contributing guidelines](./CONTRIBUTING.md) for further details.\n\n## \ud83e\udd1d Project and community\n\nslurmutils is a project of the [Ubuntu High-Performance Computing community](https://ubuntu.com/community/governance/teams/hpc).\nInterested in contributing bug fixes, new editors, documentation, or feedback? Want to join the Ubuntu HPC community? You\u2019ve come to the right place \ud83e\udd29\n\nHere\u2019s some links to help you get started with joining the community:\n\n* [Ubuntu Code of Conduct](https://ubuntu.com/community/ethos/code-of-conduct)\n* [Contributing guidelines](./CONTRIBUTING.md)\n* [Join the conversation on Matrix](https://matrix.to/#/#hpc:ubuntu.com)\n* [Get the latest news on Discourse](https://discourse.ubuntu.com/c/hpc/151)\n* [Ask and answer questions on GitHub](https://github.com/orgs/charmed-hpc/discussions/categories/q-a)\n\n## \ud83d\udccb License\n\nslurmutils is free software, distributed under the GNU Lesser General Public License, v3.0.\nSee the [LGPL-3.0 LICENSE](./LICENSE) file for further details.\n",
    "bugtrack_url": null,
    "license": null,
    "summary": "Utilities and APIs for interfacing with the Slurm workload manager.",
    "version": "1.1.5",
    "project_urls": {
        "issues": "https://github.com/charmed-hpc/slurmutils/issues",
        "repository": "https://github.com/charmed-hpc/slurmutils"
    },
    "split_keywords": [
        "hpc",
        " administration",
        " orchestration",
        " utility"
    ],
    "urls": [
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "5c473eeb525e2fdcd86c5762656c09aba19a864bf34ceb27b8ba770150dcb8a5",
                "md5": "ca75566c6dc1ddd04c50f18755704059",
                "sha256": "507059697d49ae3d1b23a481c4ca6162386106129c45de628dc9f2fead976c22"
            },
            "downloads": -1,
            "filename": "slurmutils-1.1.5-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "ca75566c6dc1ddd04c50f18755704059",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.10",
            "size": 50049,
            "upload_time": "2025-07-16T13:24:27",
            "upload_time_iso_8601": "2025-07-16T13:24:27.748730Z",
            "url": "https://files.pythonhosted.org/packages/5c/47/3eeb525e2fdcd86c5762656c09aba19a864bf34ceb27b8ba770150dcb8a5/slurmutils-1.1.5-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "8fb6d205b58796d51253352f8a14fc3742110fffb39475f6bba66633461e2190",
                "md5": "c6f7ac78dd2ef24d80084a9f1fd9653b",
                "sha256": "259f2fc880b92776e66a9916d066ff840147d644b4c6221d62f79985373c916e"
            },
            "downloads": -1,
            "filename": "slurmutils-1.1.5.tar.gz",
            "has_sig": false,
            "md5_digest": "c6f7ac78dd2ef24d80084a9f1fd9653b",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.10",
            "size": 74487,
            "upload_time": "2025-07-16T13:24:29",
            "upload_time_iso_8601": "2025-07-16T13:24:29.017452Z",
            "url": "https://files.pythonhosted.org/packages/8f/b6/d205b58796d51253352f8a14fc3742110fffb39475f6bba66633461e2190/slurmutils-1.1.5.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2025-07-16 13:24:29",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "charmed-hpc",
    "github_project": "slurmutils",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "lcname": "slurmutils"
}
        
Elapsed time: 1.72076s