cpu-quota


Namecpu-quota JSON
Version 0.1.0 PyPI version JSON
download
home_pageNone
SummaryDetermine the actual number of CPUs available based on systemd resource limits - more accurate than nproc in shared systems with CPU quotas
upload_time2025-07-29 14:48:14
maintainerNone
docs_urlNone
authorNone
requires_python>=3.8
licenseMIT
keywords cpu nproc parallel quota resource-limits shared-computing systemd
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            
# CPU Quotas (soma login nodes)

The soma login nodes are shared resources. The total CPU resources are _soft-limited_ in order to support multiple users. This is done using `systemd` quotas.

Unfortunately, `Python` libraries and command line utilities like `nproc` don't know about the soft limits and provide inaccurate user CPU core values, leading to oversubscribe processes when running custom parallel scripts.

# Workaround

In this repository you will find some useful scripts that get the user CPU qouta values for soma login nodes:

# 1. python ([get_available_cpus.py](get_available_cpus.py))

This script supports multiple usage modes and options:

## Features:

* Reads systemd CPU *quota* from `/run/systemd/system/user-$UID.slice.d/50-CPUQuota.conf`
* Handles both percentage format (e.g., "200%") and decimal format (e.g., "2.0")
* Provides a conservative mode that reserves CPU for system overhead
* Can be imported as a module in other `Python` scripts
* Includes a quiet mode ideal for scripting

## Usage:

```bash
# Make executable
chmod +x get_available_cpus.py

# Basic usage
./get_available_cpus.py

# Conservative mode (reserves some CPU for system)
./get_available_cpus.py --conservative

# Quiet mode (just the number)
./get_available_cpus.py --quiet
```

## In python scripts:

```python
from get_available_cpus import get_parallel_workers

# Use in multiprocessing
import multiprocessing as mp
num_workers = get_parallel_workers()
with mp.Pool(num_workers) as pool:
    # your parallel code here
```

# 2. bash ([get_available_cpus.sh](get_available_cpus.sh))

This is a bash version of the script above for command-line usage:

## Usage:

```bash
# Make executable
chmod +x get_available_cpus.sh

# Basic usage
./get_available_cpus.sh

# Use in other scripts
NUM_CPUS=$(./get_available_cpus.sh --quiet)
```

# 3. pip installing cpu_quota

## How to install:
```bash
python -m pip install cpu_quota
```

## Usage:

```python
from cpu_quota import get_parallel_workers
num_workers = get_parallel_workers()
```


## Features:
* **Systemd Integration:** Reads actual CPU quota limits rather than just hardware CPU count
* **Fallback:** Falls back to nproc if no systemd limits are found
* **Conservative Mode:** Optionally reserves CPU capacity for system overhead
* **Error Handling:** Gracefully handles missing files or parsing errors
* **Flexible Output:** Quiet mode for scripting, verbose mode for debugging

            

Raw data

            {
    "_id": null,
    "home_page": null,
    "name": "cpu-quota",
    "maintainer": null,
    "docs_url": null,
    "requires_python": ">=3.8",
    "maintainer_email": null,
    "keywords": "cpu, nproc, parallel, quota, resource-limits, shared-computing, systemd",
    "author": null,
    "author_email": "Omar Valerio <omar.valerio@mpinb.mpg.de>",
    "download_url": "https://files.pythonhosted.org/packages/f2/35/75899b4012cef115cfbc9e1ec3b977b204ad2d2284ec4c72577c13285c1f/cpu_quota-0.1.0.tar.gz",
    "platform": null,
    "description": "\n# CPU Quotas (soma login nodes)\n\nThe soma login nodes are shared resources. The total CPU resources are _soft-limited_ in order to support multiple users. This is done using `systemd` quotas.\n\nUnfortunately, `Python` libraries and command line utilities like `nproc` don't know about the soft limits and provide inaccurate user CPU core values, leading to oversubscribe processes when running custom parallel scripts.\n\n# Workaround\n\nIn this repository you will find some useful scripts that get the user CPU qouta values for soma login nodes:\n\n# 1. python ([get_available_cpus.py](get_available_cpus.py))\n\nThis script supports multiple usage modes and options:\n\n## Features:\n\n* Reads systemd CPU *quota* from `/run/systemd/system/user-$UID.slice.d/50-CPUQuota.conf`\n* Handles both percentage format (e.g., \"200%\") and decimal format (e.g., \"2.0\")\n* Provides a conservative mode that reserves CPU for system overhead\n* Can be imported as a module in other `Python` scripts\n* Includes a quiet mode ideal for scripting\n\n## Usage:\n\n```bash\n# Make executable\nchmod +x get_available_cpus.py\n\n# Basic usage\n./get_available_cpus.py\n\n# Conservative mode (reserves some CPU for system)\n./get_available_cpus.py --conservative\n\n# Quiet mode (just the number)\n./get_available_cpus.py --quiet\n```\n\n## In python scripts:\n\n```python\nfrom get_available_cpus import get_parallel_workers\n\n# Use in multiprocessing\nimport multiprocessing as mp\nnum_workers = get_parallel_workers()\nwith mp.Pool(num_workers) as pool:\n    # your parallel code here\n```\n\n# 2. bash ([get_available_cpus.sh](get_available_cpus.sh))\n\nThis is a bash version of the script above for command-line usage:\n\n## Usage:\n\n```bash\n# Make executable\nchmod +x get_available_cpus.sh\n\n# Basic usage\n./get_available_cpus.sh\n\n# Use in other scripts\nNUM_CPUS=$(./get_available_cpus.sh --quiet)\n```\n\n# 3. pip installing cpu_quota\n\n## How to install:\n```bash\npython -m pip install cpu_quota\n```\n\n## Usage:\n\n```python\nfrom cpu_quota import get_parallel_workers\nnum_workers = get_parallel_workers()\n```\n\n\n## Features:\n* **Systemd Integration:** Reads actual CPU quota limits rather than just hardware CPU count\n* **Fallback:** Falls back to nproc if no systemd limits are found\n* **Conservative Mode:** Optionally reserves CPU capacity for system overhead\n* **Error Handling:** Gracefully handles missing files or parsing errors\n* **Flexible Output:** Quiet mode for scripting, verbose mode for debugging\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "Determine the actual number of CPUs available based on systemd resource limits - more accurate than nproc in shared systems with CPU quotas",
    "version": "0.1.0",
    "project_urls": {
        "Homepage": "https://github.com/mpinb/howto-cpu-quota",
        "Issues": "https://github.com/mpinb/howto-cpu-quota/issues",
        "Repository": "https://github.com/mpinb/howto-cpu-quota"
    },
    "split_keywords": [
        "cpu",
        " nproc",
        " parallel",
        " quota",
        " resource-limits",
        " shared-computing",
        " systemd"
    ],
    "urls": [
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "ec5423e9d5c801a52d82afeb66a0c232cd01861f190aadb6c70c811f5ca1c8e7",
                "md5": "b976d741b982cffcad86619800c304e3",
                "sha256": "bbe5e7ae165fb8b9ad1f8d0923a41b1e285eaec13a0138fbb717cc3774a80734"
            },
            "downloads": -1,
            "filename": "cpu_quota-0.1.0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "b976d741b982cffcad86619800c304e3",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.8",
            "size": 5642,
            "upload_time": "2025-07-29T14:48:13",
            "upload_time_iso_8601": "2025-07-29T14:48:13.302613Z",
            "url": "https://files.pythonhosted.org/packages/ec/54/23e9d5c801a52d82afeb66a0c232cd01861f190aadb6c70c811f5ca1c8e7/cpu_quota-0.1.0-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "f23575899b4012cef115cfbc9e1ec3b977b204ad2d2284ec4c72577c13285c1f",
                "md5": "d133fdb13cd3a9b633d8eee98c6267d8",
                "sha256": "affe53ccc78f65974dbc1f3bc90711b89d5f343e790996e345af88254fe732ef"
            },
            "downloads": -1,
            "filename": "cpu_quota-0.1.0.tar.gz",
            "has_sig": false,
            "md5_digest": "d133fdb13cd3a9b633d8eee98c6267d8",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.8",
            "size": 7380,
            "upload_time": "2025-07-29T14:48:14",
            "upload_time_iso_8601": "2025-07-29T14:48:14.592317Z",
            "url": "https://files.pythonhosted.org/packages/f2/35/75899b4012cef115cfbc9e1ec3b977b204ad2d2284ec4c72577c13285c1f/cpu_quota-0.1.0.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2025-07-29 14:48:14",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "mpinb",
    "github_project": "howto-cpu-quota",
    "github_not_found": true,
    "lcname": "cpu-quota"
}
        
Elapsed time: 2.36498s