Name | cpu-quota JSON |
Version |
0.1.0
JSON |
| download |
home_page | None |
Summary | Determine the actual number of CPUs available based on systemd resource limits - more accurate than nproc in shared systems with CPU quotas |
upload_time | 2025-07-29 14:48:14 |
maintainer | None |
docs_url | None |
author | None |
requires_python | >=3.8 |
license | MIT |
keywords |
cpu
nproc
parallel
quota
resource-limits
shared-computing
systemd
|
VCS |
 |
bugtrack_url |
|
requirements |
No requirements were recorded.
|
Travis-CI |
No Travis.
|
coveralls test coverage |
No coveralls.
|
# CPU Quotas (soma login nodes)
The soma login nodes are shared resources. The total CPU resources are _soft-limited_ in order to support multiple users. This is done using `systemd` quotas.
Unfortunately, `Python` libraries and command line utilities like `nproc` don't know about the soft limits and provide inaccurate user CPU core values, leading to oversubscribe processes when running custom parallel scripts.
# Workaround
In this repository you will find some useful scripts that get the user CPU qouta values for soma login nodes:
# 1. python ([get_available_cpus.py](get_available_cpus.py))
This script supports multiple usage modes and options:
## Features:
* Reads systemd CPU *quota* from `/run/systemd/system/user-$UID.slice.d/50-CPUQuota.conf`
* Handles both percentage format (e.g., "200%") and decimal format (e.g., "2.0")
* Provides a conservative mode that reserves CPU for system overhead
* Can be imported as a module in other `Python` scripts
* Includes a quiet mode ideal for scripting
## Usage:
```bash
# Make executable
chmod +x get_available_cpus.py
# Basic usage
./get_available_cpus.py
# Conservative mode (reserves some CPU for system)
./get_available_cpus.py --conservative
# Quiet mode (just the number)
./get_available_cpus.py --quiet
```
## In python scripts:
```python
from get_available_cpus import get_parallel_workers
# Use in multiprocessing
import multiprocessing as mp
num_workers = get_parallel_workers()
with mp.Pool(num_workers) as pool:
# your parallel code here
```
# 2. bash ([get_available_cpus.sh](get_available_cpus.sh))
This is a bash version of the script above for command-line usage:
## Usage:
```bash
# Make executable
chmod +x get_available_cpus.sh
# Basic usage
./get_available_cpus.sh
# Use in other scripts
NUM_CPUS=$(./get_available_cpus.sh --quiet)
```
# 3. pip installing cpu_quota
## How to install:
```bash
python -m pip install cpu_quota
```
## Usage:
```python
from cpu_quota import get_parallel_workers
num_workers = get_parallel_workers()
```
## Features:
* **Systemd Integration:** Reads actual CPU quota limits rather than just hardware CPU count
* **Fallback:** Falls back to nproc if no systemd limits are found
* **Conservative Mode:** Optionally reserves CPU capacity for system overhead
* **Error Handling:** Gracefully handles missing files or parsing errors
* **Flexible Output:** Quiet mode for scripting, verbose mode for debugging
Raw data
{
"_id": null,
"home_page": null,
"name": "cpu-quota",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "cpu, nproc, parallel, quota, resource-limits, shared-computing, systemd",
"author": null,
"author_email": "Omar Valerio <omar.valerio@mpinb.mpg.de>",
"download_url": "https://files.pythonhosted.org/packages/f2/35/75899b4012cef115cfbc9e1ec3b977b204ad2d2284ec4c72577c13285c1f/cpu_quota-0.1.0.tar.gz",
"platform": null,
"description": "\n# CPU Quotas (soma login nodes)\n\nThe soma login nodes are shared resources. The total CPU resources are _soft-limited_ in order to support multiple users. This is done using `systemd` quotas.\n\nUnfortunately, `Python` libraries and command line utilities like `nproc` don't know about the soft limits and provide inaccurate user CPU core values, leading to oversubscribe processes when running custom parallel scripts.\n\n# Workaround\n\nIn this repository you will find some useful scripts that get the user CPU qouta values for soma login nodes:\n\n# 1. python ([get_available_cpus.py](get_available_cpus.py))\n\nThis script supports multiple usage modes and options:\n\n## Features:\n\n* Reads systemd CPU *quota* from `/run/systemd/system/user-$UID.slice.d/50-CPUQuota.conf`\n* Handles both percentage format (e.g., \"200%\") and decimal format (e.g., \"2.0\")\n* Provides a conservative mode that reserves CPU for system overhead\n* Can be imported as a module in other `Python` scripts\n* Includes a quiet mode ideal for scripting\n\n## Usage:\n\n```bash\n# Make executable\nchmod +x get_available_cpus.py\n\n# Basic usage\n./get_available_cpus.py\n\n# Conservative mode (reserves some CPU for system)\n./get_available_cpus.py --conservative\n\n# Quiet mode (just the number)\n./get_available_cpus.py --quiet\n```\n\n## In python scripts:\n\n```python\nfrom get_available_cpus import get_parallel_workers\n\n# Use in multiprocessing\nimport multiprocessing as mp\nnum_workers = get_parallel_workers()\nwith mp.Pool(num_workers) as pool:\n # your parallel code here\n```\n\n# 2. bash ([get_available_cpus.sh](get_available_cpus.sh))\n\nThis is a bash version of the script above for command-line usage:\n\n## Usage:\n\n```bash\n# Make executable\nchmod +x get_available_cpus.sh\n\n# Basic usage\n./get_available_cpus.sh\n\n# Use in other scripts\nNUM_CPUS=$(./get_available_cpus.sh --quiet)\n```\n\n# 3. pip installing cpu_quota\n\n## How to install:\n```bash\npython -m pip install cpu_quota\n```\n\n## Usage:\n\n```python\nfrom cpu_quota import get_parallel_workers\nnum_workers = get_parallel_workers()\n```\n\n\n## Features:\n* **Systemd Integration:** Reads actual CPU quota limits rather than just hardware CPU count\n* **Fallback:** Falls back to nproc if no systemd limits are found\n* **Conservative Mode:** Optionally reserves CPU capacity for system overhead\n* **Error Handling:** Gracefully handles missing files or parsing errors\n* **Flexible Output:** Quiet mode for scripting, verbose mode for debugging\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Determine the actual number of CPUs available based on systemd resource limits - more accurate than nproc in shared systems with CPU quotas",
"version": "0.1.0",
"project_urls": {
"Homepage": "https://github.com/mpinb/howto-cpu-quota",
"Issues": "https://github.com/mpinb/howto-cpu-quota/issues",
"Repository": "https://github.com/mpinb/howto-cpu-quota"
},
"split_keywords": [
"cpu",
" nproc",
" parallel",
" quota",
" resource-limits",
" shared-computing",
" systemd"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "ec5423e9d5c801a52d82afeb66a0c232cd01861f190aadb6c70c811f5ca1c8e7",
"md5": "b976d741b982cffcad86619800c304e3",
"sha256": "bbe5e7ae165fb8b9ad1f8d0923a41b1e285eaec13a0138fbb717cc3774a80734"
},
"downloads": -1,
"filename": "cpu_quota-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "b976d741b982cffcad86619800c304e3",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 5642,
"upload_time": "2025-07-29T14:48:13",
"upload_time_iso_8601": "2025-07-29T14:48:13.302613Z",
"url": "https://files.pythonhosted.org/packages/ec/54/23e9d5c801a52d82afeb66a0c232cd01861f190aadb6c70c811f5ca1c8e7/cpu_quota-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "f23575899b4012cef115cfbc9e1ec3b977b204ad2d2284ec4c72577c13285c1f",
"md5": "d133fdb13cd3a9b633d8eee98c6267d8",
"sha256": "affe53ccc78f65974dbc1f3bc90711b89d5f343e790996e345af88254fe732ef"
},
"downloads": -1,
"filename": "cpu_quota-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "d133fdb13cd3a9b633d8eee98c6267d8",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 7380,
"upload_time": "2025-07-29T14:48:14",
"upload_time_iso_8601": "2025-07-29T14:48:14.592317Z",
"url": "https://files.pythonhosted.org/packages/f2/35/75899b4012cef115cfbc9e1ec3b977b204ad2d2284ec4c72577c13285c1f/cpu_quota-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-29 14:48:14",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mpinb",
"github_project": "howto-cpu-quota",
"github_not_found": true,
"lcname": "cpu-quota"
}