| Name | prometheus-dirsize-exporter JSON |
| Version |
3.1
JSON |
| download |
| home_page | None |
| Summary | None |
| upload_time | 2024-03-26 21:51:05 |
| maintainer | None |
| docs_url | None |
| author | yuvipanda |
| requires_python | None |
| license | 3-BSD |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# prometheus-dirsize-exporter
[](https://badge.fury.io/py/prometheus-dirsize-exporter)
Export directory size metrics efficiently.
## Why?
When providing multi-user interactive computing services (with a HPC cluster
or with JupyterHub), it's very helpful to know the home directory sizes of
each user over time. However, as NFS is often used, running `du` constantly
takes a long time, uses too many IOPS that we may not have many of, and is
plain inefficient.
This project provides a way to keep track of directory sizes with a *budgeted*
amount of IOPS. You can ask it to take however much time it needs but not
use more than 100 IOPS, and it will do that. We do not necessarily need very
up to date directory size metrics, so this is ok.
## Installation
Install the package from PyPI:
```bash
pip install prometheus-dirsize-exporter
```
## Running
You can start the process from the commandline:
```bash
dirsize-exporter <path-to-parent-directory> <iops-budget> <wait-time-in-minutes>
```
For example, to export metrics about directories under `/home`, using no more
than 200 IO operations per second, updating every 60minutes, you would run:
```bash
dirsize-exporter /home 200 60
```
You can check out the metrics by hitting `http://localhost:8000`. The port can
be controlled via a `--port` argument.
## Metrics recorded
The following metrics are recorded for all top level subidrectories of the
parent directory:
- Total Size (in bytes)
- Last Modified (including all the descendents)
- Total Number of Entries (Files, directories & symlinks)
- Processing Time required to gather this information
This is only reported if `--enable-detailed-processing-time-metric` flag is
passed, to prevent possible explosion of stored size of prometheus metrics
when collected. This information is also not particularly useful outside
of debugging this exporter, and as it varies each run, compresses poorly.
- Last updated
## Limitations
- As directory contents might change in the course of a single run as we wait
for budgets to become available, information about a directory may not be
exactly correct immediately.
- Because we do not spread the IOPS through time, the IO usage is 'spiky' -
all IOPS get done at the beginning of a second, and then it goes silent.
If you have a big IOPS budget, this can cause performance degradation.
Raw data
{
"_id": null,
"home_page": null,
"name": "prometheus-dirsize-exporter",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": null,
"author": "yuvipanda",
"author_email": "yuvipanda@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/c3/52/57d73b087565e98f6b21579c139960ed9361c7644a1227c8d889872ef2fc/prometheus-dirsize-exporter-3.1.tar.gz",
"platform": null,
"description": "# prometheus-dirsize-exporter\n\n[](https://badge.fury.io/py/prometheus-dirsize-exporter)\n\nExport directory size metrics efficiently.\n\n## Why?\n\nWhen providing multi-user interactive computing services (with a HPC cluster\nor with JupyterHub), it's very helpful to know the home directory sizes of\neach user over time. However, as NFS is often used, running `du` constantly\ntakes a long time, uses too many IOPS that we may not have many of, and is\nplain inefficient.\n\nThis project provides a way to keep track of directory sizes with a *budgeted*\namount of IOPS. You can ask it to take however much time it needs but not\nuse more than 100 IOPS, and it will do that. We do not necessarily need very\nup to date directory size metrics, so this is ok.\n\n## Installation\n\nInstall the package from PyPI:\n\n```bash\npip install prometheus-dirsize-exporter\n```\n\n## Running\n\nYou can start the process from the commandline:\n\n```bash\ndirsize-exporter <path-to-parent-directory> <iops-budget> <wait-time-in-minutes>\n```\n\nFor example, to export metrics about directories under `/home`, using no more\nthan 200 IO operations per second, updating every 60minutes, you would run:\n\n```bash\ndirsize-exporter /home 200 60\n```\n\nYou can check out the metrics by hitting `http://localhost:8000`. The port can\nbe controlled via a `--port` argument.\n\n## Metrics recorded\n\nThe following metrics are recorded for all top level subidrectories of the\nparent directory:\n\n- Total Size (in bytes)\n- Last Modified (including all the descendents)\n- Total Number of Entries (Files, directories & symlinks)\n- Processing Time required to gather this information\n This is only reported if `--enable-detailed-processing-time-metric` flag is\n passed, to prevent possible explosion of stored size of prometheus metrics\n when collected. This information is also not particularly useful outside\n of debugging this exporter, and as it varies each run, compresses poorly.\n- Last updated\n\n## Limitations\n\n- As directory contents might change in the course of a single run as we wait\n for budgets to become available, information about a directory may not be\n exactly correct immediately.\n- Because we do not spread the IOPS through time, the IO usage is 'spiky' -\n all IOPS get done at the beginning of a second, and then it goes silent.\n If you have a big IOPS budget, this can cause performance degradation.\n",
"bugtrack_url": null,
"license": "3-BSD",
"summary": null,
"version": "3.1",
"project_urls": null,
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "99c43c5382a5b0df5d099e21d976160f997dfaa29f93cf6182c169063623dba3",
"md5": "fce67b37e297af4b3f5c7348a707e23f",
"sha256": "5f229a085e4cd6db48e444387d09583beb63136c6c9ef512f81b3537bca2f8ce"
},
"downloads": -1,
"filename": "prometheus_dirsize_exporter-3.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "fce67b37e297af4b3f5c7348a707e23f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 7233,
"upload_time": "2024-03-26T21:51:03",
"upload_time_iso_8601": "2024-03-26T21:51:03.991253Z",
"url": "https://files.pythonhosted.org/packages/99/c4/3c5382a5b0df5d099e21d976160f997dfaa29f93cf6182c169063623dba3/prometheus_dirsize_exporter-3.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c35257d73b087565e98f6b21579c139960ed9361c7644a1227c8d889872ef2fc",
"md5": "2878b23bda9d1071b42dfe4966c5acf9",
"sha256": "dd81f3fdb6155049366a281db5e8a13c51d3674e51f71a27a50c7069cd12a40b"
},
"downloads": -1,
"filename": "prometheus-dirsize-exporter-3.1.tar.gz",
"has_sig": false,
"md5_digest": "2878b23bda9d1071b42dfe4966c5acf9",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 6005,
"upload_time": "2024-03-26T21:51:05",
"upload_time_iso_8601": "2024-03-26T21:51:05.649118Z",
"url": "https://files.pythonhosted.org/packages/c3/52/57d73b087565e98f6b21579c139960ed9361c7644a1227c8d889872ef2fc/prometheus-dirsize-exporter-3.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-26 21:51:05",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "prometheus-dirsize-exporter"
}
