# 📐 ⏱ 🧮 File-Size-Age Metrics Exporter
[![Poetry][badge_poetry]][poetry]
[Prometheus][1] exporter providing size and age metrics about files.
![FSA Metrics Grafana Demo Panel][fsa_panel]
## ⚙🔧 Installation ⚙🔧
Example installation on Debian / Ubuntu:
```bash
# required for creating Python virtualenvs:
apt update
apt install -y python3-venv
# create a virtualenv in /opt:
python3 -m venv /opt/fsa-metrics
# update 'pip' and install the 'file-size-age-metrics' package:
/opt/fsa-metrics/bin/pip install --upgrade pip
/opt/fsa-metrics/bin/pip install file-size-age-metrics
```
## 🏃 Running in foreground mode 🏃
This is mostly relevant for testing configuration settings and checking if the
exporter works as expected - to do this either activate the previously created
Python environment or call the `fsa-metrics` script using the full path to that
environment.
A configuration file is required for running the metrics exporter. Simply copy
the [config-example.yaml][3] file to e.g. `config.yaml` and adjust the settings
there (alternatively, call `fsa-metrics --config SHOWCONFIGDEFAULTS` to have a
configuration example printed to stdout). Then run the exporter like this:
```bash
fsa-metrics --config config.yaml
```
The exporter running in foreground can be terminated as usual via `Ctrl+C`.
## 👟 Running as a service 👟
```bash
adduser --system fsaexporter
SITE_PKGS=$(/opt/fsa-metrics/bin/pip show file-size-age-metrics |
grep '^Location: ' |
cut -d ' ' -f 2
)
cp -v "$SITE_PKGS"/resources/systemd/fsa-metrics.service /etc/systemd/system/
cp -v "$SITE_PKGS"/resources/config-example.yaml /etc/fsa-metrics.yaml
vim /etc/fsa-metrics.yaml # <- adapt settings to your requirements
systemctl daemon-reload
systemctl edit fsa-metrics.service
```
The last command will open an editor with the override configuration of the
service's unit file. Add a section like this **at the top** of the override
file, specifying where to find your configuration file for the service:
```text
[Service]
### configuration file for the FSA exporter service:
Environment=FSA_CONFIG=/etc/fsa-metrics.yaml
```
Note: on *Ubuntu 20.04* the `systemct edit` command will present you with an
empty file, so you will have to copy the respective lines from below or the
provided *central* unit file.
Finally enable the service and start it right away. The second line will show
the log messages on the console until `Ctrl+C` is pressed. This way you should
be able to tell if the service has started up properly and is providing metrics
on the configured port:
```bash
systemctl enable --now fsa-metrics.service
journalctl --follow --unit fsa-metrics
```
Open ports for the `fsa-metrics` exporter:
```bash
SOURCE="any" # <-- put an IP address here to restrict access more
PORT="16061" # <-- adjust in case it's changed from this default value
ufw allow from $SOURCE to any port $PORT
```
## 🪂 Running via cron ⏰
In case you need the metrics exporter on a system where you are lacking
administrative privileges, running the exporter in kind of a
*poor-man's'service* approach through `cron` using a wrapper script is
absolutely feasible!
The wrapper script assumes the `fsa-metrics` venv will be placed in
`$HOME/.venvs/`, if that's not the case the path prefix in the script requires
to be adjusted.
```bash
mkdir -pv "$HOME/.venvs"
VENV_PATH="$HOME/.venvs/fsa-metrics"
python3 -m venv "$VENV_PATH"
"$VENV_PATH/bin/pip" install --upgrade pip
"$VENV_PATH/bin/pip" install file-size-age-metrics
SITE_PKGS=$("$VENV_PATH/bin/pip" show file-size-age-metrics |
grep '^Location: ' |
cut -d ' ' -f 2
)
cp -v "$SITE_PKGS/resources/config-example.yaml" "$VENV_PATH/fsa-metrics.yaml"
cp -v "$SITE_PKGS/resources/run-metrics-exporter.sh" "$VENV_PATH/bin/"
```
Obviously you also want to adapt the settings in the `.yaml` config file.
Now the wrapper can be put into a cron-job (`crontab -e`) that e.g. executes
once a minute and it will take care to only launch a new instance of the metrics
exporter if none is running. For example:
```cron
* * * * * $HOME/.venvs/fsa-metrics/bin/run-metrics-exporter.sh
```
## Visualization in Grafana
To visualize data in a way as shown in the example panel above, queries like
the following ones may be used:
```text
sort(fsa_age_seconds{instance="pg_server.example.xy"})
sort(fsa_size_bytes{instance="pg_server.example.xy"})
```
## Known limitations
Currently only a single directory tree can be monitored. Adding support for
monitoring several trees with a single process is planned though.
## Scalability and resource usage considerations
The exporter is designed with code simplicity as a goal, it's *not* optimized
for efficiency or low resource usage. A few numbers on an average laptop running
the exporter on a rather large file tree (not recommended, just for
demonstration purposes):
- Number of files monitored: ~200'000
- Memory consumption: ~350 MB
- Metrics collection duration: < 10s
[1]: https://prometheus.io/
[3]: resources/config-example.yaml
[poetry]: https://python-poetry.org/
[badge_poetry]: https://img.shields.io/endpoint?url=https://python-poetry.org/badge/v0.json
[fsa_panel]: https://imcf.one/images/fsa-metrics-demo-panel.png
Raw data
{
"_id": null,
"home_page": null,
"name": "file-size-age-metrics",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.8",
"maintainer_email": null,
"keywords": "prometheus, metrics, file-metrics",
"author": "Niko Ehrenfeuchter",
"author_email": "nikolaus.ehrenfeuchter@unibas.ch",
"download_url": "https://files.pythonhosted.org/packages/37/29/b6bfe804e574c21526cf650831f917a91b007157b826fb7b56dc50b54a94/file_size_age_metrics-1.0.0.tar.gz",
"platform": null,
"description": "# \ud83d\udcd0 \u23f1 \ud83e\uddee File-Size-Age Metrics Exporter\n\n[![Poetry][badge_poetry]][poetry]\n\n[Prometheus][1] exporter providing size and age metrics about files.\n\n![FSA Metrics Grafana Demo Panel][fsa_panel]\n\n## \u2699\ud83d\udd27 Installation \u2699\ud83d\udd27\n\nExample installation on Debian / Ubuntu:\n\n```bash\n# required for creating Python virtualenvs:\napt update\napt install -y python3-venv\n\n# create a virtualenv in /opt:\npython3 -m venv /opt/fsa-metrics\n\n# update 'pip' and install the 'file-size-age-metrics' package:\n/opt/fsa-metrics/bin/pip install --upgrade pip\n/opt/fsa-metrics/bin/pip install file-size-age-metrics\n```\n\n## \ud83c\udfc3 Running in foreground mode \ud83c\udfc3\n\nThis is mostly relevant for testing configuration settings and checking if the\nexporter works as expected - to do this either activate the previously created\nPython environment or call the `fsa-metrics` script using the full path to that\nenvironment.\n\nA configuration file is required for running the metrics exporter. Simply copy\nthe [config-example.yaml][3] file to e.g. `config.yaml` and adjust the settings\nthere (alternatively, call `fsa-metrics --config SHOWCONFIGDEFAULTS` to have a\nconfiguration example printed to stdout). Then run the exporter like this:\n\n```bash\nfsa-metrics --config config.yaml\n```\n\nThe exporter running in foreground can be terminated as usual via `Ctrl+C`.\n\n## \ud83d\udc5f Running as a service \ud83d\udc5f\n\n```bash\nadduser --system fsaexporter\nSITE_PKGS=$(/opt/fsa-metrics/bin/pip show file-size-age-metrics |\n grep '^Location: ' |\n cut -d ' ' -f 2\n)\ncp -v \"$SITE_PKGS\"/resources/systemd/fsa-metrics.service /etc/systemd/system/\ncp -v \"$SITE_PKGS\"/resources/config-example.yaml /etc/fsa-metrics.yaml\nvim /etc/fsa-metrics.yaml # <- adapt settings to your requirements\nsystemctl daemon-reload\nsystemctl edit fsa-metrics.service\n```\n\nThe last command will open an editor with the override configuration of the\nservice's unit file. Add a section like this **at the top** of the override\nfile, specifying where to find your configuration file for the service:\n\n```text\n[Service]\n### configuration file for the FSA exporter service:\nEnvironment=FSA_CONFIG=/etc/fsa-metrics.yaml\n```\n\nNote: on *Ubuntu 20.04* the `systemct edit` command will present you with an\nempty file, so you will have to copy the respective lines from below or the\nprovided *central* unit file.\n\nFinally enable the service and start it right away. The second line will show\nthe log messages on the console until `Ctrl+C` is pressed. This way you should\nbe able to tell if the service has started up properly and is providing metrics\non the configured port:\n\n```bash\nsystemctl enable --now fsa-metrics.service\njournalctl --follow --unit fsa-metrics\n```\n\nOpen ports for the `fsa-metrics` exporter:\n\n```bash\nSOURCE=\"any\" # <-- put an IP address here to restrict access more\nPORT=\"16061\" # <-- adjust in case it's changed from this default value\nufw allow from $SOURCE to any port $PORT\n```\n\n## \ud83e\ude82 Running via cron \u23f0\n\nIn case you need the metrics exporter on a system where you are lacking\nadministrative privileges, running the exporter in kind of a\n*poor-man's'service* approach through `cron` using a wrapper script is\nabsolutely feasible!\n\nThe wrapper script assumes the `fsa-metrics` venv will be placed in\n`$HOME/.venvs/`, if that's not the case the path prefix in the script requires\nto be adjusted.\n\n```bash\nmkdir -pv \"$HOME/.venvs\"\nVENV_PATH=\"$HOME/.venvs/fsa-metrics\"\npython3 -m venv \"$VENV_PATH\"\n\"$VENV_PATH/bin/pip\" install --upgrade pip\n\"$VENV_PATH/bin/pip\" install file-size-age-metrics\nSITE_PKGS=$(\"$VENV_PATH/bin/pip\" show file-size-age-metrics |\n grep '^Location: ' |\n cut -d ' ' -f 2\n)\ncp -v \"$SITE_PKGS/resources/config-example.yaml\" \"$VENV_PATH/fsa-metrics.yaml\"\ncp -v \"$SITE_PKGS/resources/run-metrics-exporter.sh\" \"$VENV_PATH/bin/\"\n```\n\nObviously you also want to adapt the settings in the `.yaml` config file.\n\nNow the wrapper can be put into a cron-job (`crontab -e`) that e.g. executes\nonce a minute and it will take care to only launch a new instance of the metrics\nexporter if none is running. For example:\n\n```cron\n* * * * * $HOME/.venvs/fsa-metrics/bin/run-metrics-exporter.sh\n```\n\n## Visualization in Grafana\n\nTo visualize data in a way as shown in the example panel above, queries like\nthe following ones may be used:\n\n```text\nsort(fsa_age_seconds{instance=\"pg_server.example.xy\"})\nsort(fsa_size_bytes{instance=\"pg_server.example.xy\"})\n```\n\n## Known limitations\n\nCurrently only a single directory tree can be monitored. Adding support for\nmonitoring several trees with a single process is planned though.\n\n## Scalability and resource usage considerations\n\nThe exporter is designed with code simplicity as a goal, it's *not* optimized\nfor efficiency or low resource usage. A few numbers on an average laptop running\nthe exporter on a rather large file tree (not recommended, just for\ndemonstration purposes):\n\n- Number of files monitored: ~200'000\n- Memory consumption: ~350 MB\n- Metrics collection duration: < 10s\n\n[1]: https://prometheus.io/\n[3]: resources/config-example.yaml\n\n[poetry]: https://python-poetry.org/\n[badge_poetry]: https://img.shields.io/endpoint?url=https://python-poetry.org/badge/v0.json\n[fsa_panel]: https://imcf.one/images/fsa-metrics-demo-panel.png\n\n",
"bugtrack_url": null,
"license": "GPL-3.0-or-later",
"summary": "Metrics collector and exporter to monitor file sizes and ages.",
"version": "1.0.0",
"project_urls": {
"Homepage": "https://pypi.org/project/file-size-age-metrics/",
"Organisation Homepage": "https://imcf.one/",
"Repository": "https://github.com/imcf/file-size-age-metrics"
},
"split_keywords": [
"prometheus",
" metrics",
" file-metrics"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "fe0ebf54a194233fdb7edfaa25c3ef0e3e88378de15236c4b4c33e503c4e43e2",
"md5": "fe0e32662ddd37e800933fbe57a1e18d",
"sha256": "93a5e7db66bc623f96ebfd9fab4b4611a2dbfaf9845da6c1fec2f52eb2f0e488"
},
"downloads": -1,
"filename": "file_size_age_metrics-1.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "fe0e32662ddd37e800933fbe57a1e18d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.8",
"size": 23876,
"upload_time": "2025-02-06T14:02:14",
"upload_time_iso_8601": "2025-02-06T14:02:14.363014Z",
"url": "https://files.pythonhosted.org/packages/fe/0e/bf54a194233fdb7edfaa25c3ef0e3e88378de15236c4b4c33e503c4e43e2/file_size_age_metrics-1.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "3729b6bfe804e574c21526cf650831f917a91b007157b826fb7b56dc50b54a94",
"md5": "376e5c9f8114450dd922516140647399",
"sha256": "b41a55bc859410c7359b543aa2b79651e683d2d96e457b276dad33a1e2028454"
},
"downloads": -1,
"filename": "file_size_age_metrics-1.0.0.tar.gz",
"has_sig": false,
"md5_digest": "376e5c9f8114450dd922516140647399",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.8",
"size": 22483,
"upload_time": "2025-02-06T14:02:15",
"upload_time_iso_8601": "2025-02-06T14:02:15.767261Z",
"url": "https://files.pythonhosted.org/packages/37/29/b6bfe804e574c21526cf650831f917a91b007157b826fb7b56dc50b54a94/file_size_age_metrics-1.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-02-06 14:02:15",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "imcf",
"github_project": "file-size-age-metrics",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "file-size-age-metrics"
}
JSON
