# HighLA - Run diagnostics on high load average
Highla checks current load average and executes set of built-in diagnostic or external script to collect data at the precise moment when load average is high.
## Install
~~~shell
# pipx way, recommended
pipx install highla
# pip3 way, obsolete. recommended to use in virtualenv
pip3 install highla
~~~
## Empty diag
highla has one required parameter - Load Average threshold. If current Load Average is lower than threshold, it will do nothing. If current Load Average is higher, it will run diagnostic specified in other arguments.
~~~shell
# no diag at all
# trigger if LA over 20 (current LA is lower, so nothing printed)
$ highla 20
# trigger if LA over 3.5 (prints timestamp and current LA on stdout)
$ highla 3.5
# 2024/11/13 01:29:01 LA: 4.05
~~~
## built-in top
prints top-N CPU-intensive processes
~~~shell
$ highla 1 --top 5
# 2024/11/13 01:35:19 LA: 3.69
## Top CPU usage
792682 wireguard-vanity-keygen 307.9%
877929 highla 2.0%
1192 irq/33-nvidia 1.0%
2158 asterisk 1.0%
3767 ibus-portal 1.0%
~~~
## built-in netstat
Prints summary and all connections in ESTABLISHED/CLOSE_WAIT states
~~~shell
$ highla 1 --tcp
# 2024/11/13 01:37:32 LA: 3.47
## TCP connections: ESTABLISHED: 13, LISTEN: 41, CLOSE_WAIT: 8
10.8.1.2:56286 31.9.10.169:443 ESTABLISHED
10.8.1.2:44456 81.217.149.89:443 ESTABLISHED
10.0.0.2:51832 10.0.0.5:1692 CLOSE_WAIT
10.0.0.2:40222 10.0.0.5:1989 CLOSE_WAIT
...
~~~
## bult-in log file comparison
This tool allows to detect log files which grows faster during high-la event. It accepts glob patterns (like `/var/log/apache2/*.log`, more than one pattern are allowed), measure size at first run, waits some time (`-w`, default it 5 seconds), measures size again, and prints 5 most fast growing log files and last line from each of them:
~~~shell
highla 0.01 -w 60 --size /var/log/apache2/*log
...
... waiting 60 seconds before 2nd part of diagnostics
## File sizes difference
/var/log/apache2/example.com-access.log sz: 57312780 diff: 36997
109.118.67.166 - - [12/Nov/2024:20:13:45 +0100] "GET /favicon.ico HTTP/1.1" 200 10225 "https://www.example.com/" "Mozilla/5.0 (iPhone; CPU iPhone OS 17_6_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.6 Mobile/15E148 Safari/604.1"
/var/log/apache2/example.org-access.log sz: 5176826 diff: 4911
66.249.76.230 - - [12/Nov/2024:20:13:47 +0100] "GET /myscript.php?ide=2283 HTTP/1.1" 200 12352 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.6723.116 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
...
~~~
Here we see example.com access log grows faster then others and we see last log line for further manual investigation.
## External diagnostic script (e.g. to see database queries)
If option `--script /path/script.sh` given, highla will execute this script and print it's stdout.
Example script:
~~~
#!/bin/sh
mysql -u root -pNotMyRealPassword -e "SHOW FULL PROCESSLIST"
~~~
Example output:
~~~shell
## User script /root/processlist.sh
+---------+------+-----------+------+---------+------+----------+-----------------------+----------+
| Id | User | Host | db | Command | Time | State | Info | Progress |
+---------+------+-----------+------+---------+------+----------+-----------------------+----------+
| 3568874 | root | localhost | NULL | Query | 0 | starting | SHOW FULL PROCESSLIST | 0.000 |
+---------+------+-----------+------+---------+------+----------+-----------------------+----------+
~~~
## Running in loop
You can start highla in tmux/screen background session or as systemd service running in loop.
In example below we run it in a loop, checking LA every 60 seconds, if if LA>5, it will print top-10 processes, TCP connections, fast-growing apache log files and custom script which will show mariadb processlist. Results will be printed to stdout and file highla.log
~~~
highla 5 -s /root/processlist.sh -w 10 --loop 60 --top 10 --tcp --size /var/log/apache2/*log | tee highla.log
~~~
Raw data
{
"_id": null,
"home_page": null,
"name": "highla",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "ddos, diagnostic, load, load average, load-average, mariadb, monitoring, mysql, netstat, performance, postgresql, top, uptime, usage",
"author": null,
"author_email": "Yaroslav Polyakov <yaroslaff@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/a4/ed/f33dd2e841fc43e38063ca288729a457e9cccc64aa579c48a662b75ed957/highla-0.1.2.tar.gz",
"platform": null,
"description": "# HighLA - Run diagnostics on high load average\n\nHighla checks current load average and executes set of built-in diagnostic or external script to collect data at the precise moment when load average is high.\n\n## Install\n~~~shell\n# pipx way, recommended\npipx install highla\n\n# pip3 way, obsolete. recommended to use in virtualenv\npip3 install highla\n~~~\n\n## Empty diag\nhighla has one required parameter - Load Average threshold. If current Load Average is lower than threshold, it will do nothing. If current Load Average is higher, it will run diagnostic specified in other arguments.\n\n~~~shell\n# no diag at all\n\n# trigger if LA over 20 (current LA is lower, so nothing printed)\n$ highla 20\n\n# trigger if LA over 3.5 (prints timestamp and current LA on stdout)\n$ highla 3.5\n# 2024/11/13 01:29:01 LA: 4.05\n~~~\n\n## built-in top \nprints top-N CPU-intensive processes\n\n~~~shell\n$ highla 1 --top 5\n# 2024/11/13 01:35:19 LA: 3.69\n## Top CPU usage\n 792682 wireguard-vanity-keygen 307.9%\n 877929 highla 2.0%\n 1192 irq/33-nvidia 1.0%\n 2158 asterisk 1.0%\n 3767 ibus-portal 1.0%\n\n~~~\n\n## built-in netstat\n\nPrints summary and all connections in ESTABLISHED/CLOSE_WAIT states\n~~~shell\n$ highla 1 --tcp\n# 2024/11/13 01:37:32 LA: 3.47\n## TCP connections: ESTABLISHED: 13, LISTEN: 41, CLOSE_WAIT: 8\n10.8.1.2:56286 31.9.10.169:443 ESTABLISHED\n10.8.1.2:44456 81.217.149.89:443 ESTABLISHED\n10.0.0.2:51832 10.0.0.5:1692 CLOSE_WAIT\n10.0.0.2:40222 10.0.0.5:1989 CLOSE_WAIT\n...\n~~~\n\n## bult-in log file comparison\nThis tool allows to detect log files which grows faster during high-la event. It accepts glob patterns (like `/var/log/apache2/*.log`, more than one pattern are allowed), measure size at first run, waits some time (`-w`, default it 5 seconds), measures size again, and prints 5 most fast growing log files and last line from each of them:\n\n~~~shell\nhighla 0.01 -w 60 --size /var/log/apache2/*log\n...\n... waiting 60 seconds before 2nd part of diagnostics\n## File sizes difference\n/var/log/apache2/example.com-access.log sz: 57312780 diff: 36997\n 109.118.67.166 - - [12/Nov/2024:20:13:45 +0100] \"GET /favicon.ico HTTP/1.1\" 200 10225 \"https://www.example.com/\" \"Mozilla/5.0 (iPhone; CPU iPhone OS 17_6_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.6 Mobile/15E148 Safari/604.1\"\n/var/log/apache2/example.org-access.log sz: 5176826 diff: 4911\n 66.249.76.230 - - [12/Nov/2024:20:13:47 +0100] \"GET /myscript.php?ide=2283 HTTP/1.1\" 200 12352 \"-\" \"Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.6723.116 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\"\n...\n~~~\nHere we see example.com access log grows faster then others and we see last log line for further manual investigation.\n\n## External diagnostic script (e.g. to see database queries)\nIf option `--script /path/script.sh` given, highla will execute this script and print it's stdout.\n\nExample script:\n~~~\n#!/bin/sh\nmysql -u root -pNotMyRealPassword -e \"SHOW FULL PROCESSLIST\"\n~~~\n\nExample output:\n~~~shell\n## User script /root/processlist.sh\n+---------+------+-----------+------+---------+------+----------+-----------------------+----------+\n| Id | User | Host | db | Command | Time | State | Info | Progress |\n+---------+------+-----------+------+---------+------+----------+-----------------------+----------+\n| 3568874 | root | localhost | NULL | Query | 0 | starting | SHOW FULL PROCESSLIST | 0.000 |\n+---------+------+-----------+------+---------+------+----------+-----------------------+----------+\n~~~\n\n## Running in loop\nYou can start highla in tmux/screen background session or as systemd service running in loop.\n\nIn example below we run it in a loop, checking LA every 60 seconds, if if LA>5, it will print top-10 processes, TCP connections, fast-growing apache log files and custom script which will show mariadb processlist. Results will be printed to stdout and file highla.log\n\n~~~\nhighla 5 -s /root/processlist.sh -w 10 --loop 60 --top 10 --tcp --size /var/log/apache2/*log | tee highla.log\n~~~\n",
"bugtrack_url": null,
"license": null,
"summary": "Run diagnostics on high Load Average",
"version": "0.1.2",
"project_urls": {
"Homepage": "https://github.com/yaroslaff/highla",
"Issues": "https://github.com/yaroslaff/highla/issues"
},
"split_keywords": [
"ddos",
" diagnostic",
" load",
" load average",
" load-average",
" mariadb",
" monitoring",
" mysql",
" netstat",
" performance",
" postgresql",
" top",
" uptime",
" usage"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "34ffe104970938b9eda1c7ee3db936977f39d8d91793c74108199e5becb21ff3",
"md5": "381717de2263630bfbf2c94dcacdb6d1",
"sha256": "505cab5042cb9c4dcb79125e1d86496426c4a5a5a4e74eecd4dfba6d54bcc246"
},
"downloads": -1,
"filename": "highla-0.1.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "381717de2263630bfbf2c94dcacdb6d1",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 6217,
"upload_time": "2024-11-13T09:59:49",
"upload_time_iso_8601": "2024-11-13T09:59:49.272453Z",
"url": "https://files.pythonhosted.org/packages/34/ff/e104970938b9eda1c7ee3db936977f39d8d91793c74108199e5becb21ff3/highla-0.1.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "a4edf33dd2e841fc43e38063ca288729a457e9cccc64aa579c48a662b75ed957",
"md5": "1f8a69ea5e41440a1f6eba32c5e7d6ac",
"sha256": "e783cc2e666dcd6516feb1386c4aaadefee6a38b09b9cb5549e59273594d3fa3"
},
"downloads": -1,
"filename": "highla-0.1.2.tar.gz",
"has_sig": false,
"md5_digest": "1f8a69ea5e41440a1f6eba32c5e7d6ac",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 6629,
"upload_time": "2024-11-13T09:59:46",
"upload_time_iso_8601": "2024-11-13T09:59:46.394113Z",
"url": "https://files.pythonhosted.org/packages/a4/ed/f33dd2e841fc43e38063ca288729a457e9cccc64aa579c48a662b75ed957/highla-0.1.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-13 09:59:46",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "yaroslaff",
"github_project": "highla",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "highla"
}
JSON
