| Name | logmerger JSON |
| Version |
0.7.0
JSON |
| download |
| home_page | |
| Summary | A command-line utility to display multiple log files, merged by log timestamp |
| upload_time | 2023-10-08 04:25:55 |
| maintainer | |
| docs_url | None |
| author | Paul McGuire |
| requires_python | >=3.9 |
| license | |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# logmerger
`logmerger` is a TUI for viewing a merged display of multiple log files, merged by timestamp.
Given these two log files:
```
# log1.txt
2023-07-14 08:00:01 WARN Connection lost due to timeout
2023-07-14 08:00:04 ERROR Request processed unsuccessfully
Something went wrong
Traceback (last line is latest):
blah
blah
ValueError("shouldn't have done that")
2023-07-14 08:00:06 INFO User authentication succeeded
2023-07-14 08:00:08 DEBUG Starting data synchronization
2023-07-14 08:00:11 INFO Processing incoming request
```
```
# log2.txt
2023-07-14 08:00:01 INFO Request processed successfully
2023-07-14 08:00:03 INFO User authentication succeeded
2023-07-14 08:00:06 DEBUG Starting data synchronization
2023-07-14 08:00:08 INFO Processing incoming request
2023-07-14 08:00:11 DEBUG Performing database backup
2023-07-14 08:00:14 WARN Invalid input received: missing required field
```
This command
logmerger log1.txt log2.txt
Shows the following browsable merged display (enabled using the [textual](https://textual.textualize.io) Python library):
Use `--output -` to send the merged logs to stdout:
```
Timestamp Files/Log1.Txt Files/Log2.Txt
────────────────────────────────────────────────────────────────────────────────────────────────────
2023-07-14 08:00:01.000 WARN Connection lost due to INFO Request processed
timeout successfully
2023-07-14 08:00:03.000 INFO User authentication
succeeded
2023-07-14 08:00:04.000 ERROR Request processed
unsuccessfully
Something went wrong
Traceback (last line is latest):
blah
blah
ValueError("shouldn't have done
that")
2023-07-14 08:00:06.000 INFO User authentication DEBUG Starting data
succeeded synchronization
2023-07-14 08:00:08.000 DEBUG Starting data INFO Processing incoming request
synchronization
2023-07-14 08:00:11.000 INFO Processing incoming request DEBUG Performing database backup
INFO Processing incoming request
(a little more...)
2023-07-14 08:00:14.000 DEBUG Performing database backup WARN Invalid input received:
missing required field
```
## Installation
Install `logmerger` from PyPI:
pip install logmerger
This will install `logmerger` as a shell/console command, so you can then run it directly without
invoking `python`.
To add support for merging pcap files, install using:
pip install logmerger[pcap]
## Command line arguments
`logmerger -h` will show the following help:
```
usage: logmerger [-h] [--interactive] [--output OUTPUT] [--start START] [--end END] [--width WIDTH]
[--line_numbers] [--csv CSV] [--encoding ENCODING]
[--timestamp_format [TIMESTAMP_FORMATS ...]] [--demo]
[files ...]
positional arguments:
files log files to be merged
options:
-h, --help show this help message and exit
--interactive, -i show merged output using interactive TUI browser (default)
--output OUTPUT, -o OUTPUT
save merged output to file ('-' for stdout; files ending in '.md' are saved
using Markdown)
--start START, -s START
start time to select time window for merging logs
--end END, -e END end time to select time window for merging logs
--width WIDTH, -w WIDTH
total screen width to use for interactive mode (defaults to current screen
width)
--line_numbers, -ln add line number column
--csv CSV, -csv CSV save merged logs to CSV file
--encoding ENCODING, -enc ENCODING
encoding to use when reading log files (defaults to the system default encoding)
--timestamp_format [TIMESTAMP_FORMATS ...]
custom timestamp format
--demo Run interactive demo
Start and end timestamps to clip the given files to a particular time window can be
given in `YYYY-MM-DD HH:MM:SS.SSS` format, with trailing milliseconds and seconds
optional, and "," permissible for the decimal point. A "T" can be included between
the date and time to simplify entering the timestamp on a command line (otherwise
would require enclosing in quotes because of the intervening space). These command
line values do not need to match the timestamp formats in the log files.
These values may also be given as relative times, such as "15m" for "15 minutes ago".
Valid units are "s", "m", "h", and "d".
```
## Supported file types
Log data is usually extracted from text log files, but can also be extracted
from other log related files.
| type | |
|------------------------------------------------------------------------------------|----------------------------|
| text log files | any file name ending |
| text log files that have been gzip compressed (such as those created by logrotate) | filename ending in `.gz` |
| CSV files (timestamp is read from first data column) | filename ending in `.csv` |
| packet capture files (experimental) | filename ending in `.pcap` |
## Merging
Log files get merged by interleaving log lines from each based on timestamps in each log line. `logmerger` tries to
use different timestamp formats until it finds a matching format for each input file. The supported formats are:
| format | description |
|------------------------------|-----------------------------------------------------------------------------------------------------|
| `YYYY-MM-DD HH:MM:SS,SSS` | date+time to milliseconds, with ',' decimal (default for Python's `asctime` log marker) |
| `YYYY-MM-DD HH:MM:SS.SSS` | date+time to milliseconds, with '.' decimal |
| `YYYY-MM-DD HH:MM:SS` | date+time to seconds |
| `YYYY-MM-DDTHH:MM:SS,SSS` | date+T+time to milliseconds, with ',' decimal |
| `YYYY-MM-DDTHH:MM:SS.SSS` | date+T+time to milliseconds, with '.' decimal |
| `YYYY-MM-DDTHH:MM:SS` | date+T+time to seconds |
| `Jan DD HH:MM:SS` | month/day + time (timestamp in syslog files); year is inferred from the create date of the log file |
| `DD/Jan/YYYY HH:MM:SS` | day/month/year + time |
| `DD/Jan/YYYY:HH:MM:SS ±ZZZZ` | day/month/year + time + timezone offset (converts timestamps to local time) |
Untimestamped log lines that contain multiple lines (such as a traceback) get combined with the previous timestamped
line (see in the example above).
Raw data
{
"_id": null,
"home_page": "",
"name": "logmerger",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": "",
"keywords": "",
"author": "Paul McGuire",
"author_email": "ptmcg.pm@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/43/df/5826e394bc193a9c68b16c2099ef579eaca91d10d9e9751fc7d6ab99baa7/logmerger-0.7.0.tar.gz",
"platform": null,
"description": "# logmerger\n\n`logmerger` is a TUI for viewing a merged display of multiple log files, merged by timestamp.\n\nGiven these two log files:\n\n```\n# log1.txt\n2023-07-14 08:00:01 WARN Connection lost due to timeout\n2023-07-14 08:00:04 ERROR Request processed unsuccessfully\nSomething went wrong\nTraceback (last line is latest):\n blah\n blah\nValueError(\"shouldn't have done that\")\n2023-07-14 08:00:06 INFO User authentication succeeded\n2023-07-14 08:00:08 DEBUG Starting data synchronization\n2023-07-14 08:00:11 INFO Processing incoming request\n```\n\n```\n# log2.txt\n2023-07-14 08:00:01 INFO Request processed successfully\n2023-07-14 08:00:03 INFO User authentication succeeded\n2023-07-14 08:00:06 DEBUG Starting data synchronization\n2023-07-14 08:00:08 INFO Processing incoming request\n2023-07-14 08:00:11 DEBUG Performing database backup\n2023-07-14 08:00:14 WARN Invalid input received: missing required field\n```\nThis command\n\n logmerger log1.txt log2.txt\n\nShows the following browsable merged display (enabled using the [textual](https://textual.textualize.io) Python library):\n\n\n\nUse `--output -` to send the merged logs to stdout:\n\n```\n Timestamp Files/Log1.Txt Files/Log2.Txt\n \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n 2023-07-14 08:00:01.000 WARN Connection lost due to INFO Request processed\n timeout successfully\n 2023-07-14 08:00:03.000 INFO User authentication\n succeeded\n 2023-07-14 08:00:04.000 ERROR Request processed\n unsuccessfully\n Something went wrong\n Traceback (last line is latest):\n blah\n blah\n ValueError(\"shouldn't have done\n that\")\n 2023-07-14 08:00:06.000 INFO User authentication DEBUG Starting data\n succeeded synchronization\n 2023-07-14 08:00:08.000 DEBUG Starting data INFO Processing incoming request\n synchronization\n 2023-07-14 08:00:11.000 INFO Processing incoming request DEBUG Performing database backup\n INFO Processing incoming request\n (a little more...)\n 2023-07-14 08:00:14.000 DEBUG Performing database backup WARN Invalid input received:\n missing required field\n```\n\n## Installation\n\nInstall `logmerger` from PyPI:\n\n pip install logmerger\n\nThis will install `logmerger` as a shell/console command, so you can then run it directly without\ninvoking `python`.\n\nTo add support for merging pcap files, install using:\n\n pip install logmerger[pcap]\n\n\n## Command line arguments\n\n`logmerger -h` will show the following help:\n\n```\nusage: logmerger [-h] [--interactive] [--output OUTPUT] [--start START] [--end END] [--width WIDTH]\n [--line_numbers] [--csv CSV] [--encoding ENCODING]\n [--timestamp_format [TIMESTAMP_FORMATS ...]] [--demo]\n [files ...]\n\npositional arguments:\n files log files to be merged\n\noptions:\n -h, --help show this help message and exit\n --interactive, -i show merged output using interactive TUI browser (default)\n --output OUTPUT, -o OUTPUT\n save merged output to file ('-' for stdout; files ending in '.md' are saved\n using Markdown)\n --start START, -s START\n start time to select time window for merging logs\n --end END, -e END end time to select time window for merging logs\n --width WIDTH, -w WIDTH\n total screen width to use for interactive mode (defaults to current screen\n width)\n --line_numbers, -ln add line number column\n --csv CSV, -csv CSV save merged logs to CSV file\n --encoding ENCODING, -enc ENCODING\n encoding to use when reading log files (defaults to the system default encoding)\n --timestamp_format [TIMESTAMP_FORMATS ...]\n custom timestamp format\n --demo Run interactive demo\n \nStart and end timestamps to clip the given files to a particular time window can be\ngiven in `YYYY-MM-DD HH:MM:SS.SSS` format, with trailing milliseconds and seconds\noptional, and \",\" permissible for the decimal point. A \"T\" can be included between\nthe date and time to simplify entering the timestamp on a command line (otherwise\nwould require enclosing in quotes because of the intervening space). These command\nline values do not need to match the timestamp formats in the log files.\n\nThese values may also be given as relative times, such as \"15m\" for \"15 minutes ago\".\nValid units are \"s\", \"m\", \"h\", and \"d\".\n```\n\n\n## Supported file types\n\nLog data is usually extracted from text log files, but can also be extracted\nfrom other log related files.\n\n| type | |\n|------------------------------------------------------------------------------------|----------------------------|\n| text log files | any file name ending |\n| text log files that have been gzip compressed (such as those created by logrotate) | filename ending in `.gz` |\n| CSV files (timestamp is read from first data column) | filename ending in `.csv` |\n| packet capture files (experimental) | filename ending in `.pcap` |\n\n\n## Merging\n\nLog files get merged by interleaving log lines from each based on timestamps in each log line. `logmerger` tries to \nuse different timestamp formats until it finds a matching format for each input file. The supported formats are:\n\n| format | description |\n|------------------------------|-----------------------------------------------------------------------------------------------------|\n| `YYYY-MM-DD HH:MM:SS,SSS` | date+time to milliseconds, with ',' decimal (default for Python's `asctime` log marker) |\n| `YYYY-MM-DD HH:MM:SS.SSS` | date+time to milliseconds, with '.' decimal |\n| `YYYY-MM-DD HH:MM:SS` | date+time to seconds |\n| `YYYY-MM-DDTHH:MM:SS,SSS` | date+T+time to milliseconds, with ',' decimal |\n| `YYYY-MM-DDTHH:MM:SS.SSS` | date+T+time to milliseconds, with '.' decimal |\n| `YYYY-MM-DDTHH:MM:SS` | date+T+time to seconds |\n| `Jan DD HH:MM:SS` | month/day + time (timestamp in syslog files); year is inferred from the create date of the log file |\n| `DD/Jan/YYYY HH:MM:SS` | day/month/year + time |\n| `DD/Jan/YYYY:HH:MM:SS \u00b1ZZZZ` | day/month/year + time + timezone offset (converts timestamps to local time) |\n\n\nUntimestamped log lines that contain multiple lines (such as a traceback) get combined with the previous timestamped\nline (see in the example above).\n",
"bugtrack_url": null,
"license": "",
"summary": "A command-line utility to display multiple log files, merged by log timestamp",
"version": "0.7.0",
"project_urls": {
"Source": "https://github.com/ptmcg/logmerger"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2467ee47e051b0d6548079f333c7908e6606cb971a3e29eb71bbc3a231d9d81c",
"md5": "563ea0234ff696ed48b2283eae5bc1c1",
"sha256": "6afe4d8195d6d21d9e53447364b1d2dcdb40eb0969d8df21eaa347013a1fc9a4"
},
"downloads": -1,
"filename": "logmerger-0.7.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "563ea0234ff696ed48b2283eae5bc1c1",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 26115,
"upload_time": "2023-10-08T04:25:53",
"upload_time_iso_8601": "2023-10-08T04:25:53.489555Z",
"url": "https://files.pythonhosted.org/packages/24/67/ee47e051b0d6548079f333c7908e6606cb971a3e29eb71bbc3a231d9d81c/logmerger-0.7.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "43df5826e394bc193a9c68b16c2099ef579eaca91d10d9e9751fc7d6ab99baa7",
"md5": "ce4dd0bf49090d222a08897dbbc8802f",
"sha256": "afd1ad5b947115fb17e5421b4885cc2ca0ec9df672bb3ffa88b29a7aaf925477"
},
"downloads": -1,
"filename": "logmerger-0.7.0.tar.gz",
"has_sig": false,
"md5_digest": "ce4dd0bf49090d222a08897dbbc8802f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 24362,
"upload_time": "2023-10-08T04:25:55",
"upload_time_iso_8601": "2023-10-08T04:25:55.173899Z",
"url": "https://files.pythonhosted.org/packages/43/df/5826e394bc193a9c68b16c2099ef579eaca91d10d9e9751fc7d6ab99baa7/logmerger-0.7.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-10-08 04:25:55",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ptmcg",
"github_project": "logmerger",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "logmerger"
}
