# What?
`wrrarms` (`pwebarc-wrrarms`) is a tool for displaying, programmatically manipulating, organizing, importing, and exporting [Personal Private Passive Web Archive (pwebarc)](https://github.com/Own-Data-Privateer/pwebarc/) (also [there](https://oxij.org/software/pwebarc/)) Web Request+Response (WRR) files produced by [pWebArc browser extension](https://github.com/Own-Data-Privateer/pwebarc/tree/master/extension/) (also [there](https://oxij.org/software/pwebarc/tree/master/extension/)).
# Quickstart
## Installation
- Install with:
```bash
pip install pwebarc-wrrarms
```
and run as
```bash
wrrarms --help
```
- Alternatively, install it via Nix
```bash
nix-env -i -f ./default.nix
wrrarms --help
```
- Alternatively, run without installing:
```bash
alias wrrarms="python3 -m wrrarms"
wrrarms --help
```
## How to build a file system tree of latest versions of all hoarded URLs
Assuming you keep your WRR dumps in `~/pwebarc/raw` you can generate a hierarchy of symlinks for each URL pointing from under `~/pwebarc/latest` to the most recent WRR file in `~/pwebarc/raw` via:
```bash
wrrarms organize --symlink --latest --output hupq --to ~/pwebarc/latest --and "status|== 200C" ~/pwebarc/raw
```
Personally, I prefer `flat_mhs` (see the documentation of the `--output` below) format as I dislike deep file hierarchies, using it also simplifies filtering in my `ranger` file browser, so I do this:
```bash
wrrarms organize --symlink --latest --output flat_mhs --to ~/pwebarc/latest --and "status|== 200C" ~/pwebarc/raw
```
These commands rescan the whole of `~/pwebarc/raw` and so take a while to complete.
If you have a lot of WRR files and you want to keep your symlink tree updated in real-time you can use a two-stage `--stdin0` pipeline shown in the [examples section](#examples) below.
## <span id="mirror"/>How to generate a local offline website mirror like `wget -mpk`
If you want to render your WRR files into a local offline website mirror containing interlinked HTML files and their resources a-la `wget -mpk` (`wget --mirror --page-requisites --convert-links`), run one of the above `--symlink --latest` command, and then do something like this:
```bash
wrrarms export mirror --to ~/pwebarc/mirror1 ~/pwebarc/latest/archiveofourown.org
```
on completion `~/pwebarc/mirror1` will contain a bunch of interlinked minimized HTML files, their resources, and any other files they link to.
By default, *all* the links in exported HTML files will be remapped to local files (even if source WRR files for those would-be exported files are missing in `~/pwebarc/latest/archiveofourown.org`), and those HTML files will also be stripped of all JavaScript, CSS, and other stuff of various levels of evil (see documentation for the `scrub` function below).
On the plus side, the result will be completely self-contained and safe to view with a dumb unconfigured browser.
If you are unhappy with this behaviour and, for instance, want to keep the CSS and produce human-readable HTML, run the following instead:
```bash
wrrarms export mirror -e 'response.body|eb|scrub response +all_refs,-actions,+styles,+pretty' --to ~/pwebarc/mirror2 ~/pwebarc/latest/archiveofourown.org
```
Note, however, that CSS resource filtering and remapping is not implemented yet.
If you also want to keep links that point to not yet hoarded Internet URLs to still point those URLs in the exported files instead of them pointing to non-existent local files, similarly to what `wget -mpk` does, run:
```bash
wrrarms export mirror -e 'response.body|eb|scrub response +all_refs,-actions,+styles,+pretty' --remap-open --to ~/pwebarc/mirror3 ~/pwebarc/latest/archiveofourown.org
```
Finally, if you want a mirror made of raw files without any content censorship or link conversions, run:
```bash
wrrarms export mirror -e 'response.body|eb' --to ~/pwebarc/mirror-raw ~/pwebarc/latest/archiveofourown.org
```
The later command will render your mirror pretty quick, but the other above-mentioned commands will call the `scrub` function, and that will be pretty slow (as in avg ~5Mb, ~3 files per second on my 2013-era laptop), mostly because `html5lib` that `wrrarms` uses for paranoid HTML parsing and filtering is fairly slow.
## How to generate previews for WRR files, listen to them via TTS, open them with `xdg-open`, etc
See [`script` sub-directory](./script/README.md) for examples that show how to use `pandoc` and/or `w3m` to turn WRR files into previews and readable plain-text that can viewed or listened to via other tools, or dump them into temporary raw data files that can then be immediately fed to `xdg-open` for one-click viewing.
# <span id="todo"/>What is left TODO
- Converters from HAR and WARC to WRR.
- Data de-duplication between different WRR files.
- Non-dumb server with time+URL index and replay, i.e. a local [Wayback Machine](https://web.archive.org/).
- Full text indexing and search.
- Converter from WRR to WARC.
- Converter from PCAP ito WRR.
# Usage
## wrrarms
A tool to pretty-print, compute and print values from, search, organize (programmatically rename/move/symlink/hardlink files), import, export, (WIP: check, deduplicate, and edit) pWebArc WRR (WEBREQRES, Web REQuest+RESponse) archive files.
Terminology: a `reqres` (`Reqres` when a Python type) is an instance of a structure representing HTTP request+response pair with some additional metadata.
- options:
- `--version`
: show program's version number and exit
- `-h, --help`
: show this help message and exit
- `--markdown`
: show help messages formatted in Markdown
- subcommands:
- `{pprint,get,run,stream,find,organize,import,export}`
- `pprint`
: pretty-print given WRR files
- `get`
: print values produced by computing given expressions on a given WRR file
- `run`
: spawn a process with generated temporary files produced by given expressions computed on given WRR files as arguments
- `stream`
: produce a stream of structured lists containing values produced by computing given expressions on given WRR files, a generalized `wrrarms get`
- `find`
: print paths of WRR files matching specified criteria
- `organize`
: programmatically rename/move/hardlink/symlink WRR files based on their contents
- `import`
: convert other HTTP archive formats into WRR
- `export`
: convert WRR archives into other formats
### wrrarms pprint
Pretty-print given WRR files to stdout.
- positional arguments:
- `PATH`
: inputs, can be a mix of files and directories (which will be traversed recursively)
- options:
- `-u, --unabridged`
: print all data in full
- `--abridged`
: shorten long strings for brevity (useful when you want to visually scan through batch data dumps) (default)
- `--stdin0`
: read zero-terminated `PATH`s from stdin, these will be processed after `PATH`s specified as command-line arguments
- error handling:
- `--errors {fail,skip,ignore}`
: when an error occurs:
- `fail`: report failure and stop the execution (default)
- `skip`: report failure but skip the reqres that produced it from the output and continue
- `ignore`: `skip`, but don't report the failure
- filters:
- `--or EXPR`
: only print reqres which match any of these expressions...
- `--and EXPR`
: ... and all of these expressions, both can be specified multiple times, both use the same expression format as `wrrarms get --expr`, which see
- MIME type sniffing:
- `--naive`
: populate "potentially" lists like `wrrarms (get|run|export) --expr '(request|response).body|eb|scrub \2 defaults'` does; default
- `--paranoid`
: populate "potentially" lists in the output using paranoid MIME type sniffing like `wrrarms (get|run|export) --expr '(request|response).body|eb|scrub \2 +paranoid'` does; this exists to answer "Hey! Why did it censor out my data?!" questions
- file system path ordering:
- `--paths-given-order`
: `argv` and `--stdin0` `PATH`s are processed in the order they are given (default)
- `--paths-sorted`
: `argv` and `--stdin0` `PATH`s are processed in lexicographic order
- `--paths-reversed`
: `argv` and `--stdin0` `PATH`s are processed in reverse lexicographic order
- `--walk-fs-order`
: recursive file system walk is done in the order `readdir(2)` gives results
- `--walk-sorted`
: recursive file system walk is done in lexicographic order (default)
- `--walk-reversed`
: recursive file system walk is done in reverse lexicographic order
### wrrarms get
Compute output values by evaluating expressions `EXPR`s on a given reqres stored at `PATH`, then print them to stdout terminating each value as specified.
- positional arguments:
- `PATH`
: input WRR file path
- expression evaluation:
- `-e EXPR, --expr EXPR`
: an expression to compute; can be specified multiple times in which case computed outputs will be printed sequentially; see also "output" options below; (default: `response.body|eb`); each EXPR describes a state-transformer (pipeline) which starts from value `None` and evaluates a script built from the following:
- constants and functions:
- `es`: replace `None` value with an empty string `""`
- `eb`: replace `None` value with an empty byte string `b""`
- `false`: replace `None` value with `False`
- `true`: replace `None` value with `True`
- `missing`: `True` if the value is `None`
- `0`: replace `None` value with `0`
- `1`: replace `None` value with `1`
- `not`: apply logical `not` to value
- `len`: apply `len` to value
- `str`: cast value to `str` or fail
- `bytes`: cast value to `bytes` or fail
- `bool`: cast value to `bool` or fail
- `int`: cast value to `int` or fail
- `float`: cast value to `float` or fail
- `echo`: replace the value with the given string
- `quote`: URL-percent-encoding quote value
- `quote_plus`: URL-percent-encoding quote value and replace spaces with `+` symbols
- `unquote`: URL-percent-encoding unquote value
- `unquote_plus`: URL-percent-encoding unquote value and replace `+` symbols with spaces
- `to_ascii`: encode `str` value into `bytes` with "ascii" codec
- `to_utf8`: encode `str` value into `bytes` with "utf-8" codec
- `sha256`: replace `bytes` value with its `sha256` hex digest (`hex(sha256(value))`)
- `==`: apply `== arg`, `arg` is cast to the same type as the current value
- `!=`: apply `!= arg`, similarly
- `<`: apply `< arg`, similarly
- `<=`: apply `<= arg`, similarly
- `>`: apply `> arg`, similarly
- `>=`: apply `>= arg`, similarly
- `add_prefix`: add prefix to the current value
- `add_suffix`: add suffix to the current value
- `take_prefix`: take first `arg` characters or list elements from the current value
- `take_suffix`: take last `arg` characters or list elements from the current value
- `abbrev`: leave the current value as-is if if its length is less or equal than `arg` characters, otherwise take first `arg/2` followed by last `arg/2` characters
- `abbrev_each`: `abbrev arg` each element in a value `list`
- `replace`: replace all occurences of the first argument in the current value with the second argument, casts arguments to the same type as the current value
- `pp_to_path`: encode `path_parts` `list` into a POSIX path, quoting as little as needed
- `qsl_urlencode`: encode parsed `query` `list` into a URL's query component `str`
- `qsl_to_path`: encode `query` `list` into a POSIX path, quoting as little as needed
- `scrub`: scrub the value by optionally rewriting links and/or removing dynamic content from it; what gets done depends on `--remap-*` command line options, the MIME type of the value itself, and the scrubbing options described below; this fuction takes two arguments:
- the first must be either of `request|response`, it controls which HTTP headers `scrub` should inspect to help it detect the MIME type;
- the second is either `defaults` or ","-separated string of `(+|-)(paranoid|unknown|jumps|actions|srcs|all_refs|scripts|iframes|styles|iepragmas|prefetches|tracking|dyndoc|all_dyns|verbose|whitespace|optional_tags|indent|pretty|debug)` tokens which control the scrubbing behaviour:
- `+paranoid` will assume the server is lying in its `Content-Type` and `X-Content-Type-Options` HTTP headers, sniff the contents of `(request|response).body` to determine what it actually contains regardless of what the server said, and then use the most paranoid interpretation of both the HTTP headers and the sniffed possible MIME types to decide what should be kept and what sholuld be removed by the options below; i.e., this will make `-unknown`, `-scripts`, and `-styles` options below to censor out more things, in particular, at the moment, most plain text files will get censored out as potential JavaScript; the default is `-paranoid`;
- `(+|-)unknown` controls if the data with unknown content types should passed to the output unchanged or censored out (respectively); the default is `+unknown`, which will keep data of unknown content types as-is;
- `(+|-)(jumps|actions|srcs)` control which kinds of references to other documents should be remapped or censored out (respectively); i.e. it controls whether jump-links (HTML `a href`, `area href`, and similar), action-links (HTML `a ping`, `form action`, and similar), and/or resource references (HTML `img src`, `iframe src`, CSS `url` references, and similar) should be remapped using the specified `--remap-*` option (which see) or censored out similarly to how `--remap-void` will do it; the default is `+jumps,-actions,-srcs` which will produce a self-contained result that can be fed into another tool --- be it a web browser or `pandoc` --- without that tool trying to access the Internet;
- `(+|-)all_refs` is equivalent to enabling or disabling all of the above options simultaneously;
- `(+|-)(scripts|iframes|styles|iepragmas|prefetches|tracking)` control which things should be kept or censored out w.r.t. to HTML, CSS, and JavaScript, i.e. it controls whether JavaScript (both separate files and HTML tags and attributes), `<iframe>` HTML tags, CSS (both separate files and HTML tags and attributes; why? because CSS is Turing-complete), HTML Internet-Explorer pragmas, HTML content prefetch `link` tags, and other tracking HTML tags and attributes (like `a ping` attributes), should be respectively kept in or censored out from the input; the default is `-scripts,-iframes,-styles,-iepragmas,-prefetches,-tracking` which ensures the result will not produce any prefetch and tracking requests when loaded in a web browser, and that the whole result is simple data, not a program in some Turing-complete language, thus making it safe to feed the result to other tools too smart for their own users' good;
- `(+|-)all_dyns` is equivalent to enabling or disabling all of the above (`scripts|...`) options simultaneously;
- `(+|-)verbose` controls whether tag censoring controlled by the above options is to be reported in the output (as comments) or stuff should be wiped from existence without evidence instead; the default is `-verbose`;
- `(+|-)whitespace` controls whether HTML renderer should keep the original HTML whitespace as-is or collapse it away (respectively); the default is `-whitespace`;
- `(+|-)optional_tags` controls whether HTML renderer should put optional HTML tags into the output or skip them (respectively); the default is `+optional_tags` (because many tools fail to parse minimized HTML properly);
- `(+|-)indent` controls whether HTML renderer should indent HTML elements (where whitespace placement in the original markup allows for it) or not (respectively); the default is `-indent`;
- `+pretty` is an alias for `+verbose,-whitespace,+indent` which produces the prettiest possible human-readable output that keeps the original whitespace semantics; `-pretty` is an alias for `+verbose,+whitespace,-indent` which produces the approximation of the original markup with censoring applied; neither is the default;
- `+debug` is an alias for `+pretty` that also uses a much more aggressive version of `indent` that ignores the semantics of original whitespace placement, i.e. it will indent `<p>not<em>sep</em>arated</p>` as if there was whitespace before and after `p`, `em`, `/em`, and `/p` tags; this is useful for debugging custom mutations; `-debug` is noop, which is the default;
- reqres fields, these work the same way as constants above, i.e. they replace current value of `None` with field's value, if reqres is missing the field in question, which could happen for `response*` fields, the result is `None`:
- `version`: WEBREQRES format version; int
- `source`: `+`-separated list of applications that produced this reqres; str
- `protocol`: protocol; e.g. `"HTTP/1.1"`, `"HTTP/2.0"`; str
- `request.started_at`: request start time in seconds since 1970-01-01 00:00; Epoch
- `request.method`: request HTTP method; e.g. `"GET"`, `"POST"`, etc; str
- `request.url`: request URL, including the fragment/hash part; str
- `request.headers`: request headers; list[tuple[str, bytes]]
- `request.complete`: is request body complete?; bool
- `request.body`: request body; bytes
- `response.started_at`: response start time in seconds since 1970-01-01 00:00; Epoch
- `response.code`: HTTP response code; e.g. `200`, `404`, etc; int
- `response.reason`: HTTP response reason; e.g. `"OK"`, `"Not Found"`, etc; usually empty for Chromium and filled for Firefox; str
- `response.headers`: response headers; list[tuple[str, bytes]]
- `response.complete`: is response body complete?; bool
- `response.body`: response body; Firefox gives raw bytes, Chromium gives UTF-8 encoded strings; bytes | str
- `finished_at`: request completion time in seconds since 1970-01-01 00:00; Epoch
- `websocket`: a list of WebSocket frames
- derived attributes:
- `fs_path`: file system path for the WRR file containing this reqres; str | bytes | None
- `qtime`: aliast for `request.started_at`; mnemonic: "reQuest TIME"; seconds since UNIX epoch; decimal float
- `qtime_ms`: `qtime` in milliseconds rounded down to nearest integer; milliseconds since UNIX epoch; int
- `qtime_msq`: three least significant digits of `qtime_ms`; int
- `qyear`: year number of `gmtime(qtime)` (UTC year number of `qtime`); int
- `qmonth`: month number of `gmtime(qtime)`; int
- `qday`: day of the month of `gmtime(qtime)`; int
- `qhour`: hour of `gmtime(qtime)` in 24h format; int
- `qminute`: minute of `gmtime(qtime)`; int
- `qsecond`: second of `gmtime(qtime)`; int
- `stime`: `response.started_at` if there was a response, `finished_at` otherwise; mnemonic: "reSponse TIME"; seconds since UNIX epoch; decimal float
- `stime_ms`: `stime` in milliseconds rounded down to nearest integer; milliseconds since UNIX epoch, int
- `stime_msq`: three least significant digits of `stime_msq`; int
- `syear`: similar to `syear`, but for `stime`; int
- `smonth`: similar to `smonth`, but for `stime`; int
- `sday`: similar to `sday`, but for `stime`; int
- `shour`: similar to `shour`, but for `stime`; int
- `sminute`: similar to `sminute`, but for `stime`; int
- `ssecond`: similar to `ssecond`, but for `stime`; int
- `ftime`: aliast for `finished_at`; seconds since UNIX epoch; decimal float
- `ftime_ms`: `ftime` in milliseconds rounded down to nearest integer; milliseconds since UNIX epoch; int
- `ftime_msq`: three least significant digits of `ftime_msq`; int
- `fyear`: similar to `syear`, but for `ftime`; int
- `fmonth`: similar to `smonth`, but for `ftime`; int
- `fday`: similar to `sday`, but for `ftime`; int
- `fhour`: similar to `shour`, but for `ftime`; int
- `fminute`: similar to `sminute`, but for `ftime`; int
- `fsecond`: similar to `ssecond`, but for `ftime`; int
- `status`: `"NR"` if there was no response, `str(response.code) + "C"` if response was complete, `str(response.code) + "N"` otherwise; str
- `method`: aliast for `request.method`; str
- `raw_url`: aliast for `request.url`; str
- `net_url`: `raw_url` with Punycode UTS46 IDNA encoded hostname, unsafe characters quoted, and without the fragment/hash part; this is the URL that actually gets sent to the server; str
- `pretty_url`: `raw_url`, but using `hostname`, `mq_path`, and `mq_query`; str
- `pretty_nurl`: `raw_url`, but using `hostname`, `mq_path`, and `mq_nquery`; str
- `scheme`: scheme part of `raw_url`; e.g. `http`, `https`, etc; str
- `raw_hostname`: hostname part of `raw_url` as it is recorded in the reqres; str
- `net_hostname`: hostname part of `raw_url`, encoded as Punycode UTS46 IDNA; this is what actually gets sent to the server; ASCII str
- `hostname`: `net_hostname` decoded back into UNICODE; this is the canonical hostname representation for which IDNA-encoding and decoding are bijective; UNICODE str
- `rhostname`: `hostname` with the order of its parts reversed; e.g. `"www.example.org"` -> `"com.example.www"`; str
- `port`: port part of `raw_url`; str
- `netloc`: netloc part of `raw_url`; i.e., in the most general case, `<username>:<password>@<hostname>:<port>`; str
- `raw_path`: raw path part of `raw_url` as it is recorded is the reqres; e.g. `"https://www.example.org"` -> `""`, `"https://www.example.org/"` -> `"/"`, `"https://www.example.org/index.html"` -> `"/index.html"`; str
- `path_parts`: component-wise unquoted "/"-split `raw_path` with empty components removed and dots and double dots interpreted away; e.g. `"https://www.example.org"` -> `[]`, `"https://www.example.org/"` -> `[]`, `"https://www.example.org/index.html"` -> `["index.html"]` , `"https://www.example.org/skipped/.//../used/"` -> `["used"]`; list[str]
- `mq_path`: `path_parts` turned back into a minimally-quoted string; str
- `filepath_parts`: `path_parts` transformed into components usable as an exportable file name; i.e. `path_parts` with an optional additional `"index"` appended, depending on `raw_url` and `response` MIME type; extension will be stored separately in `filepath_ext`; e.g. for HTML documents `"https://www.example.org/"` -> `["index"]`, `"https://www.example.org/test.html"` -> `["test"]`, `"https://www.example.org/test"` -> `["test", "index"]`, `"https://www.example.org/test.json"` -> `["test.json", "index"]`, but if it has a JSON MIME type then `"https://www.example.org/test.json"` -> `["test"]` (and `filepath_ext` will be set to `".json"`); this is similar to what `wget -mpk` does, but a bit smarter; list[str]
- `filepath_ext`: extension of the last component of `filepath_parts` for recognized MIME types, `".data"` otherwise; str
- `raw_query`: query part of `raw_url` (i.e. everything after the `?` character and before the `#` character) as it is recorded in the reqres; str
- `query_parts`: parsed (and component-wise unquoted) `raw_query`; list[tuple[str, str]]
- `query_ne_parts`: `query_parts` with empty query parameters removed; list[tuple[str, str]]
- `mq_query`: `query_parts` turned back into a minimally-quoted string; str
- `mq_nquery`: `query_ne_parts` turned back into a minimally-quoted string; str
- `oqm`: optional query mark: `?` character if `query` is non-empty, an empty string otherwise; str
- `fragment`: fragment (hash) part of the url; str
- `ofm`: optional fragment mark: `#` character if `fragment` is non-empty, an empty string otherwise; str
- a compound expression built by piping (`|`) the above, for example:
- `response.body|eb` (the default for `get`) will print raw `response.body` or an empty byte string, if there was no response;
- `response.body|eb|scrub response defaults` will take the above value, `scrub` it using default content scrubbing settings which will censor out all action and resource reference URLs;
- `response.body|eb|scrub response +all_refs,-actions` (the default for `export`) will remap all `href` jump-links and `src` resource references to local files while still censoring out all action URLs (since those don't make sense for a static mirror);
- `response.complete` will print the value of `response.complete` or `None`, if there was no response;
- `response.complete|false` will print `response.complete` or `False`;
- `net_url|to_ascii|sha256` will print `sha256` hash of the URL that was actually sent over the network;
- `net_url|to_ascii|sha256|take_prefix 4` will print the first 4 characters of the above;
- `path_parts|take_prefix 3|pp_to_path` will print first 3 path components of the URL, minimally quoted to be used as a path;
- `query_ne_parts|take_prefix 3|qsl_to_path|abbrev 128` will print first 3 non-empty query parameters of the URL, abbreviated to 128 characters or less, minimally quoted to be used as a path;
- URL remapping, used by `scrub` `--expr` atom:
- `--remap-id`
: remap all URLs with an identity function; i.e. don't remap anything (default)
- `--remap-void`
: remap all jump-link and action URLs to `javascript:void(0)` and all resource URLs into empty `data:` URLs; the result will be self-contained
- output:
- `--not-separated`
: don't separate output values with anything, just concatenate them
- `-l, --lf-separated`
: separate output values with `\n` (LF) newline characters (default)
- `-z, --zero-separated`
: separate output values with `\0` (NUL) bytes
### wrrarms run
Compute output values by evaluating expressions `EXPR`s for each of `NUM` reqres stored at `PATH`s, dump the results into into newly generated temporary files terminating each value as specified, spawn a given `COMMAND` with given arguments `ARG`s and the resulting temporary file paths appended as the last `NUM` arguments, wait for it to finish, delete the temporary files, exit with the return code of the spawned process.
- positional arguments:
- `COMMAND`
: command to spawn
- `ARG`
: additional arguments to give to the `COMMAND`
- `PATH`
: input WRR file paths to be mapped into new temporary files
- options:
- `-n NUM, --num-args NUM`
: number of `PATH`s (default: `1`)
- expression evaluation:
- `-e EXPR, --expr EXPR`
: see `wrrarms get`
- URL remapping, used by `scrub` `--expr` atom:
- `--remap-id`
: remap all URLs with an identity function; i.e. don't remap anything (default)
- `--remap-void`
: remap all jump-link and action URLs to `javascript:void(0)` and all resource URLs into empty `data:` URLs; the result will be self-contained
- output:
- `--not-separated`
: don't separate output values with anything, just concatenate them
- `-l, --lf-separated`
: separate output values with `\n` (LF) newline characters (default)
- `-z, --zero-separated`
: separate output values with `\0` (NUL) bytes
### wrrarms stream
Compute given expressions for each of given WRR files, encode them into a requested format, and print the result to stdout.
- positional arguments:
- `PATH`
: inputs, can be a mix of files and directories (which will be traversed recursively)
- options:
- `-u, --unabridged`
: print all data in full
- `--abridged`
: shorten long strings for brevity (useful when you want to visually scan through batch data dumps) (default)
- `--format {py,cbor,json,raw}`
: generate output in:
- py: Pythonic Object Representation aka `repr` (default)
- cbor: CBOR (RFC8949)
- json: JavaScript Object Notation aka JSON; **binary data can't be represented, UNICODE replacement characters will be used**
- raw: concatenate raw values; termination is controlled by `*-terminated` options
- `--stdin0`
: read zero-terminated `PATH`s from stdin, these will be processed after `PATH`s specified as command-line arguments
- error handling:
- `--errors {fail,skip,ignore}`
: when an error occurs:
- `fail`: report failure and stop the execution (default)
- `skip`: report failure but skip the reqres that produced it from the output and continue
- `ignore`: `skip`, but don't report the failure
- filters:
- `--or EXPR`
: only print reqres which match any of these expressions...
- `--and EXPR`
: ... and all of these expressions, both can be specified multiple times, both use the same expression format as `wrrarms get --expr`, which see
- expression evaluation:
- `-e EXPR, --expr EXPR`
: an expression to compute, see `wrrarms get --expr` for more info on expression format; can be specified multiple times; the default is `.` which will dump the whole reqres structure
- URL remapping, used by `scrub` `--expr` atom:
- `--remap-id`
: remap all URLs with an identity function; i.e. don't remap anything (default)
- `--remap-void`
: remap all jump-link and action URLs to `javascript:void(0)` and all resource URLs into empty `data:` URLs; the result will be self-contained
- `--format=raw` output:
- `--not-terminated`
: don't terminate `--format=raw` output values with anything, just concatenate them
- `-l, --lf-terminated`
: terminate `--format=raw` output values with `\n` (LF) newline characters (default)
- `-z, --zero-terminated`
: terminate `--format=raw` output values with `\0` (NUL) bytes
- file system path ordering:
- `--paths-given-order`
: `argv` and `--stdin0` `PATH`s are processed in the order they are given (default)
- `--paths-sorted`
: `argv` and `--stdin0` `PATH`s are processed in lexicographic order
- `--paths-reversed`
: `argv` and `--stdin0` `PATH`s are processed in reverse lexicographic order
- `--walk-fs-order`
: recursive file system walk is done in the order `readdir(2)` gives results
- `--walk-sorted`
: recursive file system walk is done in lexicographic order (default)
- `--walk-reversed`
: recursive file system walk is done in reverse lexicographic order
### wrrarms find
Print paths of WRR files matching specified criteria.
- positional arguments:
- `PATH`
: inputs, can be a mix of files and directories (which will be traversed recursively)
- options:
- `--stdin0`
: read zero-terminated `PATH`s from stdin, these will be processed after `PATH`s specified as command-line arguments
- error handling:
- `--errors {fail,skip,ignore}`
: when an error occurs:
- `fail`: report failure and stop the execution (default)
- `skip`: report failure but skip the reqres that produced it from the output and continue
- `ignore`: `skip`, but don't report the failure
- filters:
- `--or EXPR`
: only output paths to reqres which match any of these expressions...
- `--and EXPR`
: ... and all of these expressions, both can be specified multiple times, both use the same expression format as `wrrarms get --expr`, which see
- output:
- `-l, --lf-terminated`
: terminate output absolute paths of matching WRR files with `\n` (LF) newline characters (default)
- `-z, --zero-terminated`
: terminate output absolute paths of matching WRR files with `\0` (NUL) bytes
- file system path ordering:
- `--paths-given-order`
: `argv` and `--stdin0` `PATH`s are processed in the order they are given (default)
- `--paths-sorted`
: `argv` and `--stdin0` `PATH`s are processed in lexicographic order
- `--paths-reversed`
: `argv` and `--stdin0` `PATH`s are processed in reverse lexicographic order
- `--walk-fs-order`
: recursive file system walk is done in the order `readdir(2)` gives results
- `--walk-sorted`
: recursive file system walk is done in lexicographic order (default)
- `--walk-reversed`
: recursive file system walk is done in reverse lexicographic order
### wrrarms organize
Parse given WRR files into their respective reqres and then rename/move/hardlink/symlink each file to `DESTINATION` with the new path derived from each reqres' metadata.
Operations that could lead to accidental data loss are not permitted.
E.g. `wrrarms organize --move` will not overwrite any files, which is why the default `--output` contains `%(num)d`.
- positional arguments:
- `PATH`
: inputs, can be a mix of files and directories (which will be traversed recursively)
- options:
- `--dry-run`
: perform a trial run without actually performing any changes
- `-q, --quiet`
: don't log computed updates to stderr
- `-t DESTINATION, --to DESTINATION`
: destination directory, when unset each source `PATH` must be a directory which will be treated as its own `DESTINATION`
- `-o FORMAT, --output FORMAT`
: format describing generated output paths, an alias name or "format:" followed by a custom pythonic %-substitution string:
- available aliases and corresponding %-substitutions:
- `default` : `%(syear)d/%(smonth)02d/%(sday)02d/%(shour)02d%(sminute)02d%(ssecond)02d%(stime_msq)03d_%(qtime_ms)s_%(method)s_%(net_url|to_ascii|sha256|take_prefix 4)s_%(status)s_%(hostname)s.%(num)d` (default)
- `https://example.org` -> `1970/01/01/001640000_0_GET_50d7_200C_example.org.0`
- `https://example.org/` -> `1970/01/01/001640000_0_GET_8198_200C_example.org.0`
- `https://example.org/index.html` -> `1970/01/01/001640000_0_GET_f0dc_200C_example.org.0`
- `https://example.org/media` -> `1970/01/01/001640000_0_GET_086d_200C_example.org.0`
- `https://example.org/media/` -> `1970/01/01/001640000_0_GET_3fbb_200C_example.org.0`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `1970/01/01/001640000_0_GET_5658_200C_example.org.0`
- `https://königsgäßchen.example.org/index.html` -> `1970/01/01/001640000_0_GET_4f11_200C_königsgäßchen.example.org.0`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `1970/01/01/001640000_0_GET_c4ae_200C_ジャジェメント.ですの.example.org.0`
- `short` : `%(syear)d/%(smonth)02d/%(sday)02d/%(stime_ms)d_%(qtime_ms)s.%(num)d`
- `https://example.org`, `https://example.org/`, `https://example.org/index.html`, `https://example.org/media`, `https://example.org/media/`, `https://example.org/view?one=1&two=2&three=&three=3#fragment`, `https://königsgäßchen.example.org/index.html`, `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `1970/01/01/1000000_0.0`
- `surl` : `%(scheme)s/%(netloc)s/%(mq_path)s%(oqm)s%(mq_query)s`
- `https://example.org`, `https://example.org/` -> `https/example.org/`
- `https://example.org/index.html` -> `https/example.org/index.html`
- `https://example.org/media`, `https://example.org/media/` -> `https/example.org/media`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/example.org/view?one=1&two=2&three&three=3`
- `https://königsgäßchen.example.org/index.html` -> `https/königsgäßchen.example.org/index.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/ジャジェメント.ですの.example.org/испытание/is`
- `surl_msn` : `%(scheme)s/%(netloc)s/%(mq_path)s%(oqm)s%(mq_query)s_%(method)s_%(status)s.%(num)d`
- `https://example.org`, `https://example.org/` -> `https/example.org/_GET_200C.0`
- `https://example.org/index.html` -> `https/example.org/index.html_GET_200C.0`
- `https://example.org/media`, `https://example.org/media/` -> `https/example.org/media_GET_200C.0`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/example.org/view?one=1&two=2&three&three=3_GET_200C.0`
- `https://königsgäßchen.example.org/index.html` -> `https/königsgäßchen.example.org/index.html_GET_200C.0`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/ジャジェメント.ですの.example.org/испытание/is_GET_200C.0`
- `shupq` : `%(scheme)s/%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_query|abbrev 120)s%(filepath_ext)s`
- `https://example.org`, `https://example.org/` -> `https/example.org/index.htm`
- `https://example.org/index.html` -> `https/example.org/index.html`
- `https://example.org/media`, `https://example.org/media/` -> `https/example.org/media/index.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/example.org/view/index?one=1&two=2&three&three=3.htm`
- `https://königsgäßchen.example.org/index.html` -> `https/königsgäßchen.example.org/index.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/ジャジェメント.ですの.example.org/испытание/is/index.htm`
- `shupq_msn` : `%(scheme)s/%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_query|abbrev 100)s_%(method)s_%(status)s.%(num)d%(filepath_ext)s`
- `https://example.org`, `https://example.org/` -> `https/example.org/index_GET_200C.0.htm`
- `https://example.org/index.html` -> `https/example.org/index_GET_200C.0.html`
- `https://example.org/media`, `https://example.org/media/` -> `https/example.org/media/index_GET_200C.0.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/example.org/view/index?one=1&two=2&three&three=3_GET_200C.0.htm`
- `https://königsgäßchen.example.org/index.html` -> `https/königsgäßchen.example.org/index_GET_200C.0.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/ジャジェメント.ですの.example.org/испытание/is/index_GET_200C.0.htm`
- `shupnq` : `%(scheme)s/%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 120)s%(filepath_ext)s`
- `https://example.org`, `https://example.org/` -> `https/example.org/index.htm`
- `https://example.org/index.html` -> `https/example.org/index.html`
- `https://example.org/media`, `https://example.org/media/` -> `https/example.org/media/index.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/example.org/view/index?one=1&two=2&three=3.htm`
- `https://königsgäßchen.example.org/index.html` -> `https/königsgäßchen.example.org/index.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/ジャジェメント.ですの.example.org/испытание/is/index.htm`
- `shupnq_msn` : `%(scheme)s/%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 100)s_%(method)s_%(status)s.%(num)d%(filepath_ext)s`
- `https://example.org`, `https://example.org/` -> `https/example.org/index_GET_200C.0.htm`
- `https://example.org/index.html` -> `https/example.org/index_GET_200C.0.html`
- `https://example.org/media`, `https://example.org/media/` -> `https/example.org/media/index_GET_200C.0.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/example.org/view/index?one=1&two=2&three=3_GET_200C.0.htm`
- `https://königsgäßchen.example.org/index.html` -> `https/königsgäßchen.example.org/index_GET_200C.0.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/ジャジェメント.ですの.example.org/испытание/is/index_GET_200C.0.htm`
- `shupnq_mhs` : `%(scheme)s/%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 120)s_%(method)s_%(net_url|to_ascii|sha256|take_prefix 4)s_%(status)s%(filepath_ext)s`
- `https://example.org` -> `https/example.org/index_GET_50d7_200C.htm`
- `https://example.org/` -> `https/example.org/index_GET_8198_200C.htm`
- `https://example.org/index.html` -> `https/example.org/index_GET_f0dc_200C.html`
- `https://example.org/media` -> `https/example.org/media/index_GET_086d_200C.htm`
- `https://example.org/media/` -> `https/example.org/media/index_GET_3fbb_200C.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/example.org/view/index?one=1&two=2&three=3_GET_5658_200C.htm`
- `https://königsgäßchen.example.org/index.html` -> `https/königsgäßchen.example.org/index_GET_4f11_200C.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/ジャジェメント.ですの.example.org/испытание/is/index_GET_c4ae_200C.htm`
- `shupnq_mhsn` : `%(scheme)s/%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 100)s_%(method)s_%(net_url|to_ascii|sha256|take_prefix 4)s_%(status)s.%(num)d%(filepath_ext)s`
- `https://example.org` -> `https/example.org/index_GET_50d7_200C.0.htm`
- `https://example.org/` -> `https/example.org/index_GET_8198_200C.0.htm`
- `https://example.org/index.html` -> `https/example.org/index_GET_f0dc_200C.0.html`
- `https://example.org/media` -> `https/example.org/media/index_GET_086d_200C.0.htm`
- `https://example.org/media/` -> `https/example.org/media/index_GET_3fbb_200C.0.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/example.org/view/index?one=1&two=2&three=3_GET_5658_200C.0.htm`
- `https://königsgäßchen.example.org/index.html` -> `https/königsgäßchen.example.org/index_GET_4f11_200C.0.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/ジャジェメント.ですの.example.org/испытание/is/index_GET_c4ae_200C.0.htm`
- `srhupq` : `%(scheme)s/%(rhostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_query|abbrev 120)s%(filepath_ext)s`
- `https://example.org`, `https://example.org/` -> `https/org.example/index.htm`
- `https://example.org/index.html` -> `https/org.example/index.html`
- `https://example.org/media`, `https://example.org/media/` -> `https/org.example/media/index.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/org.example/view/index?one=1&two=2&three&three=3.htm`
- `https://königsgäßchen.example.org/index.html` -> `https/org.example.königsgäßchen/index.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/org.example.ですの.ジャジェメント/испытание/is/index.htm`
- `srhupq_msn` : `%(scheme)s/%(rhostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_query|abbrev 100)s_%(method)s_%(status)s.%(num)d%(filepath_ext)s`
- `https://example.org`, `https://example.org/` -> `https/org.example/index_GET_200C.0.htm`
- `https://example.org/index.html` -> `https/org.example/index_GET_200C.0.html`
- `https://example.org/media`, `https://example.org/media/` -> `https/org.example/media/index_GET_200C.0.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/org.example/view/index?one=1&two=2&three&three=3_GET_200C.0.htm`
- `https://königsgäßchen.example.org/index.html` -> `https/org.example.königsgäßchen/index_GET_200C.0.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/org.example.ですの.ジャジェメント/испытание/is/index_GET_200C.0.htm`
- `srhupnq` : `%(scheme)s/%(rhostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 120)s%(filepath_ext)s`
- `https://example.org`, `https://example.org/` -> `https/org.example/index.htm`
- `https://example.org/index.html` -> `https/org.example/index.html`
- `https://example.org/media`, `https://example.org/media/` -> `https/org.example/media/index.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/org.example/view/index?one=1&two=2&three=3.htm`
- `https://königsgäßchen.example.org/index.html` -> `https/org.example.königsgäßchen/index.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/org.example.ですの.ジャジェメント/испытание/is/index.htm`
- `srhupnq_msn` : `%(scheme)s/%(rhostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 100)s_%(method)s_%(status)s.%(num)d%(filepath_ext)s`
- `https://example.org`, `https://example.org/` -> `https/org.example/index_GET_200C.0.htm`
- `https://example.org/index.html` -> `https/org.example/index_GET_200C.0.html`
- `https://example.org/media`, `https://example.org/media/` -> `https/org.example/media/index_GET_200C.0.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/org.example/view/index?one=1&two=2&three=3_GET_200C.0.htm`
- `https://königsgäßchen.example.org/index.html` -> `https/org.example.königsgäßchen/index_GET_200C.0.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/org.example.ですの.ジャジェメント/испытание/is/index_GET_200C.0.htm`
- `srhupnq_mhs` : `%(scheme)s/%(rhostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 120)s_%(method)s_%(net_url|to_ascii|sha256|take_prefix 4)s_%(status)s%(filepath_ext)s`
- `https://example.org` -> `https/org.example/index_GET_50d7_200C.htm`
- `https://example.org/` -> `https/org.example/index_GET_8198_200C.htm`
- `https://example.org/index.html` -> `https/org.example/index_GET_f0dc_200C.html`
- `https://example.org/media` -> `https/org.example/media/index_GET_086d_200C.htm`
- `https://example.org/media/` -> `https/org.example/media/index_GET_3fbb_200C.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/org.example/view/index?one=1&two=2&three=3_GET_5658_200C.htm`
- `https://königsgäßchen.example.org/index.html` -> `https/org.example.königsgäßchen/index_GET_4f11_200C.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/org.example.ですの.ジャジェメント/испытание/is/index_GET_c4ae_200C.htm`
- `srhupnq_mhsn`: `%(scheme)s/%(rhostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 100)s_%(method)s_%(net_url|to_ascii|sha256|take_prefix 4)s_%(status)s.%(num)d%(filepath_ext)s`
- `https://example.org` -> `https/org.example/index_GET_50d7_200C.0.htm`
- `https://example.org/` -> `https/org.example/index_GET_8198_200C.0.htm`
- `https://example.org/index.html` -> `https/org.example/index_GET_f0dc_200C.0.html`
- `https://example.org/media` -> `https/org.example/media/index_GET_086d_200C.0.htm`
- `https://example.org/media/` -> `https/org.example/media/index_GET_3fbb_200C.0.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/org.example/view/index?one=1&two=2&three=3_GET_5658_200C.0.htm`
- `https://königsgäßchen.example.org/index.html` -> `https/org.example.königsgäßchen/index_GET_4f11_200C.0.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/org.example.ですの.ジャジェメント/испытание/is/index_GET_c4ae_200C.0.htm`
- `url` : `%(netloc)s/%(mq_path)s%(oqm)s%(mq_query)s`
- `https://example.org`, `https://example.org/` -> `example.org/`
- `https://example.org/index.html` -> `example.org/index.html`
- `https://example.org/media`, `https://example.org/media/` -> `example.org/media`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `example.org/view?one=1&two=2&three&three=3`
- `https://königsgäßchen.example.org/index.html` -> `königsgäßchen.example.org/index.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `ジャジェメント.ですの.example.org/испытание/is`
- `url_msn` : `%(netloc)s/%(mq_path)s%(oqm)s%(mq_query)s_%(method)s_%(status)s.%(num)d`
- `https://example.org`, `https://example.org/` -> `example.org/_GET_200C.0`
- `https://example.org/index.html` -> `example.org/index.html_GET_200C.0`
- `https://example.org/media`, `https://example.org/media/` -> `example.org/media_GET_200C.0`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `example.org/view?one=1&two=2&three&three=3_GET_200C.0`
- `https://königsgäßchen.example.org/index.html` -> `königsgäßchen.example.org/index.html_GET_200C.0`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `ジャジェメント.ですの.example.org/испытание/is_GET_200C.0`
- `hupq` : `%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_query|abbrev 120)s%(filepath_ext)s`
- `https://example.org`, `https://example.org/` -> `example.org/index.htm`
- `https://example.org/index.html` -> `example.org/index.html`
- `https://example.org/media`, `https://example.org/media/` -> `example.org/media/index.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `example.org/view/index?one=1&two=2&three&three=3.htm`
- `https://königsgäßchen.example.org/index.html` -> `königsgäßchen.example.org/index.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `ジャジェメント.ですの.example.org/испытание/is/index.htm`
- `hupq_msn` : `%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_query|abbrev 100)s_%(method)s_%(status)s.%(num)d%(filepath_ext)s`
- `https://example.org`, `https://example.org/` -> `example.org/index_GET_200C.0.htm`
- `https://example.org/index.html` -> `example.org/index_GET_200C.0.html`
- `https://example.org/media`, `https://example.org/media/` -> `example.org/media/index_GET_200C.0.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `example.org/view/index?one=1&two=2&three&three=3_GET_200C.0.htm`
- `https://königsgäßchen.example.org/index.html` -> `königsgäßchen.example.org/index_GET_200C.0.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `ジャジェメント.ですの.example.org/испытание/is/index_GET_200C.0.htm`
- `hupnq` : `%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 120)s%(filepath_ext)s`
- `https://example.org`, `https://example.org/` -> `example.org/index.htm`
- `https://example.org/index.html` -> `example.org/index.html`
- `https://example.org/media`, `https://example.org/media/` -> `example.org/media/index.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `example.org/view/index?one=1&two=2&three=3.htm`
- `https://königsgäßchen.example.org/index.html` -> `königsgäßchen.example.org/index.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `ジャジェメント.ですの.example.org/испытание/is/index.htm`
- `hupnq_msn` : `%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 100)s_%(method)s_%(status)s.%(num)d%(filepath_ext)s`
- `https://example.org`, `https://example.org/` -> `example.org/index_GET_200C.0.htm`
- `https://example.org/index.html` -> `example.org/index_GET_200C.0.html`
- `https://example.org/media`, `https://example.org/media/` -> `example.org/media/index_GET_200C.0.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `example.org/view/index?one=1&two=2&three=3_GET_200C.0.htm`
- `https://königsgäßchen.example.org/index.html` -> `königsgäßchen.example.org/index_GET_200C.0.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `ジャジェメント.ですの.example.org/испытание/is/index_GET_200C.0.htm`
- `hupnq_mhs` : `%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 120)s_%(method)s_%(net_url|to_ascii|sha256|take_prefix 4)s_%(status)s%(filepath_ext)s`
- `https://example.org` -> `example.org/index_GET_50d7_200C.htm`
- `https://example.org/` -> `example.org/index_GET_8198_200C.htm`
- `https://example.org/index.html` -> `example.org/index_GET_f0dc_200C.html`
- `https://example.org/media` -> `example.org/media/index_GET_086d_200C.htm`
- `https://example.org/media/` -> `example.org/media/index_GET_3fbb_200C.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `example.org/view/index?one=1&two=2&three=3_GET_5658_200C.htm`
- `https://königsgäßchen.example.org/index.html` -> `königsgäßchen.example.org/index_GET_4f11_200C.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `ジャジェメント.ですの.example.org/испытание/is/index_GET_c4ae_200C.htm`
- `hupnq_mhsn` : `%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 100)s_%(method)s_%(net_url|to_ascii|sha256|take_prefix 4)s_%(status)s.%(num)d%(filepath_ext)s`
- `https://example.org` -> `example.org/index_GET_50d7_200C.0.htm`
- `https://example.org/` -> `example.org/index_GET_8198_200C.0.htm`
- `https://example.org/index.html` -> `example.org/index_GET_f0dc_200C.0.html`
- `https://example.org/media` -> `example.org/media/index_GET_086d_200C.0.htm`
- `https://example.org/media/` -> `example.org/media/index_GET_3fbb_200C.0.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `example.org/view/index?one=1&two=2&three=3_GET_5658_200C.0.htm`
- `https://königsgäßchen.example.org/index.html` -> `königsgäßchen.example.org/index_GET_4f11_200C.0.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `ジャジェメント.ですの.example.org/испытание/is/index_GET_c4ae_200C.0.htm`
- `rhupq` : `%(rhostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_query|abbrev 120)s%(filepath_ext)s`
- `https://example.org`, `https://example.org/` -> `org.example/index.htm`
- `https://example.org/index.html` -> `org.example/index.html`
- `https://example.org/media`, `https://example.org/media/` -> `org.example/media/index.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `org.example/view/index?one=1&two=2&three&three=3.htm`
- `https://königsgäßchen.example.org/index.html` -> `org.example.königsgäßchen/index.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `org.example.ですの.ジャジェメント/испытание/is/index.htm`
- `rhupq_msn` : `%(rhostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_query|abbrev 100)s_%(method)s_%(status)s.%(num)d%(filepath_ext)s`
- `https://example.org`, `https://example.org/` -> `org.example/index_GET_200C.0.htm`
- `https://example.org/index.html` -> `org.example/index_GET_200C.0.html`
- `https://example.org/media`, `https://example.org/media/` -> `org.example/media/index_GET_200C.0.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `org.example/view/index?one=1&two=2&three&three=3_GET_200C.0.htm`
- `https://königsgäßchen.example.org/index.html` -> `org.example.königsgäßchen/index_GET_200C.0.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `org.example.ですの.ジャジェメント/испытание/is/index_GET_200C.0.htm`
- `rhupnq` : `%(rhostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 120)s%(filepath_ext)s`
- `https://example.org`, `https://example.org/` -> `org.example/index.htm`
- `https://example.org/index.html` -> `org.example/index.html`
- `https://example.org/media`, `https://example.org/media/` -> `org.example/media/index.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `org.example/view/index?one=1&two=2&three=3.htm`
- `https://königsgäßchen.example.org/index.html` -> `org.example.königsgäßchen/index.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `org.example.ですの.ジャジェメント/испытание/is/index.htm`
- `rhupnq_msn` : `%(rhostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 100)s_%(method)s_%(status)s.%(num)d%(filepath_ext)s`
- `https://example.org`, `https://example.org/` -> `org.example/index_GET_200C.0.htm`
- `https://example.org/index.html` -> `org.example/index_GET_200C.0.html`
- `https://example.org/media`, `https://example.org/media/` -> `org.example/media/index_GET_200C.0.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `org.example/view/index?one=1&two=2&three=3_GET_200C.0.htm`
- `https://königsgäßchen.example.org/index.html` -> `org.example.königsgäßchen/index_GET_200C.0.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `org.example.ですの.ジャジェメント/испытание/is/index_GET_200C.0.htm`
- `rhupnq_mhs` : `%(rhostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 120)s_%(method)s_%(net_url|to_ascii|sha256|take_prefix 4)s_%(status)s%(filepath_ext)s`
- `https://example.org` -> `org.example/index_GET_50d7_200C.htm`
- `https://example.org/` -> `org.example/index_GET_8198_200C.htm`
- `https://example.org/index.html` -> `org.example/index_GET_f0dc_200C.html`
- `https://example.org/media` -> `org.example/media/index_GET_086d_200C.htm`
- `https://example.org/media/` -> `org.example/media/index_GET_3fbb_200C.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `org.example/view/index?one=1&two=2&three=3_GET_5658_200C.htm`
- `https://königsgäßchen.example.org/index.html` -> `org.example.königsgäßchen/index_GET_4f11_200C.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `org.example.ですの.ジャジェメント/испытание/is/index_GET_c4ae_200C.htm`
- `rhupnq_mhsn` : `%(rhostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 100)s_%(method)s_%(net_url|to_ascii|sha256|take_prefix 4)s_%(status)s.%(num)d%(filepath_ext)s`
- `https://example.org` -> `org.example/index_GET_50d7_200C.0.htm`
- `https://example.org/` -> `org.example/index_GET_8198_200C.0.htm`
- `https://example.org/index.html` -> `org.example/index_GET_f0dc_200C.0.html`
- `https://example.org/media` -> `org.example/media/index_GET_086d_200C.0.htm`
- `https://example.org/media/` -> `org.example/media/index_GET_3fbb_200C.0.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `org.example/view/index?one=1&two=2&three=3_GET_5658_200C.0.htm`
- `https://königsgäßchen.example.org/index.html` -> `org.example.königsgäßchen/index_GET_4f11_200C.0.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `org.example.ですの.ジャジェメント/испытание/is/index_GET_c4ae_200C.0.htm`
- `flat` : `%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path|replace / __|abbrev 120)s%(oqm)s%(mq_nquery|abbrev 100)s%(filepath_ext)s`
- `https://example.org`, `https://example.org/` -> `example.org/index.htm`
- `https://example.org/index.html` -> `example.org/index.html`
- `https://example.org/media`, `https://example.org/media/` -> `example.org/media__index.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `example.org/view__index?one=1&two=2&three=3.htm`
- `https://königsgäßchen.example.org/index.html` -> `königsgäßchen.example.org/index.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `ジャジェメント.ですの.example.org/испытание__is__index.htm`
- `flat_ms` : `%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path|replace / __|abbrev 120)s%(oqm)s%(mq_nquery|abbrev 100)s_%(method)s_%(status)s%(filepath_ext)s`
- `https://example.org`, `https://example.org/` -> `example.org/index_GET_200C.htm`
- `https://example.org/index.html` -> `example.org/index_GET_200C.html`
- `https://example.org/media`, `https://example.org/media/` -> `example.org/media__index_GET_200C.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `example.org/view__index?one=1&two=2&three=3_GET_200C.htm`
- `https://königsgäßchen.example.org/index.html` -> `königsgäßchen.example.org/index_GET_200C.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `ジャジェメント.ですの.example.org/испытание__is__index_GET_200C.htm`
- `flat_msn` : `%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path|replace / __|abbrev 120)s%(oqm)s%(mq_nquery|abbrev 100)s_%(method)s_%(status)s.%(num)d%(filepath_ext)s`
- `https://example.org`, `https://example.org/` -> `example.org/index_GET_200C.0.htm`
- `https://example.org/index.html` -> `example.org/index_GET_200C.0.html`
- `https://example.org/media`, `https://example.org/media/` -> `example.org/media__index_GET_200C.0.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `example.org/view__index?one=1&two=2&three=3_GET_200C.0.htm`
- `https://königsgäßchen.example.org/index.html` -> `königsgäßchen.example.org/index_GET_200C.0.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `ジャジェメント.ですの.example.org/испытание__is__index_GET_200C.0.htm`
- `flat_mhs` : `%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path|replace / __|abbrev 120)s%(oqm)s%(mq_nquery|abbrev 100)s_%(method)s_%(net_url|to_ascii|sha256|take_prefix 4)s_%(status)s%(filepath_ext)s`
- `https://example.org` -> `example.org/index_GET_50d7_200C.htm`
- `https://example.org/` -> `example.org/index_GET_8198_200C.htm`
- `https://example.org/index.html` -> `example.org/index_GET_f0dc_200C.html`
- `https://example.org/media` -> `example.org/media__index_GET_086d_200C.htm`
- `https://example.org/media/` -> `example.org/media__index_GET_3fbb_200C.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `example.org/view__index?one=1&two=2&three=3_GET_5658_200C.htm`
- `https://königsgäßchen.example.org/index.html` -> `königsgäßchen.example.org/index_GET_4f11_200C.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `ジャジェメント.ですの.example.org/испытание__is__index_GET_c4ae_200C.htm`
- `flat_mhsn` : `%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path|replace / __|abbrev 120)s%(oqm)s%(mq_nquery|abbrev 100)s_%(method)s_%(net_url|to_ascii|sha256|take_prefix 4)s_%(status)s.%(num)d%(filepath_ext)s`
- `https://example.org` -> `example.org/index_GET_50d7_200C.0.htm`
- `https://example.org/` -> `example.org/index_GET_8198_200C.0.htm`
- `https://example.org/index.html` -> `example.org/index_GET_f0dc_200C.0.html`
- `https://example.org/media` -> `example.org/media__index_GET_086d_200C.0.htm`
- `https://example.org/media/` -> `example.org/media__index_GET_3fbb_200C.0.htm`
- `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `example.org/view__index?one=1&two=2&three=3_GET_5658_200C.0.htm`
- `https://königsgäßchen.example.org/index.html` -> `königsgäßchen.example.org/index_GET_4f11_200C.0.html`
- `https://ジャジェメント.ですの.example.org/испытание/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `ジャジェメント.ですの.example.org/испытание__is__index_GET_c4ae_200C.0.htm`
- available substitutions:
- `num`: number of times the resulting output path was encountered before; adding this parameter to your `--output` format will ensure all generated file names will be unique
- all expressions of `wrrarms get --expr`, which see
- `--stdin0`
: read zero-terminated `PATH`s from stdin, these will be processed after `PATH`s specified as command-line arguments
- error handling:
- `--errors {fail,skip,ignore}`
: when an error occurs:
- `fail`: report failure and stop the execution (default)
- `skip`: report failure but skip the reqres that produced it from the output and continue
- `ignore`: `skip`, but don't report the failure
- filters:
- `--or EXPR`
: only work on reqres which match any of these expressions...
- `--and EXPR`
: ... and all of these expressions, both can be specified multiple times, both use the same expression format as `wrrarms get --expr`, which see
- output:
- `--no-output`
: don't print anything (default)
- `-l, --lf-terminated`
: terminate output absolute paths of newly produced files with `\n` (LF) newline characters
- `-z, --zero-terminated`
: terminate output absolute paths of newly produced files with `\0` (NUL) bytes
- action:
- `--move`
: move source files under `DESTINATION` (default)
- `--copy`
: copy source files to files under `DESTINATION`
- `--hardlink`
: create hardlinks from source files to paths under `DESTINATION`
- `--symlink`
: create symlinks from source files to paths under `DESTINATION`
- updates:
- `--keep`
: disallow replacements and overwrites for any existing files under `DESTINATION` (default);
broken symlinks are allowed to be replaced;
if source and target directories are the same then some files can still be renamed into previously non-existing names;
all other updates are disallowed
- `--latest`
: replace files under `DESTINATION` if `stime_ms` for the source reqres is newer than the same value for reqres stored at the destination
- caching, deferring, and batching:
- `--seen-number INT`
: track at most this many distinct generated `--output` values; default: `16384`;
making this larger improves disk performance at the cost of increased memory consumption;
setting it to zero will force force `wrrarms` to constantly re-check existence of `--output` files and force `wrrarms` to execute all IO actions immediately, disregarding `--defer-number` setting
- `--cache-number INT`
: cache `stat(2)` information about this many files in memory; default: `8192`;
making this larger improves performance at the cost of increased memory consumption;
setting this to a too small number will likely force `wrrarms` into repeatedly performing lots of `stat(2)` system calls on the same files;
setting this to a value smaller than `--defer-number` will not improve memory consumption very much since deferred IO actions also cache information about their own files
- `--defer-number INT`
: defer at most this many IO actions; default: `1024`;
making this larger improves performance at the cost of increased memory consumption;
setting it to zero will force all IO actions to be applied immediately
- `--batch-number INT`
: queue at most this many deferred IO actions to be applied together in a batch; this queue will only be used if all other resource constraints are met; default: 128
- `--max-memory INT`
: the caches, the deferred actions queue, and the batch queue, all taken together, must not take more than this much memory in MiB; default: `1024`;
making this larger improves performance;
the actual maximum whole-program memory consumption is `O(<size of the largest reqres> + <--seen-number> + <sum of lengths of the last --seen-number generated --output paths> + <--cache-number> + <--defer-number> + <--batch-number> + <--max-memory>)`
- `--lazy`
: sets all of the above options to positive infinity;
most useful when doing `wrrarms organize --symlink --latest --output flat` or similar, where the number of distinct generated `--output` values and the amount of other data `wrrarms` needs to keep in memory is small, in which case it will force `wrrarms` to compute the desired file system state first and then perform all disk writes in a single batch
- file system path ordering:
- `--paths-given-order`
: `argv` and `--stdin0` `PATH`s are processed in the order they are given (default when `--keep`)
- `--paths-sorted`
: `argv` and `--stdin0` `PATH`s are processed in lexicographic order
- `--paths-reversed`
: `argv` and `--stdin0` `PATH`s are processed in reverse lexicographic order (default when `--latest`)
- `--walk-fs-order`
: recursive file system walk is done in the order `readdir(2)` gives results
- `--walk-sorted`
: recursive file system walk is done in lexicographic order (default when `--keep`)
- `--walk-reversed`
: recursive file system walk is done in reverse lexicographic order (default when `--latest`)
### wrrarms import
Use specified parser to parse data in each `INPUT` `PATH` into reqres and dump them under `DESTINATION` with paths derived from their metadata.
In short, this is `wrrarms organize --copy` but for non-WRR `INPUT` files.
- file formats:
- `{mitmproxy}`
- `mitmproxy`
: convert `mitmproxy` stream dumps into WRR files
### wrrarms import mitmproxy
Parse each `INPUT` `PATH` as `mitmproxy` stream dump (by using `mitmproxy`'s own parser) into a sequence of reqres and dump them under `DESTINATION` with paths derived from their metadata.
- positional arguments:
- `PATH`
: inputs, can be a mix of files and directories (which will be traversed recursively)
- options:
- `--dry-run`
: perform a trial run without actually performing any changes
- `-q, --quiet`
: don't log computed updates to stderr
- `-t DESTINATION, --to DESTINATION`
: destination directory
- `-o FORMAT, --output FORMAT`
: format describing generated output paths, an alias name or "format:" followed by a custom pythonic %-substitution string; same as `wrrarms organize --output`, which see
- `--stdin0`
: read zero-terminated `PATH`s from stdin, these will be processed after `PATH`s specified as command-line arguments
- error handling:
- `--errors {fail,skip,ignore}`
: when an error occurs:
- `fail`: report failure and stop the execution (default)
- `skip`: report failure but skip the reqres that produced it from the output and continue
- `ignore`: `skip`, but don't report the failure
- filters:
- `--or EXPR`
: only import reqres which match any of these expressions...
- `--and EXPR`
: ... and all of these expressions, both can be specified multiple times, both use the same expression format as `wrrarms get --expr`, which see
- output:
- `--no-output`
: don't print anything (default)
- `-l, --lf-terminated`
: terminate output absolute paths of newly produced files with `\n` (LF) newline characters
- `-z, --zero-terminated`
: terminate output absolute paths of newly produced files with `\0` (NUL) bytes
- caching, deferring, and batching:
- `--seen-number INT`
: track at most this many distinct generated `--output` values; default: `16384`;
making this larger improves disk performance at the cost of increased memory consumption;
setting it to zero will force force `wrrarms` to constantly re-check existence of `--output` files and force `wrrarms` to execute all IO actions immediately, disregarding `--defer-number` setting
- `--cache-number INT`
: cache `stat(2)` information about this many files in memory; default: `8192`;
making this larger improves performance at the cost of increased memory consumption;
setting this to a too small number will likely force `wrrarms` into repeatedly performing lots of `stat(2)` system calls on the same files;
setting this to a value smaller than `--defer-number` will not improve memory consumption very much since deferred IO actions also cache information about their own files
- `--defer-number INT`
: defer at most this many IO actions; default: `0`;
making this larger improves performance at the cost of increased memory consumption;
setting it to zero will force all IO actions to be applied immediately
- `--batch-number INT`
: queue at most this many deferred IO actions to be applied together in a batch; this queue will only be used if all other resource constraints are met; default: 128
- `--max-memory INT`
: the caches, the deferred actions queue, and the batch queue, all taken together, must not take more than this much memory in MiB; default: `1024`;
making this larger improves performance;
the actual maximum whole-program memory consumption is `O(<size of the largest reqres> + <--seen-number> + <sum of lengths of the last --seen-number generated --output paths> + <--cache-number> + <--defer-number> + <--batch-number> + <--max-memory>)`
- `--lazy`
: sets all of the above options to positive infinity;
most useful when doing `wrrarms organize --symlink --latest --output flat` or similar, where the number of distinct generated `--output` values and the amount of other data `wrrarms` needs to keep in memory is small, in which case it will force `wrrarms` to compute the desired file system state first and then perform all disk writes in a single batch
- file system path ordering:
- `--paths-given-order`
: `argv` and `--stdin0` `PATH`s are processed in the order they are given (default)
- `--paths-sorted`
: `argv` and `--stdin0` `PATH`s are processed in lexicographic order
- `--paths-reversed`
: `argv` and `--stdin0` `PATH`s are processed in reverse lexicographic order
- `--walk-fs-order`
: recursive file system walk is done in the order `readdir(2)` gives results
- `--walk-sorted`
: recursive file system walk is done in lexicographic order (default)
- `--walk-reversed`
: recursive file system walk is done in reverse lexicographic order
### wrrarms export
Parse given WRR files into their respective reqres, convert to another file format, and then dump the result under `DESTINATION` with the new path derived from each reqres' metadata.
- file formats:
- `{mirror}`
- `mirror`
: convert given WRR files into a local website mirror stored in interlinked plain files
### wrrarms export mirror
Parse given WRR files, filter out those that have no responses, transform and then dump their response bodies into separate files under `DESTINATION` with the new path derived from each reqres' metadata.
In short, this is a combination of `wrrarms organize --copy` followed by in-place `wrrarms get`.
In other words, this generates static offline website mirrors, producing results similar to those of `wget -mpk`.
- positional arguments:
- `PATH`
: inputs, can be a mix of files and directories (which will be traversed recursively)
- options:
- `--dry-run`
: perform a trial run without actually performing any changes
- `-q, --quiet`
: don't log computed updates to stderr
- `-t DESTINATION, --to DESTINATION`
: target directory
- `-o FORMAT, --output FORMAT`
: format describing generated output paths, an alias name or a custom pythonic %-substitution string; same as `wrrarms organize --output`, which see
- `--stdin0`
: read zero-terminated `PATH`s from stdin, these will be processed after `PATH`s specified as command-line arguments
- error handling:
- `--errors {fail,skip,ignore}`
: when an error occurs:
- `fail`: report failure and stop the execution (default)
- `skip`: report failure but skip the reqres that produced it from the output and continue
- `ignore`: `skip`, but don't report the failure
- filters:
- `--or EXPR`
: only export reqres which match any of these expressions...
- `--and EXPR`
: ... and all of these expressions, both can be specified multiple times, both use the same expression format as `wrrarms get --expr`, which see
- output:
- `--no-output`
: don't print anything (default)
- `-l, --lf-terminated`
: terminate output absolute paths of newly produced files with `\n` (LF) newline characters
- `-z, --zero-terminated`
: terminate output absolute paths of newly produced files with `\0` (NUL) bytes
- expression evaluation:
- `-e EXPR, --expr EXPR`
: an expression to export, see `wrrarms get --expr` for more info on expression format (default: `response.body|eb|scrub response +all_refs,-actions`)
- URL remapping, used by `scrub` `--expr` atom:
- `--remap-id`
: remap all URLs with an identity function; i.e. don't remap anything
- `--remap-void`
: remap all jump-link and action URLs to `javascript:void(0)` and all resource URLs into empty `data:` URLs; the result will be self-contained
- `--remap-open, -k, --convert-links`
: point all available URLs present in input `PATH`s to their corresponding output paths, remap all unavailable URLs like `--remap-id` does; this is similar to `wget (-k|--convert-links)`
- `--remap-closed`
: remap all available URLs like `--remap-open` does, remap all unavailable URLs like `--remap-void` does; the result will be self-contained
- `--remap-all`
: remap all available URLs like `--remap-open` does, point each unavailable URL to a path produced by the current `--output` format for a trivial `GET <URL> -> 200 OK` reqres; this will produce broken links if the `--output` format depends on anything but the URL itself, but for a simple `--output` (like the default `hupq`) this allows `wrrarms export` to be used incrementally; the result will be self-contained (default)
- export targets (default: `net_url`s of all input `PATH`s):
- `-r URL, --root URL`
: recursion root; a URL which will be used as a root for recursive export; can be specified multiple times; if none are specified, then all URLs available from `PATH`s are treated as roots
- `-d DEPTH, --depth DEPTH`
: maximum recursion depth level; the default is `0`, which means "documents and their resources only"; setting this to `1` will also export one level of documents referenced via jump and action links, if those are being remapped to local files with `--remap-*`; higher values will mean even more recursion
- file system path ordering:
- `--paths-given-order`
: `argv` and `--stdin0` `PATH`s are processed in the order they are given (default)
- `--paths-sorted`
: `argv` and `--stdin0` `PATH`s are processed in lexicographic order
- `--paths-reversed`
: `argv` and `--stdin0` `PATH`s are processed in reverse lexicographic order
- `--walk-fs-order`
: recursive file system walk is done in the order `readdir(2)` gives results
- `--walk-sorted`
: recursive file system walk is done in lexicographic order (default)
- `--walk-reversed`
: recursive file system walk is done in reverse lexicographic order
## Examples
- Pretty-print all reqres in `../dumb_server/pwebarc-dump` using an abridged (for ease of reading and rendering) verbose textual representation:
```
wrrarms pprint ../dumb_server/pwebarc-dump
```
- Pipe response body scrubbed of dynamic content (see `wrrarms get` documentation above) from a given WRR file to stdout:
```
wrrarms get ../dumb_server/pwebarc-dump/path/to/file.wrr
```
- Pipe raw response body from a given WRR file to stdout:
```
wrrarms get -e "response.body|eb" ../dumb_server/pwebarc-dump/path/to/file.wrr
```
- Get first 4 characters of a hex digest of sha256 hash computed on the URL without the fragment/hash part:
```
wrrarms get -e "net_url|to_ascii|sha256|take_prefix 4" ../dumb_server/pwebarc-dump/path/to/file.wrr
```
- Pipe response body from a given WRR file to stdout, but less efficiently, by generating a temporary file and giving it to `cat`:
```
wrrarms run cat ../dumb_server/pwebarc-dump/path/to/file.wrr
```
Thus `wrrarms run` can be used to do almost anything you want, e.g.
```
wrrarms run less ../dumb_server/pwebarc-dump/path/to/file.wrr
```
```
wrrarms run -- sort -R ../dumb_server/pwebarc-dump/path/to/file.wrr
```
```
wrrarms run -n 2 -- diff -u ../dumb_server/pwebarc-dump/path/to/file-v1.wrr ../dumb_server/pwebarc-dump/path/to/file-v2.wrr
```
- List paths of all WRR files from `../dumb_server/pwebarc-dump` that contain only complete `200 OK` responses with bodies larger than 1K:
```
wrrarms find --and "status|== 200C" --and "response.body|len|> 1024" ../dumb_server/pwebarc-dump
```
- Rename all WRR files in `../dumb_server/pwebarc-dump/default` according to their metadata using `--output default` (see the `wrrarms organize` section for its definition, the `default` format is designed to be human-readable while causing almost no collisions, thus making `num` substitution parameter to almost always stay equal to `0`, making things nice and deterministic):
```
wrrarms organize ../dumb_server/pwebarc-dump/default
```
alternatively, just show what would be done
```
wrrarms organize --dry-run ../dumb_server/pwebarc-dump/default
```
- The output of `wrrarms organize --zero-terminated` can be piped into `wrrarms organize --stdin0` to perform complex updates. E.g. the following will rename new reqres from `../dumb_server/pwebarc-dump` to `~/pwebarc/raw` renaming them with `--output default`, the `for` loop is there to preserve profiles:
```
for arg in ../dumb_server/pwebarc-dump/* ; do
wrrarms organize --zero-terminated --to ~/pwebarc/raw/"$(basename "$arg")" "$arg"
done > changes
```
then, we can reuse `changes` to symlink all new files from `~/pwebarc/raw` to `~/pwebarc/all` using `--output hupq_msn`, which would show most of the URL in the file name:
```
wrrarms organize --stdin0 --symlink --to ~/pwebarc/all --output hupq_msn < changes
```
and then, we can reuse `changes` again and use them to update `~/pwebarc/latest`, filling it with symlinks pointing to the latest `200 OK` complete reqres from `~/pwebarc/raw`, similar to what `wget -r` would produce (except `wget` would do network requests and produce responce bodies, while this will build a file system tree of symlinks to WRR files in `/pwebarc/raw`):
```
wrrarms organize --stdin0 --symlink --latest --to ~/pwebarc/latest --output hupq --and "status|== 200C" < changes
```
- `wrrarms organize --move` is de-duplicating when possible, while `--copy`, `--hardlink`, and `--symlink` are non-duplicating when possible, i.e.:
```
wrrarms organize --copy --to ~/pwebarc/copy1 ~/pwebarc/original
wrrarms organize --copy --to ~/pwebarc/copy2 ~/pwebarc/original
wrrarms organize --hardlink --to ~/pwebarc/copy3 ~/pwebarc/original
# noops
wrrarms organize --copy --to ~/pwebarc/copy1 ~/pwebarc/original
wrrarms organize --hardlink --to ~/pwebarc/copy1 ~/pwebarc/original
wrrarms organize --copy --to ~/pwebarc/copy2 ~/pwebarc/original
wrrarms organize --hardlink --to ~/pwebarc/copy2 ~/pwebarc/original
wrrarms organize --copy --to ~/pwebarc/copy3 ~/pwebarc/original
wrrarms organize --hardlink --to ~/pwebarc/copy3 ~/pwebarc/original
# de-duplicate
wrrarms organize --move --to ~/pwebarc/all ~/pwebarc/original ~/pwebarc/copy1 ~/pwebarc/copy2 ~/pwebarc/copy3
```
will produce `~/pwebarc/all` which has each duplicated file stored only once. Similarly,
```
wrrarms organize --symlink --output hupq_msn --to ~/pwebarc/pointers ~/pwebarc/original
wrrarms organize --symlink --output shupq_msn --to ~/pwebarc/schemed ~/pwebarc/original
# noop
wrrarms organize --symlink --output hupq_msn --to ~/pwebarc/pointers ~/pwebarc/original ~/pwebarc/schemed
```
will produce `~/pwebarc/pointers` which has each symlink only once.
## Advanced examples
- Pretty-print all reqres in `../dumb_server/pwebarc-dump` by dumping their whole structure into an abridged Pythonic Object Representation (repr):
```
wrrarms stream --expr . ../dumb_server/pwebarc-dump
```
```
wrrarms stream -e . ../dumb_server/pwebarc-dump
```
- Pretty-print all reqres in `../dumb_server/pwebarc-dump` using the unabridged verbose textual representation:
```
wrrarms pprint --unabridged ../dumb_server/pwebarc-dump
```
```
wrrarms pprint -u ../dumb_server/pwebarc-dump
```
- Pretty-print all reqres in `../dumb_server/pwebarc-dump` by dumping their whole structure into the unabridged Pythonic Object Representation (repr) format:
```
wrrarms stream --unabridged --expr . ../dumb_server/pwebarc-dump
```
```
wrrarms stream -ue . ../dumb_server/pwebarc-dump
```
- Produce a JSON list of `[<file path>, <time it finished loading in milliseconds since UNIX epoch>, <URL>]` tuples (one per reqres) and pipe it into `jq` for indented and colored output:
```
wrrarms stream --format=json -ue fs_path -e finished_at -e request.url ../dumb_server/pwebarc-dump | jq .
```
- Similarly, but produce a CBOR output:
```
wrrarms stream --format=cbor -ue fs_path -e finished_at -e request.url ../dumb_server/pwebarc-dump | less
```
- Concatenate all response bodies of all the requests in `../dumb_server/pwebarc-dump`:
```
wrrarms stream --format=raw --not-terminated -ue "response.body|es" ../dumb_server/pwebarc-dump | less
```
- Print all unique visited URLs, one per line:
```
wrrarms stream --format=raw --lf-terminated -ue request.url ../dumb_server/pwebarc-dump | sort | uniq
```
- Same idea, but using NUL bytes while processing, and prints two URLs per line:
```
wrrarms stream --format=raw --zero-terminated -ue request.url ../dumb_server/pwebarc-dump | sort -z | uniq -z | xargs -0 -n2 echo
```
### How to handle binary data
Trying to use response bodies produced by `wrrarms stream --format=json` is likely to result garbled data as JSON can't represent raw sequences of bytes, thus binary data will have to be encoded into UNICODE using replacement characters:
```
wrrarms stream --format=json -ue . ../dumb_server/pwebarc-dump/path/to/file.wrr | jq .
```
The most generic solution to this is to use `--format=cbor` instead, which would produce a verbose CBOR representation equivalent to the one used by `--format=json` but with binary data preserved as-is:
```
wrrarms stream --format=cbor -ue . ../dumb_server/pwebarc-dump/path/to/file.wrr | less
```
Or you could just dump raw response bodies separately:
```
wrrarms stream --format=raw -ue response.body ../dumb_server/pwebarc-dump/path/to/file.wrr | less
```
```
wrrarms get ../dumb_server/pwebarc-dump/path/to/file.wrr | less
```
Raw data
{
"_id": null,
"home_page": null,
"name": "pwebarc-wrrarms",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "HTTP, HTTPS, archive, wayback machine, download",
"author": null,
"author_email": "Jan Malakhovski <oxij@oxij.org>",
"download_url": "https://files.pythonhosted.org/packages/41/11/1f43c3e03423d6acdbbafcf81b2ccdf6e7e613443332d6ea1b4e451c28ed/pwebarc-wrrarms-0.11.0.tar.gz",
"platform": null,
"description": "# What?\n\n`wrrarms` (`pwebarc-wrrarms`) is a tool for displaying, programmatically manipulating, organizing, importing, and exporting [Personal Private Passive Web Archive (pwebarc)](https://github.com/Own-Data-Privateer/pwebarc/) (also [there](https://oxij.org/software/pwebarc/)) Web Request+Response (WRR) files produced by [pWebArc browser extension](https://github.com/Own-Data-Privateer/pwebarc/tree/master/extension/) (also [there](https://oxij.org/software/pwebarc/tree/master/extension/)).\n\n# Quickstart\n\n## Installation\n\n- Install with:\n ```bash\n pip install pwebarc-wrrarms\n ```\n and run as\n ```bash\n wrrarms --help\n ```\n- Alternatively, install it via Nix\n ```bash\n nix-env -i -f ./default.nix\n wrrarms --help\n ```\n- Alternatively, run without installing:\n ```bash\n alias wrrarms=\"python3 -m wrrarms\"\n wrrarms --help\n ```\n\n## How to build a file system tree of latest versions of all hoarded URLs\n\nAssuming you keep your WRR dumps in `~/pwebarc/raw` you can generate a hierarchy of symlinks for each URL pointing from under `~/pwebarc/latest` to the most recent WRR file in `~/pwebarc/raw` via:\n\n```bash\nwrrarms organize --symlink --latest --output hupq --to ~/pwebarc/latest --and \"status|== 200C\" ~/pwebarc/raw\n```\n\nPersonally, I prefer `flat_mhs` (see the documentation of the `--output` below) format as I dislike deep file hierarchies, using it also simplifies filtering in my `ranger` file browser, so I do this:\n\n```bash\nwrrarms organize --symlink --latest --output flat_mhs --to ~/pwebarc/latest --and \"status|== 200C\" ~/pwebarc/raw\n```\n\nThese commands rescan the whole of `~/pwebarc/raw` and so take a while to complete.\nIf you have a lot of WRR files and you want to keep your symlink tree updated in real-time you can use a two-stage `--stdin0` pipeline shown in the [examples section](#examples) below.\n\n## <span id=\"mirror\"/>How to generate a local offline website mirror like `wget -mpk`\n\nIf you want to render your WRR files into a local offline website mirror containing interlinked HTML files and their resources a-la `wget -mpk` (`wget --mirror --page-requisites --convert-links`), run one of the above `--symlink --latest` command, and then do something like this:\n\n```bash\nwrrarms export mirror --to ~/pwebarc/mirror1 ~/pwebarc/latest/archiveofourown.org\n```\n\non completion `~/pwebarc/mirror1` will contain a bunch of interlinked minimized HTML files, their resources, and any other files they link to.\nBy default, *all* the links in exported HTML files will be remapped to local files (even if source WRR files for those would-be exported files are missing in `~/pwebarc/latest/archiveofourown.org`), and those HTML files will also be stripped of all JavaScript, CSS, and other stuff of various levels of evil (see documentation for the `scrub` function below).\n\nOn the plus side, the result will be completely self-contained and safe to view with a dumb unconfigured browser.\n\nIf you are unhappy with this behaviour and, for instance, want to keep the CSS and produce human-readable HTML, run the following instead:\n\n```bash\nwrrarms export mirror -e 'response.body|eb|scrub response +all_refs,-actions,+styles,+pretty' --to ~/pwebarc/mirror2 ~/pwebarc/latest/archiveofourown.org\n```\n\nNote, however, that CSS resource filtering and remapping is not implemented yet.\n\nIf you also want to keep links that point to not yet hoarded Internet URLs to still point those URLs in the exported files instead of them pointing to non-existent local files, similarly to what `wget -mpk` does, run:\n\n```bash\nwrrarms export mirror -e 'response.body|eb|scrub response +all_refs,-actions,+styles,+pretty' --remap-open --to ~/pwebarc/mirror3 ~/pwebarc/latest/archiveofourown.org\n```\n\nFinally, if you want a mirror made of raw files without any content censorship or link conversions, run:\n\n```bash\nwrrarms export mirror -e 'response.body|eb' --to ~/pwebarc/mirror-raw ~/pwebarc/latest/archiveofourown.org\n```\n\nThe later command will render your mirror pretty quick, but the other above-mentioned commands will call the `scrub` function, and that will be pretty slow (as in avg ~5Mb, ~3 files per second on my 2013-era laptop), mostly because `html5lib` that `wrrarms` uses for paranoid HTML parsing and filtering is fairly slow.\n\n## How to generate previews for WRR files, listen to them via TTS, open them with `xdg-open`, etc\n\nSee [`script` sub-directory](./script/README.md) for examples that show how to use `pandoc` and/or `w3m` to turn WRR files into previews and readable plain-text that can viewed or listened to via other tools, or dump them into temporary raw data files that can then be immediately fed to `xdg-open` for one-click viewing.\n\n# <span id=\"todo\"/>What is left TODO\n\n- Converters from HAR and WARC to WRR.\n- Data de-duplication between different WRR files.\n- Non-dumb server with time+URL index and replay, i.e. a local [Wayback Machine](https://web.archive.org/).\n- Full text indexing and search.\n- Converter from WRR to WARC.\n- Converter from PCAP ito WRR.\n\n# Usage\n\n## wrrarms\n\nA tool to pretty-print, compute and print values from, search, organize (programmatically rename/move/symlink/hardlink files), import, export, (WIP: check, deduplicate, and edit) pWebArc WRR (WEBREQRES, Web REQuest+RESponse) archive files.\n\nTerminology: a `reqres` (`Reqres` when a Python type) is an instance of a structure representing HTTP request+response pair with some additional metadata.\n\n- options:\n - `--version`\n : show program's version number and exit\n - `-h, --help`\n : show this help message and exit\n - `--markdown`\n : show help messages formatted in Markdown\n\n- subcommands:\n - `{pprint,get,run,stream,find,organize,import,export}`\n - `pprint`\n : pretty-print given WRR files\n - `get`\n : print values produced by computing given expressions on a given WRR file\n - `run`\n : spawn a process with generated temporary files produced by given expressions computed on given WRR files as arguments\n - `stream`\n : produce a stream of structured lists containing values produced by computing given expressions on given WRR files, a generalized `wrrarms get`\n - `find`\n : print paths of WRR files matching specified criteria\n - `organize`\n : programmatically rename/move/hardlink/symlink WRR files based on their contents\n - `import`\n : convert other HTTP archive formats into WRR\n - `export`\n : convert WRR archives into other formats\n\n### wrrarms pprint\n\nPretty-print given WRR files to stdout.\n\n- positional arguments:\n - `PATH`\n : inputs, can be a mix of files and directories (which will be traversed recursively)\n\n- options:\n - `-u, --unabridged`\n : print all data in full\n - `--abridged`\n : shorten long strings for brevity (useful when you want to visually scan through batch data dumps) (default)\n - `--stdin0`\n : read zero-terminated `PATH`s from stdin, these will be processed after `PATH`s specified as command-line arguments\n\n- error handling:\n - `--errors {fail,skip,ignore}`\n : when an error occurs:\n - `fail`: report failure and stop the execution (default)\n - `skip`: report failure but skip the reqres that produced it from the output and continue\n - `ignore`: `skip`, but don't report the failure\n\n- filters:\n - `--or EXPR`\n : only print reqres which match any of these expressions...\n - `--and EXPR`\n : ... and all of these expressions, both can be specified multiple times, both use the same expression format as `wrrarms get --expr`, which see\n\n- MIME type sniffing:\n - `--naive`\n : populate \"potentially\" lists like `wrrarms (get|run|export) --expr '(request|response).body|eb|scrub \\2 defaults'` does; default\n - `--paranoid`\n : populate \"potentially\" lists in the output using paranoid MIME type sniffing like `wrrarms (get|run|export) --expr '(request|response).body|eb|scrub \\2 +paranoid'` does; this exists to answer \"Hey! Why did it censor out my data?!\" questions\n\n- file system path ordering:\n - `--paths-given-order`\n : `argv` and `--stdin0` `PATH`s are processed in the order they are given (default)\n - `--paths-sorted`\n : `argv` and `--stdin0` `PATH`s are processed in lexicographic order\n - `--paths-reversed`\n : `argv` and `--stdin0` `PATH`s are processed in reverse lexicographic order\n - `--walk-fs-order`\n : recursive file system walk is done in the order `readdir(2)` gives results\n - `--walk-sorted`\n : recursive file system walk is done in lexicographic order (default)\n - `--walk-reversed`\n : recursive file system walk is done in reverse lexicographic order\n\n### wrrarms get\n\nCompute output values by evaluating expressions `EXPR`s on a given reqres stored at `PATH`, then print them to stdout terminating each value as specified.\n\n- positional arguments:\n - `PATH`\n : input WRR file path\n\n- expression evaluation:\n - `-e EXPR, --expr EXPR`\n : an expression to compute; can be specified multiple times in which case computed outputs will be printed sequentially; see also \"output\" options below; (default: `response.body|eb`); each EXPR describes a state-transformer (pipeline) which starts from value `None` and evaluates a script built from the following:\n - constants and functions:\n - `es`: replace `None` value with an empty string `\"\"`\n - `eb`: replace `None` value with an empty byte string `b\"\"`\n - `false`: replace `None` value with `False`\n - `true`: replace `None` value with `True`\n - `missing`: `True` if the value is `None`\n - `0`: replace `None` value with `0`\n - `1`: replace `None` value with `1`\n - `not`: apply logical `not` to value\n - `len`: apply `len` to value\n - `str`: cast value to `str` or fail\n - `bytes`: cast value to `bytes` or fail\n - `bool`: cast value to `bool` or fail\n - `int`: cast value to `int` or fail\n - `float`: cast value to `float` or fail\n - `echo`: replace the value with the given string\n - `quote`: URL-percent-encoding quote value\n - `quote_plus`: URL-percent-encoding quote value and replace spaces with `+` symbols\n - `unquote`: URL-percent-encoding unquote value\n - `unquote_plus`: URL-percent-encoding unquote value and replace `+` symbols with spaces\n - `to_ascii`: encode `str` value into `bytes` with \"ascii\" codec\n - `to_utf8`: encode `str` value into `bytes` with \"utf-8\" codec\n - `sha256`: replace `bytes` value with its `sha256` hex digest (`hex(sha256(value))`)\n - `==`: apply `== arg`, `arg` is cast to the same type as the current value\n - `!=`: apply `!= arg`, similarly\n - `<`: apply `< arg`, similarly\n - `<=`: apply `<= arg`, similarly\n - `>`: apply `> arg`, similarly\n - `>=`: apply `>= arg`, similarly\n - `add_prefix`: add prefix to the current value\n - `add_suffix`: add suffix to the current value\n - `take_prefix`: take first `arg` characters or list elements from the current value\n - `take_suffix`: take last `arg` characters or list elements from the current value\n - `abbrev`: leave the current value as-is if if its length is less or equal than `arg` characters, otherwise take first `arg/2` followed by last `arg/2` characters\n - `abbrev_each`: `abbrev arg` each element in a value `list`\n - `replace`: replace all occurences of the first argument in the current value with the second argument, casts arguments to the same type as the current value\n - `pp_to_path`: encode `path_parts` `list` into a POSIX path, quoting as little as needed\n - `qsl_urlencode`: encode parsed `query` `list` into a URL's query component `str`\n - `qsl_to_path`: encode `query` `list` into a POSIX path, quoting as little as needed\n - `scrub`: scrub the value by optionally rewriting links and/or removing dynamic content from it; what gets done depends on `--remap-*` command line options, the MIME type of the value itself, and the scrubbing options described below; this fuction takes two arguments:\n - the first must be either of `request|response`, it controls which HTTP headers `scrub` should inspect to help it detect the MIME type;\n - the second is either `defaults` or \",\"-separated string of `(+|-)(paranoid|unknown|jumps|actions|srcs|all_refs|scripts|iframes|styles|iepragmas|prefetches|tracking|dyndoc|all_dyns|verbose|whitespace|optional_tags|indent|pretty|debug)` tokens which control the scrubbing behaviour:\n - `+paranoid` will assume the server is lying in its `Content-Type` and `X-Content-Type-Options` HTTP headers, sniff the contents of `(request|response).body` to determine what it actually contains regardless of what the server said, and then use the most paranoid interpretation of both the HTTP headers and the sniffed possible MIME types to decide what should be kept and what sholuld be removed by the options below; i.e., this will make `-unknown`, `-scripts`, and `-styles` options below to censor out more things, in particular, at the moment, most plain text files will get censored out as potential JavaScript; the default is `-paranoid`;\n - `(+|-)unknown` controls if the data with unknown content types should passed to the output unchanged or censored out (respectively); the default is `+unknown`, which will keep data of unknown content types as-is;\n - `(+|-)(jumps|actions|srcs)` control which kinds of references to other documents should be remapped or censored out (respectively); i.e. it controls whether jump-links (HTML `a href`, `area href`, and similar), action-links (HTML `a ping`, `form action`, and similar), and/or resource references (HTML `img src`, `iframe src`, CSS `url` references, and similar) should be remapped using the specified `--remap-*` option (which see) or censored out similarly to how `--remap-void` will do it; the default is `+jumps,-actions,-srcs` which will produce a self-contained result that can be fed into another tool --- be it a web browser or `pandoc` --- without that tool trying to access the Internet;\n - `(+|-)all_refs` is equivalent to enabling or disabling all of the above options simultaneously;\n - `(+|-)(scripts|iframes|styles|iepragmas|prefetches|tracking)` control which things should be kept or censored out w.r.t. to HTML, CSS, and JavaScript, i.e. it controls whether JavaScript (both separate files and HTML tags and attributes), `<iframe>` HTML tags, CSS (both separate files and HTML tags and attributes; why? because CSS is Turing-complete), HTML Internet-Explorer pragmas, HTML content prefetch `link` tags, and other tracking HTML tags and attributes (like `a ping` attributes), should be respectively kept in or censored out from the input; the default is `-scripts,-iframes,-styles,-iepragmas,-prefetches,-tracking` which ensures the result will not produce any prefetch and tracking requests when loaded in a web browser, and that the whole result is simple data, not a program in some Turing-complete language, thus making it safe to feed the result to other tools too smart for their own users' good;\n - `(+|-)all_dyns` is equivalent to enabling or disabling all of the above (`scripts|...`) options simultaneously;\n - `(+|-)verbose` controls whether tag censoring controlled by the above options is to be reported in the output (as comments) or stuff should be wiped from existence without evidence instead; the default is `-verbose`;\n - `(+|-)whitespace` controls whether HTML renderer should keep the original HTML whitespace as-is or collapse it away (respectively); the default is `-whitespace`;\n - `(+|-)optional_tags` controls whether HTML renderer should put optional HTML tags into the output or skip them (respectively); the default is `+optional_tags` (because many tools fail to parse minimized HTML properly);\n - `(+|-)indent` controls whether HTML renderer should indent HTML elements (where whitespace placement in the original markup allows for it) or not (respectively); the default is `-indent`;\n - `+pretty` is an alias for `+verbose,-whitespace,+indent` which produces the prettiest possible human-readable output that keeps the original whitespace semantics; `-pretty` is an alias for `+verbose,+whitespace,-indent` which produces the approximation of the original markup with censoring applied; neither is the default;\n - `+debug` is an alias for `+pretty` that also uses a much more aggressive version of `indent` that ignores the semantics of original whitespace placement, i.e. it will indent `<p>not<em>sep</em>arated</p>` as if there was whitespace before and after `p`, `em`, `/em`, and `/p` tags; this is useful for debugging custom mutations; `-debug` is noop, which is the default;\n - reqres fields, these work the same way as constants above, i.e. they replace current value of `None` with field's value, if reqres is missing the field in question, which could happen for `response*` fields, the result is `None`:\n - `version`: WEBREQRES format version; int\n - `source`: `+`-separated list of applications that produced this reqres; str\n - `protocol`: protocol; e.g. `\"HTTP/1.1\"`, `\"HTTP/2.0\"`; str\n - `request.started_at`: request start time in seconds since 1970-01-01 00:00; Epoch\n - `request.method`: request HTTP method; e.g. `\"GET\"`, `\"POST\"`, etc; str\n - `request.url`: request URL, including the fragment/hash part; str\n - `request.headers`: request headers; list[tuple[str, bytes]]\n - `request.complete`: is request body complete?; bool\n - `request.body`: request body; bytes\n - `response.started_at`: response start time in seconds since 1970-01-01 00:00; Epoch\n - `response.code`: HTTP response code; e.g. `200`, `404`, etc; int\n - `response.reason`: HTTP response reason; e.g. `\"OK\"`, `\"Not Found\"`, etc; usually empty for Chromium and filled for Firefox; str\n - `response.headers`: response headers; list[tuple[str, bytes]]\n - `response.complete`: is response body complete?; bool\n - `response.body`: response body; Firefox gives raw bytes, Chromium gives UTF-8 encoded strings; bytes | str\n - `finished_at`: request completion time in seconds since 1970-01-01 00:00; Epoch\n - `websocket`: a list of WebSocket frames\n - derived attributes:\n - `fs_path`: file system path for the WRR file containing this reqres; str | bytes | None\n - `qtime`: aliast for `request.started_at`; mnemonic: \"reQuest TIME\"; seconds since UNIX epoch; decimal float\n - `qtime_ms`: `qtime` in milliseconds rounded down to nearest integer; milliseconds since UNIX epoch; int\n - `qtime_msq`: three least significant digits of `qtime_ms`; int\n - `qyear`: year number of `gmtime(qtime)` (UTC year number of `qtime`); int\n - `qmonth`: month number of `gmtime(qtime)`; int\n - `qday`: day of the month of `gmtime(qtime)`; int\n - `qhour`: hour of `gmtime(qtime)` in 24h format; int\n - `qminute`: minute of `gmtime(qtime)`; int\n - `qsecond`: second of `gmtime(qtime)`; int\n - `stime`: `response.started_at` if there was a response, `finished_at` otherwise; mnemonic: \"reSponse TIME\"; seconds since UNIX epoch; decimal float\n - `stime_ms`: `stime` in milliseconds rounded down to nearest integer; milliseconds since UNIX epoch, int\n - `stime_msq`: three least significant digits of `stime_msq`; int\n - `syear`: similar to `syear`, but for `stime`; int\n - `smonth`: similar to `smonth`, but for `stime`; int\n - `sday`: similar to `sday`, but for `stime`; int\n - `shour`: similar to `shour`, but for `stime`; int\n - `sminute`: similar to `sminute`, but for `stime`; int\n - `ssecond`: similar to `ssecond`, but for `stime`; int\n - `ftime`: aliast for `finished_at`; seconds since UNIX epoch; decimal float\n - `ftime_ms`: `ftime` in milliseconds rounded down to nearest integer; milliseconds since UNIX epoch; int\n - `ftime_msq`: three least significant digits of `ftime_msq`; int\n - `fyear`: similar to `syear`, but for `ftime`; int\n - `fmonth`: similar to `smonth`, but for `ftime`; int\n - `fday`: similar to `sday`, but for `ftime`; int\n - `fhour`: similar to `shour`, but for `ftime`; int\n - `fminute`: similar to `sminute`, but for `ftime`; int\n - `fsecond`: similar to `ssecond`, but for `ftime`; int\n - `status`: `\"NR\"` if there was no response, `str(response.code) + \"C\"` if response was complete, `str(response.code) + \"N\"` otherwise; str\n - `method`: aliast for `request.method`; str\n - `raw_url`: aliast for `request.url`; str\n - `net_url`: `raw_url` with Punycode UTS46 IDNA encoded hostname, unsafe characters quoted, and without the fragment/hash part; this is the URL that actually gets sent to the server; str\n - `pretty_url`: `raw_url`, but using `hostname`, `mq_path`, and `mq_query`; str\n - `pretty_nurl`: `raw_url`, but using `hostname`, `mq_path`, and `mq_nquery`; str\n - `scheme`: scheme part of `raw_url`; e.g. `http`, `https`, etc; str\n - `raw_hostname`: hostname part of `raw_url` as it is recorded in the reqres; str\n - `net_hostname`: hostname part of `raw_url`, encoded as Punycode UTS46 IDNA; this is what actually gets sent to the server; ASCII str\n - `hostname`: `net_hostname` decoded back into UNICODE; this is the canonical hostname representation for which IDNA-encoding and decoding are bijective; UNICODE str\n - `rhostname`: `hostname` with the order of its parts reversed; e.g. `\"www.example.org\"` -> `\"com.example.www\"`; str\n - `port`: port part of `raw_url`; str\n - `netloc`: netloc part of `raw_url`; i.e., in the most general case, `<username>:<password>@<hostname>:<port>`; str\n - `raw_path`: raw path part of `raw_url` as it is recorded is the reqres; e.g. `\"https://www.example.org\"` -> `\"\"`, `\"https://www.example.org/\"` -> `\"/\"`, `\"https://www.example.org/index.html\"` -> `\"/index.html\"`; str\n - `path_parts`: component-wise unquoted \"/\"-split `raw_path` with empty components removed and dots and double dots interpreted away; e.g. `\"https://www.example.org\"` -> `[]`, `\"https://www.example.org/\"` -> `[]`, `\"https://www.example.org/index.html\"` -> `[\"index.html\"]` , `\"https://www.example.org/skipped/.//../used/\"` -> `[\"used\"]`; list[str]\n - `mq_path`: `path_parts` turned back into a minimally-quoted string; str\n - `filepath_parts`: `path_parts` transformed into components usable as an exportable file name; i.e. `path_parts` with an optional additional `\"index\"` appended, depending on `raw_url` and `response` MIME type; extension will be stored separately in `filepath_ext`; e.g. for HTML documents `\"https://www.example.org/\"` -> `[\"index\"]`, `\"https://www.example.org/test.html\"` -> `[\"test\"]`, `\"https://www.example.org/test\"` -> `[\"test\", \"index\"]`, `\"https://www.example.org/test.json\"` -> `[\"test.json\", \"index\"]`, but if it has a JSON MIME type then `\"https://www.example.org/test.json\"` -> `[\"test\"]` (and `filepath_ext` will be set to `\".json\"`); this is similar to what `wget -mpk` does, but a bit smarter; list[str]\n - `filepath_ext`: extension of the last component of `filepath_parts` for recognized MIME types, `\".data\"` otherwise; str\n - `raw_query`: query part of `raw_url` (i.e. everything after the `?` character and before the `#` character) as it is recorded in the reqres; str\n - `query_parts`: parsed (and component-wise unquoted) `raw_query`; list[tuple[str, str]]\n - `query_ne_parts`: `query_parts` with empty query parameters removed; list[tuple[str, str]]\n - `mq_query`: `query_parts` turned back into a minimally-quoted string; str\n - `mq_nquery`: `query_ne_parts` turned back into a minimally-quoted string; str\n - `oqm`: optional query mark: `?` character if `query` is non-empty, an empty string otherwise; str\n - `fragment`: fragment (hash) part of the url; str\n - `ofm`: optional fragment mark: `#` character if `fragment` is non-empty, an empty string otherwise; str\n - a compound expression built by piping (`|`) the above, for example:\n - `response.body|eb` (the default for `get`) will print raw `response.body` or an empty byte string, if there was no response;\n - `response.body|eb|scrub response defaults` will take the above value, `scrub` it using default content scrubbing settings which will censor out all action and resource reference URLs;\n - `response.body|eb|scrub response +all_refs,-actions` (the default for `export`) will remap all `href` jump-links and `src` resource references to local files while still censoring out all action URLs (since those don't make sense for a static mirror);\n - `response.complete` will print the value of `response.complete` or `None`, if there was no response;\n - `response.complete|false` will print `response.complete` or `False`;\n - `net_url|to_ascii|sha256` will print `sha256` hash of the URL that was actually sent over the network;\n - `net_url|to_ascii|sha256|take_prefix 4` will print the first 4 characters of the above;\n - `path_parts|take_prefix 3|pp_to_path` will print first 3 path components of the URL, minimally quoted to be used as a path;\n - `query_ne_parts|take_prefix 3|qsl_to_path|abbrev 128` will print first 3 non-empty query parameters of the URL, abbreviated to 128 characters or less, minimally quoted to be used as a path;\n\n- URL remapping, used by `scrub` `--expr` atom:\n - `--remap-id`\n : remap all URLs with an identity function; i.e. don't remap anything (default)\n - `--remap-void`\n : remap all jump-link and action URLs to `javascript:void(0)` and all resource URLs into empty `data:` URLs; the result will be self-contained\n\n- output:\n - `--not-separated`\n : don't separate output values with anything, just concatenate them\n - `-l, --lf-separated`\n : separate output values with `\\n` (LF) newline characters (default)\n - `-z, --zero-separated`\n : separate output values with `\\0` (NUL) bytes\n\n### wrrarms run\n\nCompute output values by evaluating expressions `EXPR`s for each of `NUM` reqres stored at `PATH`s, dump the results into into newly generated temporary files terminating each value as specified, spawn a given `COMMAND` with given arguments `ARG`s and the resulting temporary file paths appended as the last `NUM` arguments, wait for it to finish, delete the temporary files, exit with the return code of the spawned process.\n\n- positional arguments:\n - `COMMAND`\n : command to spawn\n - `ARG`\n : additional arguments to give to the `COMMAND`\n - `PATH`\n : input WRR file paths to be mapped into new temporary files\n\n- options:\n - `-n NUM, --num-args NUM`\n : number of `PATH`s (default: `1`)\n\n- expression evaluation:\n - `-e EXPR, --expr EXPR`\n : see `wrrarms get`\n\n- URL remapping, used by `scrub` `--expr` atom:\n - `--remap-id`\n : remap all URLs with an identity function; i.e. don't remap anything (default)\n - `--remap-void`\n : remap all jump-link and action URLs to `javascript:void(0)` and all resource URLs into empty `data:` URLs; the result will be self-contained\n\n- output:\n - `--not-separated`\n : don't separate output values with anything, just concatenate them\n - `-l, --lf-separated`\n : separate output values with `\\n` (LF) newline characters (default)\n - `-z, --zero-separated`\n : separate output values with `\\0` (NUL) bytes\n\n### wrrarms stream\n\nCompute given expressions for each of given WRR files, encode them into a requested format, and print the result to stdout.\n\n- positional arguments:\n - `PATH`\n : inputs, can be a mix of files and directories (which will be traversed recursively)\n\n- options:\n - `-u, --unabridged`\n : print all data in full\n - `--abridged`\n : shorten long strings for brevity (useful when you want to visually scan through batch data dumps) (default)\n - `--format {py,cbor,json,raw}`\n : generate output in:\n - py: Pythonic Object Representation aka `repr` (default)\n - cbor: CBOR (RFC8949)\n - json: JavaScript Object Notation aka JSON; **binary data can't be represented, UNICODE replacement characters will be used**\n - raw: concatenate raw values; termination is controlled by `*-terminated` options\n - `--stdin0`\n : read zero-terminated `PATH`s from stdin, these will be processed after `PATH`s specified as command-line arguments\n\n- error handling:\n - `--errors {fail,skip,ignore}`\n : when an error occurs:\n - `fail`: report failure and stop the execution (default)\n - `skip`: report failure but skip the reqres that produced it from the output and continue\n - `ignore`: `skip`, but don't report the failure\n\n- filters:\n - `--or EXPR`\n : only print reqres which match any of these expressions...\n - `--and EXPR`\n : ... and all of these expressions, both can be specified multiple times, both use the same expression format as `wrrarms get --expr`, which see\n\n- expression evaluation:\n - `-e EXPR, --expr EXPR`\n : an expression to compute, see `wrrarms get --expr` for more info on expression format; can be specified multiple times; the default is `.` which will dump the whole reqres structure\n\n- URL remapping, used by `scrub` `--expr` atom:\n - `--remap-id`\n : remap all URLs with an identity function; i.e. don't remap anything (default)\n - `--remap-void`\n : remap all jump-link and action URLs to `javascript:void(0)` and all resource URLs into empty `data:` URLs; the result will be self-contained\n\n- `--format=raw` output:\n - `--not-terminated`\n : don't terminate `--format=raw` output values with anything, just concatenate them\n - `-l, --lf-terminated`\n : terminate `--format=raw` output values with `\\n` (LF) newline characters (default)\n - `-z, --zero-terminated`\n : terminate `--format=raw` output values with `\\0` (NUL) bytes\n\n- file system path ordering:\n - `--paths-given-order`\n : `argv` and `--stdin0` `PATH`s are processed in the order they are given (default)\n - `--paths-sorted`\n : `argv` and `--stdin0` `PATH`s are processed in lexicographic order\n - `--paths-reversed`\n : `argv` and `--stdin0` `PATH`s are processed in reverse lexicographic order\n - `--walk-fs-order`\n : recursive file system walk is done in the order `readdir(2)` gives results\n - `--walk-sorted`\n : recursive file system walk is done in lexicographic order (default)\n - `--walk-reversed`\n : recursive file system walk is done in reverse lexicographic order\n\n### wrrarms find\n\nPrint paths of WRR files matching specified criteria.\n\n- positional arguments:\n - `PATH`\n : inputs, can be a mix of files and directories (which will be traversed recursively)\n\n- options:\n - `--stdin0`\n : read zero-terminated `PATH`s from stdin, these will be processed after `PATH`s specified as command-line arguments\n\n- error handling:\n - `--errors {fail,skip,ignore}`\n : when an error occurs:\n - `fail`: report failure and stop the execution (default)\n - `skip`: report failure but skip the reqres that produced it from the output and continue\n - `ignore`: `skip`, but don't report the failure\n\n- filters:\n - `--or EXPR`\n : only output paths to reqres which match any of these expressions...\n - `--and EXPR`\n : ... and all of these expressions, both can be specified multiple times, both use the same expression format as `wrrarms get --expr`, which see\n\n- output:\n - `-l, --lf-terminated`\n : terminate output absolute paths of matching WRR files with `\\n` (LF) newline characters (default)\n - `-z, --zero-terminated`\n : terminate output absolute paths of matching WRR files with `\\0` (NUL) bytes\n\n- file system path ordering:\n - `--paths-given-order`\n : `argv` and `--stdin0` `PATH`s are processed in the order they are given (default)\n - `--paths-sorted`\n : `argv` and `--stdin0` `PATH`s are processed in lexicographic order\n - `--paths-reversed`\n : `argv` and `--stdin0` `PATH`s are processed in reverse lexicographic order\n - `--walk-fs-order`\n : recursive file system walk is done in the order `readdir(2)` gives results\n - `--walk-sorted`\n : recursive file system walk is done in lexicographic order (default)\n - `--walk-reversed`\n : recursive file system walk is done in reverse lexicographic order\n\n### wrrarms organize\n\nParse given WRR files into their respective reqres and then rename/move/hardlink/symlink each file to `DESTINATION` with the new path derived from each reqres' metadata.\n\nOperations that could lead to accidental data loss are not permitted.\nE.g. `wrrarms organize --move` will not overwrite any files, which is why the default `--output` contains `%(num)d`.\n\n- positional arguments:\n - `PATH`\n : inputs, can be a mix of files and directories (which will be traversed recursively)\n\n- options:\n - `--dry-run`\n : perform a trial run without actually performing any changes\n - `-q, --quiet`\n : don't log computed updates to stderr\n - `-t DESTINATION, --to DESTINATION`\n : destination directory, when unset each source `PATH` must be a directory which will be treated as its own `DESTINATION`\n - `-o FORMAT, --output FORMAT`\n : format describing generated output paths, an alias name or \"format:\" followed by a custom pythonic %-substitution string:\n - available aliases and corresponding %-substitutions:\n - `default` : `%(syear)d/%(smonth)02d/%(sday)02d/%(shour)02d%(sminute)02d%(ssecond)02d%(stime_msq)03d_%(qtime_ms)s_%(method)s_%(net_url|to_ascii|sha256|take_prefix 4)s_%(status)s_%(hostname)s.%(num)d` (default)\n - `https://example.org` -> `1970/01/01/001640000_0_GET_50d7_200C_example.org.0`\n - `https://example.org/` -> `1970/01/01/001640000_0_GET_8198_200C_example.org.0`\n - `https://example.org/index.html` -> `1970/01/01/001640000_0_GET_f0dc_200C_example.org.0`\n - `https://example.org/media` -> `1970/01/01/001640000_0_GET_086d_200C_example.org.0`\n - `https://example.org/media/` -> `1970/01/01/001640000_0_GET_3fbb_200C_example.org.0`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `1970/01/01/001640000_0_GET_5658_200C_example.org.0`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `1970/01/01/001640000_0_GET_4f11_200C_k\u00f6nigsg\u00e4\u00dfchen.example.org.0`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `1970/01/01/001640000_0_GET_c4ae_200C_\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org.0`\n - `short` : `%(syear)d/%(smonth)02d/%(sday)02d/%(stime_ms)d_%(qtime_ms)s.%(num)d`\n - `https://example.org`, `https://example.org/`, `https://example.org/index.html`, `https://example.org/media`, `https://example.org/media/`, `https://example.org/view?one=1&two=2&three=&three=3#fragment`, `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html`, `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `1970/01/01/1000000_0.0`\n - `surl` : `%(scheme)s/%(netloc)s/%(mq_path)s%(oqm)s%(mq_query)s`\n - `https://example.org`, `https://example.org/` -> `https/example.org/`\n - `https://example.org/index.html` -> `https/example.org/index.html`\n - `https://example.org/media`, `https://example.org/media/` -> `https/example.org/media`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/example.org/view?one=1&two=2&three&three=3`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `https/k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is`\n - `surl_msn` : `%(scheme)s/%(netloc)s/%(mq_path)s%(oqm)s%(mq_query)s_%(method)s_%(status)s.%(num)d`\n - `https://example.org`, `https://example.org/` -> `https/example.org/_GET_200C.0`\n - `https://example.org/index.html` -> `https/example.org/index.html_GET_200C.0`\n - `https://example.org/media`, `https://example.org/media/` -> `https/example.org/media_GET_200C.0`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/example.org/view?one=1&two=2&three&three=3_GET_200C.0`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `https/k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html_GET_200C.0`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is_GET_200C.0`\n - `shupq` : `%(scheme)s/%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_query|abbrev 120)s%(filepath_ext)s`\n - `https://example.org`, `https://example.org/` -> `https/example.org/index.htm`\n - `https://example.org/index.html` -> `https/example.org/index.html`\n - `https://example.org/media`, `https://example.org/media/` -> `https/example.org/media/index.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/example.org/view/index?one=1&two=2&three&three=3.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `https/k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/index.htm`\n - `shupq_msn` : `%(scheme)s/%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_query|abbrev 100)s_%(method)s_%(status)s.%(num)d%(filepath_ext)s`\n - `https://example.org`, `https://example.org/` -> `https/example.org/index_GET_200C.0.htm`\n - `https://example.org/index.html` -> `https/example.org/index_GET_200C.0.html`\n - `https://example.org/media`, `https://example.org/media/` -> `https/example.org/media/index_GET_200C.0.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/example.org/view/index?one=1&two=2&three&three=3_GET_200C.0.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `https/k\u00f6nigsg\u00e4\u00dfchen.example.org/index_GET_200C.0.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/index_GET_200C.0.htm`\n - `shupnq` : `%(scheme)s/%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 120)s%(filepath_ext)s`\n - `https://example.org`, `https://example.org/` -> `https/example.org/index.htm`\n - `https://example.org/index.html` -> `https/example.org/index.html`\n - `https://example.org/media`, `https://example.org/media/` -> `https/example.org/media/index.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/example.org/view/index?one=1&two=2&three=3.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `https/k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/index.htm`\n - `shupnq_msn` : `%(scheme)s/%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 100)s_%(method)s_%(status)s.%(num)d%(filepath_ext)s`\n - `https://example.org`, `https://example.org/` -> `https/example.org/index_GET_200C.0.htm`\n - `https://example.org/index.html` -> `https/example.org/index_GET_200C.0.html`\n - `https://example.org/media`, `https://example.org/media/` -> `https/example.org/media/index_GET_200C.0.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/example.org/view/index?one=1&two=2&three=3_GET_200C.0.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `https/k\u00f6nigsg\u00e4\u00dfchen.example.org/index_GET_200C.0.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/index_GET_200C.0.htm`\n - `shupnq_mhs` : `%(scheme)s/%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 120)s_%(method)s_%(net_url|to_ascii|sha256|take_prefix 4)s_%(status)s%(filepath_ext)s`\n - `https://example.org` -> `https/example.org/index_GET_50d7_200C.htm`\n - `https://example.org/` -> `https/example.org/index_GET_8198_200C.htm`\n - `https://example.org/index.html` -> `https/example.org/index_GET_f0dc_200C.html`\n - `https://example.org/media` -> `https/example.org/media/index_GET_086d_200C.htm`\n - `https://example.org/media/` -> `https/example.org/media/index_GET_3fbb_200C.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/example.org/view/index?one=1&two=2&three=3_GET_5658_200C.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `https/k\u00f6nigsg\u00e4\u00dfchen.example.org/index_GET_4f11_200C.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/index_GET_c4ae_200C.htm`\n - `shupnq_mhsn` : `%(scheme)s/%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 100)s_%(method)s_%(net_url|to_ascii|sha256|take_prefix 4)s_%(status)s.%(num)d%(filepath_ext)s`\n - `https://example.org` -> `https/example.org/index_GET_50d7_200C.0.htm`\n - `https://example.org/` -> `https/example.org/index_GET_8198_200C.0.htm`\n - `https://example.org/index.html` -> `https/example.org/index_GET_f0dc_200C.0.html`\n - `https://example.org/media` -> `https/example.org/media/index_GET_086d_200C.0.htm`\n - `https://example.org/media/` -> `https/example.org/media/index_GET_3fbb_200C.0.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/example.org/view/index?one=1&two=2&three=3_GET_5658_200C.0.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `https/k\u00f6nigsg\u00e4\u00dfchen.example.org/index_GET_4f11_200C.0.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/index_GET_c4ae_200C.0.htm`\n - `srhupq` : `%(scheme)s/%(rhostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_query|abbrev 120)s%(filepath_ext)s`\n - `https://example.org`, `https://example.org/` -> `https/org.example/index.htm`\n - `https://example.org/index.html` -> `https/org.example/index.html`\n - `https://example.org/media`, `https://example.org/media/` -> `https/org.example/media/index.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/org.example/view/index?one=1&two=2&three&three=3.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `https/org.example.k\u00f6nigsg\u00e4\u00dfchen/index.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/org.example.\u3067\u3059\u306e.\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/index.htm`\n - `srhupq_msn` : `%(scheme)s/%(rhostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_query|abbrev 100)s_%(method)s_%(status)s.%(num)d%(filepath_ext)s`\n - `https://example.org`, `https://example.org/` -> `https/org.example/index_GET_200C.0.htm`\n - `https://example.org/index.html` -> `https/org.example/index_GET_200C.0.html`\n - `https://example.org/media`, `https://example.org/media/` -> `https/org.example/media/index_GET_200C.0.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/org.example/view/index?one=1&two=2&three&three=3_GET_200C.0.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `https/org.example.k\u00f6nigsg\u00e4\u00dfchen/index_GET_200C.0.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/org.example.\u3067\u3059\u306e.\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/index_GET_200C.0.htm`\n - `srhupnq` : `%(scheme)s/%(rhostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 120)s%(filepath_ext)s`\n - `https://example.org`, `https://example.org/` -> `https/org.example/index.htm`\n - `https://example.org/index.html` -> `https/org.example/index.html`\n - `https://example.org/media`, `https://example.org/media/` -> `https/org.example/media/index.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/org.example/view/index?one=1&two=2&three=3.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `https/org.example.k\u00f6nigsg\u00e4\u00dfchen/index.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/org.example.\u3067\u3059\u306e.\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/index.htm`\n - `srhupnq_msn` : `%(scheme)s/%(rhostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 100)s_%(method)s_%(status)s.%(num)d%(filepath_ext)s`\n - `https://example.org`, `https://example.org/` -> `https/org.example/index_GET_200C.0.htm`\n - `https://example.org/index.html` -> `https/org.example/index_GET_200C.0.html`\n - `https://example.org/media`, `https://example.org/media/` -> `https/org.example/media/index_GET_200C.0.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/org.example/view/index?one=1&two=2&three=3_GET_200C.0.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `https/org.example.k\u00f6nigsg\u00e4\u00dfchen/index_GET_200C.0.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/org.example.\u3067\u3059\u306e.\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/index_GET_200C.0.htm`\n - `srhupnq_mhs` : `%(scheme)s/%(rhostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 120)s_%(method)s_%(net_url|to_ascii|sha256|take_prefix 4)s_%(status)s%(filepath_ext)s`\n - `https://example.org` -> `https/org.example/index_GET_50d7_200C.htm`\n - `https://example.org/` -> `https/org.example/index_GET_8198_200C.htm`\n - `https://example.org/index.html` -> `https/org.example/index_GET_f0dc_200C.html`\n - `https://example.org/media` -> `https/org.example/media/index_GET_086d_200C.htm`\n - `https://example.org/media/` -> `https/org.example/media/index_GET_3fbb_200C.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/org.example/view/index?one=1&two=2&three=3_GET_5658_200C.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `https/org.example.k\u00f6nigsg\u00e4\u00dfchen/index_GET_4f11_200C.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/org.example.\u3067\u3059\u306e.\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/index_GET_c4ae_200C.htm`\n - `srhupnq_mhsn`: `%(scheme)s/%(rhostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 100)s_%(method)s_%(net_url|to_ascii|sha256|take_prefix 4)s_%(status)s.%(num)d%(filepath_ext)s`\n - `https://example.org` -> `https/org.example/index_GET_50d7_200C.0.htm`\n - `https://example.org/` -> `https/org.example/index_GET_8198_200C.0.htm`\n - `https://example.org/index.html` -> `https/org.example/index_GET_f0dc_200C.0.html`\n - `https://example.org/media` -> `https/org.example/media/index_GET_086d_200C.0.htm`\n - `https://example.org/media/` -> `https/org.example/media/index_GET_3fbb_200C.0.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `https/org.example/view/index?one=1&two=2&three=3_GET_5658_200C.0.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `https/org.example.k\u00f6nigsg\u00e4\u00dfchen/index_GET_4f11_200C.0.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `https/org.example.\u3067\u3059\u306e.\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/index_GET_c4ae_200C.0.htm`\n - `url` : `%(netloc)s/%(mq_path)s%(oqm)s%(mq_query)s`\n - `https://example.org`, `https://example.org/` -> `example.org/`\n - `https://example.org/index.html` -> `example.org/index.html`\n - `https://example.org/media`, `https://example.org/media/` -> `example.org/media`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `example.org/view?one=1&two=2&three&three=3`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is`\n - `url_msn` : `%(netloc)s/%(mq_path)s%(oqm)s%(mq_query)s_%(method)s_%(status)s.%(num)d`\n - `https://example.org`, `https://example.org/` -> `example.org/_GET_200C.0`\n - `https://example.org/index.html` -> `example.org/index.html_GET_200C.0`\n - `https://example.org/media`, `https://example.org/media/` -> `example.org/media_GET_200C.0`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `example.org/view?one=1&two=2&three&three=3_GET_200C.0`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html_GET_200C.0`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is_GET_200C.0`\n - `hupq` : `%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_query|abbrev 120)s%(filepath_ext)s`\n - `https://example.org`, `https://example.org/` -> `example.org/index.htm`\n - `https://example.org/index.html` -> `example.org/index.html`\n - `https://example.org/media`, `https://example.org/media/` -> `example.org/media/index.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `example.org/view/index?one=1&two=2&three&three=3.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/index.htm`\n - `hupq_msn` : `%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_query|abbrev 100)s_%(method)s_%(status)s.%(num)d%(filepath_ext)s`\n - `https://example.org`, `https://example.org/` -> `example.org/index_GET_200C.0.htm`\n - `https://example.org/index.html` -> `example.org/index_GET_200C.0.html`\n - `https://example.org/media`, `https://example.org/media/` -> `example.org/media/index_GET_200C.0.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `example.org/view/index?one=1&two=2&three&three=3_GET_200C.0.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `k\u00f6nigsg\u00e4\u00dfchen.example.org/index_GET_200C.0.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/index_GET_200C.0.htm`\n - `hupnq` : `%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 120)s%(filepath_ext)s`\n - `https://example.org`, `https://example.org/` -> `example.org/index.htm`\n - `https://example.org/index.html` -> `example.org/index.html`\n - `https://example.org/media`, `https://example.org/media/` -> `example.org/media/index.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `example.org/view/index?one=1&two=2&three=3.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/index.htm`\n - `hupnq_msn` : `%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 100)s_%(method)s_%(status)s.%(num)d%(filepath_ext)s`\n - `https://example.org`, `https://example.org/` -> `example.org/index_GET_200C.0.htm`\n - `https://example.org/index.html` -> `example.org/index_GET_200C.0.html`\n - `https://example.org/media`, `https://example.org/media/` -> `example.org/media/index_GET_200C.0.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `example.org/view/index?one=1&two=2&three=3_GET_200C.0.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `k\u00f6nigsg\u00e4\u00dfchen.example.org/index_GET_200C.0.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/index_GET_200C.0.htm`\n - `hupnq_mhs` : `%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 120)s_%(method)s_%(net_url|to_ascii|sha256|take_prefix 4)s_%(status)s%(filepath_ext)s`\n - `https://example.org` -> `example.org/index_GET_50d7_200C.htm`\n - `https://example.org/` -> `example.org/index_GET_8198_200C.htm`\n - `https://example.org/index.html` -> `example.org/index_GET_f0dc_200C.html`\n - `https://example.org/media` -> `example.org/media/index_GET_086d_200C.htm`\n - `https://example.org/media/` -> `example.org/media/index_GET_3fbb_200C.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `example.org/view/index?one=1&two=2&three=3_GET_5658_200C.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `k\u00f6nigsg\u00e4\u00dfchen.example.org/index_GET_4f11_200C.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/index_GET_c4ae_200C.htm`\n - `hupnq_mhsn` : `%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 100)s_%(method)s_%(net_url|to_ascii|sha256|take_prefix 4)s_%(status)s.%(num)d%(filepath_ext)s`\n - `https://example.org` -> `example.org/index_GET_50d7_200C.0.htm`\n - `https://example.org/` -> `example.org/index_GET_8198_200C.0.htm`\n - `https://example.org/index.html` -> `example.org/index_GET_f0dc_200C.0.html`\n - `https://example.org/media` -> `example.org/media/index_GET_086d_200C.0.htm`\n - `https://example.org/media/` -> `example.org/media/index_GET_3fbb_200C.0.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `example.org/view/index?one=1&two=2&three=3_GET_5658_200C.0.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `k\u00f6nigsg\u00e4\u00dfchen.example.org/index_GET_4f11_200C.0.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/index_GET_c4ae_200C.0.htm`\n - `rhupq` : `%(rhostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_query|abbrev 120)s%(filepath_ext)s`\n - `https://example.org`, `https://example.org/` -> `org.example/index.htm`\n - `https://example.org/index.html` -> `org.example/index.html`\n - `https://example.org/media`, `https://example.org/media/` -> `org.example/media/index.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `org.example/view/index?one=1&two=2&three&three=3.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `org.example.k\u00f6nigsg\u00e4\u00dfchen/index.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `org.example.\u3067\u3059\u306e.\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/index.htm`\n - `rhupq_msn` : `%(rhostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_query|abbrev 100)s_%(method)s_%(status)s.%(num)d%(filepath_ext)s`\n - `https://example.org`, `https://example.org/` -> `org.example/index_GET_200C.0.htm`\n - `https://example.org/index.html` -> `org.example/index_GET_200C.0.html`\n - `https://example.org/media`, `https://example.org/media/` -> `org.example/media/index_GET_200C.0.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `org.example/view/index?one=1&two=2&three&three=3_GET_200C.0.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `org.example.k\u00f6nigsg\u00e4\u00dfchen/index_GET_200C.0.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `org.example.\u3067\u3059\u306e.\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/index_GET_200C.0.htm`\n - `rhupnq` : `%(rhostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 120)s%(filepath_ext)s`\n - `https://example.org`, `https://example.org/` -> `org.example/index.htm`\n - `https://example.org/index.html` -> `org.example/index.html`\n - `https://example.org/media`, `https://example.org/media/` -> `org.example/media/index.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `org.example/view/index?one=1&two=2&three=3.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `org.example.k\u00f6nigsg\u00e4\u00dfchen/index.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `org.example.\u3067\u3059\u306e.\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/index.htm`\n - `rhupnq_msn` : `%(rhostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 100)s_%(method)s_%(status)s.%(num)d%(filepath_ext)s`\n - `https://example.org`, `https://example.org/` -> `org.example/index_GET_200C.0.htm`\n - `https://example.org/index.html` -> `org.example/index_GET_200C.0.html`\n - `https://example.org/media`, `https://example.org/media/` -> `org.example/media/index_GET_200C.0.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `org.example/view/index?one=1&two=2&three=3_GET_200C.0.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `org.example.k\u00f6nigsg\u00e4\u00dfchen/index_GET_200C.0.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `org.example.\u3067\u3059\u306e.\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/index_GET_200C.0.htm`\n - `rhupnq_mhs` : `%(rhostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 120)s_%(method)s_%(net_url|to_ascii|sha256|take_prefix 4)s_%(status)s%(filepath_ext)s`\n - `https://example.org` -> `org.example/index_GET_50d7_200C.htm`\n - `https://example.org/` -> `org.example/index_GET_8198_200C.htm`\n - `https://example.org/index.html` -> `org.example/index_GET_f0dc_200C.html`\n - `https://example.org/media` -> `org.example/media/index_GET_086d_200C.htm`\n - `https://example.org/media/` -> `org.example/media/index_GET_3fbb_200C.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `org.example/view/index?one=1&two=2&three=3_GET_5658_200C.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `org.example.k\u00f6nigsg\u00e4\u00dfchen/index_GET_4f11_200C.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `org.example.\u3067\u3059\u306e.\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/index_GET_c4ae_200C.htm`\n - `rhupnq_mhsn` : `%(rhostname)s/%(filepath_parts|abbrev_each 120|pp_to_path)s%(oqm)s%(mq_nquery|abbrev 100)s_%(method)s_%(net_url|to_ascii|sha256|take_prefix 4)s_%(status)s.%(num)d%(filepath_ext)s`\n - `https://example.org` -> `org.example/index_GET_50d7_200C.0.htm`\n - `https://example.org/` -> `org.example/index_GET_8198_200C.0.htm`\n - `https://example.org/index.html` -> `org.example/index_GET_f0dc_200C.0.html`\n - `https://example.org/media` -> `org.example/media/index_GET_086d_200C.0.htm`\n - `https://example.org/media/` -> `org.example/media/index_GET_3fbb_200C.0.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `org.example/view/index?one=1&two=2&three=3_GET_5658_200C.0.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `org.example.k\u00f6nigsg\u00e4\u00dfchen/index_GET_4f11_200C.0.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `org.example.\u3067\u3059\u306e.\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/index_GET_c4ae_200C.0.htm`\n - `flat` : `%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path|replace / __|abbrev 120)s%(oqm)s%(mq_nquery|abbrev 100)s%(filepath_ext)s`\n - `https://example.org`, `https://example.org/` -> `example.org/index.htm`\n - `https://example.org/index.html` -> `example.org/index.html`\n - `https://example.org/media`, `https://example.org/media/` -> `example.org/media__index.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `example.org/view__index?one=1&two=2&three=3.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435__is__index.htm`\n - `flat_ms` : `%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path|replace / __|abbrev 120)s%(oqm)s%(mq_nquery|abbrev 100)s_%(method)s_%(status)s%(filepath_ext)s`\n - `https://example.org`, `https://example.org/` -> `example.org/index_GET_200C.htm`\n - `https://example.org/index.html` -> `example.org/index_GET_200C.html`\n - `https://example.org/media`, `https://example.org/media/` -> `example.org/media__index_GET_200C.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `example.org/view__index?one=1&two=2&three=3_GET_200C.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `k\u00f6nigsg\u00e4\u00dfchen.example.org/index_GET_200C.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435__is__index_GET_200C.htm`\n - `flat_msn` : `%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path|replace / __|abbrev 120)s%(oqm)s%(mq_nquery|abbrev 100)s_%(method)s_%(status)s.%(num)d%(filepath_ext)s`\n - `https://example.org`, `https://example.org/` -> `example.org/index_GET_200C.0.htm`\n - `https://example.org/index.html` -> `example.org/index_GET_200C.0.html`\n - `https://example.org/media`, `https://example.org/media/` -> `example.org/media__index_GET_200C.0.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `example.org/view__index?one=1&two=2&three=3_GET_200C.0.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `k\u00f6nigsg\u00e4\u00dfchen.example.org/index_GET_200C.0.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435__is__index_GET_200C.0.htm`\n - `flat_mhs` : `%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path|replace / __|abbrev 120)s%(oqm)s%(mq_nquery|abbrev 100)s_%(method)s_%(net_url|to_ascii|sha256|take_prefix 4)s_%(status)s%(filepath_ext)s`\n - `https://example.org` -> `example.org/index_GET_50d7_200C.htm`\n - `https://example.org/` -> `example.org/index_GET_8198_200C.htm`\n - `https://example.org/index.html` -> `example.org/index_GET_f0dc_200C.html`\n - `https://example.org/media` -> `example.org/media__index_GET_086d_200C.htm`\n - `https://example.org/media/` -> `example.org/media__index_GET_3fbb_200C.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `example.org/view__index?one=1&two=2&three=3_GET_5658_200C.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `k\u00f6nigsg\u00e4\u00dfchen.example.org/index_GET_4f11_200C.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435__is__index_GET_c4ae_200C.htm`\n - `flat_mhsn` : `%(hostname)s/%(filepath_parts|abbrev_each 120|pp_to_path|replace / __|abbrev 120)s%(oqm)s%(mq_nquery|abbrev 100)s_%(method)s_%(net_url|to_ascii|sha256|take_prefix 4)s_%(status)s.%(num)d%(filepath_ext)s`\n - `https://example.org` -> `example.org/index_GET_50d7_200C.0.htm`\n - `https://example.org/` -> `example.org/index_GET_8198_200C.0.htm`\n - `https://example.org/index.html` -> `example.org/index_GET_f0dc_200C.0.html`\n - `https://example.org/media` -> `example.org/media__index_GET_086d_200C.0.htm`\n - `https://example.org/media/` -> `example.org/media__index_GET_3fbb_200C.0.htm`\n - `https://example.org/view?one=1&two=2&three=&three=3#fragment` -> `example.org/view__index?one=1&two=2&three=3_GET_5658_200C.0.htm`\n - `https://k\u00f6nigsg\u00e4\u00dfchen.example.org/index.html` -> `k\u00f6nigsg\u00e4\u00dfchen.example.org/index_GET_4f11_200C.0.html`\n - `https://\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435/is/`, `https://xn--hck7aa9d8fj9i.xn--88j1aw.example.org/%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/is/` -> `\u30b8\u30e3\u30b8\u30a7\u30e1\u30f3\u30c8.\u3067\u3059\u306e.example.org/\u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u0435__is__index_GET_c4ae_200C.0.htm`\n - available substitutions:\n - `num`: number of times the resulting output path was encountered before; adding this parameter to your `--output` format will ensure all generated file names will be unique\n - all expressions of `wrrarms get --expr`, which see\n - `--stdin0`\n : read zero-terminated `PATH`s from stdin, these will be processed after `PATH`s specified as command-line arguments\n\n- error handling:\n - `--errors {fail,skip,ignore}`\n : when an error occurs:\n - `fail`: report failure and stop the execution (default)\n - `skip`: report failure but skip the reqres that produced it from the output and continue\n - `ignore`: `skip`, but don't report the failure\n\n- filters:\n - `--or EXPR`\n : only work on reqres which match any of these expressions...\n - `--and EXPR`\n : ... and all of these expressions, both can be specified multiple times, both use the same expression format as `wrrarms get --expr`, which see\n\n- output:\n - `--no-output`\n : don't print anything (default)\n - `-l, --lf-terminated`\n : terminate output absolute paths of newly produced files with `\\n` (LF) newline characters\n - `-z, --zero-terminated`\n : terminate output absolute paths of newly produced files with `\\0` (NUL) bytes\n\n- action:\n - `--move`\n : move source files under `DESTINATION` (default)\n - `--copy`\n : copy source files to files under `DESTINATION`\n - `--hardlink`\n : create hardlinks from source files to paths under `DESTINATION`\n - `--symlink`\n : create symlinks from source files to paths under `DESTINATION`\n\n- updates:\n - `--keep`\n : disallow replacements and overwrites for any existing files under `DESTINATION` (default);\n broken symlinks are allowed to be replaced;\n if source and target directories are the same then some files can still be renamed into previously non-existing names;\n all other updates are disallowed\n - `--latest`\n : replace files under `DESTINATION` if `stime_ms` for the source reqres is newer than the same value for reqres stored at the destination\n\n- caching, deferring, and batching:\n - `--seen-number INT`\n : track at most this many distinct generated `--output` values; default: `16384`;\n making this larger improves disk performance at the cost of increased memory consumption;\n setting it to zero will force force `wrrarms` to constantly re-check existence of `--output` files and force `wrrarms` to execute all IO actions immediately, disregarding `--defer-number` setting\n - `--cache-number INT`\n : cache `stat(2)` information about this many files in memory; default: `8192`;\n making this larger improves performance at the cost of increased memory consumption;\n setting this to a too small number will likely force `wrrarms` into repeatedly performing lots of `stat(2)` system calls on the same files;\n setting this to a value smaller than `--defer-number` will not improve memory consumption very much since deferred IO actions also cache information about their own files\n - `--defer-number INT`\n : defer at most this many IO actions; default: `1024`;\n making this larger improves performance at the cost of increased memory consumption;\n setting it to zero will force all IO actions to be applied immediately\n - `--batch-number INT`\n : queue at most this many deferred IO actions to be applied together in a batch; this queue will only be used if all other resource constraints are met; default: 128\n - `--max-memory INT`\n : the caches, the deferred actions queue, and the batch queue, all taken together, must not take more than this much memory in MiB; default: `1024`;\n making this larger improves performance;\n the actual maximum whole-program memory consumption is `O(<size of the largest reqres> + <--seen-number> + <sum of lengths of the last --seen-number generated --output paths> + <--cache-number> + <--defer-number> + <--batch-number> + <--max-memory>)`\n - `--lazy`\n : sets all of the above options to positive infinity;\n most useful when doing `wrrarms organize --symlink --latest --output flat` or similar, where the number of distinct generated `--output` values and the amount of other data `wrrarms` needs to keep in memory is small, in which case it will force `wrrarms` to compute the desired file system state first and then perform all disk writes in a single batch\n\n- file system path ordering:\n - `--paths-given-order`\n : `argv` and `--stdin0` `PATH`s are processed in the order they are given (default when `--keep`)\n - `--paths-sorted`\n : `argv` and `--stdin0` `PATH`s are processed in lexicographic order\n - `--paths-reversed`\n : `argv` and `--stdin0` `PATH`s are processed in reverse lexicographic order (default when `--latest`)\n - `--walk-fs-order`\n : recursive file system walk is done in the order `readdir(2)` gives results\n - `--walk-sorted`\n : recursive file system walk is done in lexicographic order (default when `--keep`)\n - `--walk-reversed`\n : recursive file system walk is done in reverse lexicographic order (default when `--latest`)\n\n### wrrarms import\n\nUse specified parser to parse data in each `INPUT` `PATH` into reqres and dump them under `DESTINATION` with paths derived from their metadata.\nIn short, this is `wrrarms organize --copy` but for non-WRR `INPUT` files.\n\n- file formats:\n - `{mitmproxy}`\n - `mitmproxy`\n : convert `mitmproxy` stream dumps into WRR files\n\n### wrrarms import mitmproxy\n\nParse each `INPUT` `PATH` as `mitmproxy` stream dump (by using `mitmproxy`'s own parser) into a sequence of reqres and dump them under `DESTINATION` with paths derived from their metadata.\n\n- positional arguments:\n - `PATH`\n : inputs, can be a mix of files and directories (which will be traversed recursively)\n\n- options:\n - `--dry-run`\n : perform a trial run without actually performing any changes\n - `-q, --quiet`\n : don't log computed updates to stderr\n - `-t DESTINATION, --to DESTINATION`\n : destination directory\n - `-o FORMAT, --output FORMAT`\n : format describing generated output paths, an alias name or \"format:\" followed by a custom pythonic %-substitution string; same as `wrrarms organize --output`, which see\n - `--stdin0`\n : read zero-terminated `PATH`s from stdin, these will be processed after `PATH`s specified as command-line arguments\n\n- error handling:\n - `--errors {fail,skip,ignore}`\n : when an error occurs:\n - `fail`: report failure and stop the execution (default)\n - `skip`: report failure but skip the reqres that produced it from the output and continue\n - `ignore`: `skip`, but don't report the failure\n\n- filters:\n - `--or EXPR`\n : only import reqres which match any of these expressions...\n - `--and EXPR`\n : ... and all of these expressions, both can be specified multiple times, both use the same expression format as `wrrarms get --expr`, which see\n\n- output:\n - `--no-output`\n : don't print anything (default)\n - `-l, --lf-terminated`\n : terminate output absolute paths of newly produced files with `\\n` (LF) newline characters\n - `-z, --zero-terminated`\n : terminate output absolute paths of newly produced files with `\\0` (NUL) bytes\n\n- caching, deferring, and batching:\n - `--seen-number INT`\n : track at most this many distinct generated `--output` values; default: `16384`;\n making this larger improves disk performance at the cost of increased memory consumption;\n setting it to zero will force force `wrrarms` to constantly re-check existence of `--output` files and force `wrrarms` to execute all IO actions immediately, disregarding `--defer-number` setting\n - `--cache-number INT`\n : cache `stat(2)` information about this many files in memory; default: `8192`;\n making this larger improves performance at the cost of increased memory consumption;\n setting this to a too small number will likely force `wrrarms` into repeatedly performing lots of `stat(2)` system calls on the same files;\n setting this to a value smaller than `--defer-number` will not improve memory consumption very much since deferred IO actions also cache information about their own files\n - `--defer-number INT`\n : defer at most this many IO actions; default: `0`;\n making this larger improves performance at the cost of increased memory consumption;\n setting it to zero will force all IO actions to be applied immediately\n - `--batch-number INT`\n : queue at most this many deferred IO actions to be applied together in a batch; this queue will only be used if all other resource constraints are met; default: 128\n - `--max-memory INT`\n : the caches, the deferred actions queue, and the batch queue, all taken together, must not take more than this much memory in MiB; default: `1024`;\n making this larger improves performance;\n the actual maximum whole-program memory consumption is `O(<size of the largest reqres> + <--seen-number> + <sum of lengths of the last --seen-number generated --output paths> + <--cache-number> + <--defer-number> + <--batch-number> + <--max-memory>)`\n - `--lazy`\n : sets all of the above options to positive infinity;\n most useful when doing `wrrarms organize --symlink --latest --output flat` or similar, where the number of distinct generated `--output` values and the amount of other data `wrrarms` needs to keep in memory is small, in which case it will force `wrrarms` to compute the desired file system state first and then perform all disk writes in a single batch\n\n- file system path ordering:\n - `--paths-given-order`\n : `argv` and `--stdin0` `PATH`s are processed in the order they are given (default)\n - `--paths-sorted`\n : `argv` and `--stdin0` `PATH`s are processed in lexicographic order\n - `--paths-reversed`\n : `argv` and `--stdin0` `PATH`s are processed in reverse lexicographic order\n - `--walk-fs-order`\n : recursive file system walk is done in the order `readdir(2)` gives results\n - `--walk-sorted`\n : recursive file system walk is done in lexicographic order (default)\n - `--walk-reversed`\n : recursive file system walk is done in reverse lexicographic order\n\n### wrrarms export\n\nParse given WRR files into their respective reqres, convert to another file format, and then dump the result under `DESTINATION` with the new path derived from each reqres' metadata.\n\n- file formats:\n - `{mirror}`\n - `mirror`\n : convert given WRR files into a local website mirror stored in interlinked plain files\n\n### wrrarms export mirror\n\nParse given WRR files, filter out those that have no responses, transform and then dump their response bodies into separate files under `DESTINATION` with the new path derived from each reqres' metadata.\nIn short, this is a combination of `wrrarms organize --copy` followed by in-place `wrrarms get`.\nIn other words, this generates static offline website mirrors, producing results similar to those of `wget -mpk`.\n\n- positional arguments:\n - `PATH`\n : inputs, can be a mix of files and directories (which will be traversed recursively)\n\n- options:\n - `--dry-run`\n : perform a trial run without actually performing any changes\n - `-q, --quiet`\n : don't log computed updates to stderr\n - `-t DESTINATION, --to DESTINATION`\n : target directory\n - `-o FORMAT, --output FORMAT`\n : format describing generated output paths, an alias name or a custom pythonic %-substitution string; same as `wrrarms organize --output`, which see\n - `--stdin0`\n : read zero-terminated `PATH`s from stdin, these will be processed after `PATH`s specified as command-line arguments\n\n- error handling:\n - `--errors {fail,skip,ignore}`\n : when an error occurs:\n - `fail`: report failure and stop the execution (default)\n - `skip`: report failure but skip the reqres that produced it from the output and continue\n - `ignore`: `skip`, but don't report the failure\n\n- filters:\n - `--or EXPR`\n : only export reqres which match any of these expressions...\n - `--and EXPR`\n : ... and all of these expressions, both can be specified multiple times, both use the same expression format as `wrrarms get --expr`, which see\n\n- output:\n - `--no-output`\n : don't print anything (default)\n - `-l, --lf-terminated`\n : terminate output absolute paths of newly produced files with `\\n` (LF) newline characters\n - `-z, --zero-terminated`\n : terminate output absolute paths of newly produced files with `\\0` (NUL) bytes\n\n- expression evaluation:\n - `-e EXPR, --expr EXPR`\n : an expression to export, see `wrrarms get --expr` for more info on expression format (default: `response.body|eb|scrub response +all_refs,-actions`)\n\n- URL remapping, used by `scrub` `--expr` atom:\n - `--remap-id`\n : remap all URLs with an identity function; i.e. don't remap anything\n - `--remap-void`\n : remap all jump-link and action URLs to `javascript:void(0)` and all resource URLs into empty `data:` URLs; the result will be self-contained\n - `--remap-open, -k, --convert-links`\n : point all available URLs present in input `PATH`s to their corresponding output paths, remap all unavailable URLs like `--remap-id` does; this is similar to `wget (-k|--convert-links)`\n - `--remap-closed`\n : remap all available URLs like `--remap-open` does, remap all unavailable URLs like `--remap-void` does; the result will be self-contained\n - `--remap-all`\n : remap all available URLs like `--remap-open` does, point each unavailable URL to a path produced by the current `--output` format for a trivial `GET <URL> -> 200 OK` reqres; this will produce broken links if the `--output` format depends on anything but the URL itself, but for a simple `--output` (like the default `hupq`) this allows `wrrarms export` to be used incrementally; the result will be self-contained (default)\n\n- export targets (default: `net_url`s of all input `PATH`s):\n - `-r URL, --root URL`\n : recursion root; a URL which will be used as a root for recursive export; can be specified multiple times; if none are specified, then all URLs available from `PATH`s are treated as roots\n - `-d DEPTH, --depth DEPTH`\n : maximum recursion depth level; the default is `0`, which means \"documents and their resources only\"; setting this to `1` will also export one level of documents referenced via jump and action links, if those are being remapped to local files with `--remap-*`; higher values will mean even more recursion\n\n- file system path ordering:\n - `--paths-given-order`\n : `argv` and `--stdin0` `PATH`s are processed in the order they are given (default)\n - `--paths-sorted`\n : `argv` and `--stdin0` `PATH`s are processed in lexicographic order\n - `--paths-reversed`\n : `argv` and `--stdin0` `PATH`s are processed in reverse lexicographic order\n - `--walk-fs-order`\n : recursive file system walk is done in the order `readdir(2)` gives results\n - `--walk-sorted`\n : recursive file system walk is done in lexicographic order (default)\n - `--walk-reversed`\n : recursive file system walk is done in reverse lexicographic order\n\n## Examples\n\n- Pretty-print all reqres in `../dumb_server/pwebarc-dump` using an abridged (for ease of reading and rendering) verbose textual representation:\n ```\n wrrarms pprint ../dumb_server/pwebarc-dump\n ```\n\n- Pipe response body scrubbed of dynamic content (see `wrrarms get` documentation above) from a given WRR file to stdout:\n ```\n wrrarms get ../dumb_server/pwebarc-dump/path/to/file.wrr\n ```\n\n- Pipe raw response body from a given WRR file to stdout:\n ```\n wrrarms get -e \"response.body|eb\" ../dumb_server/pwebarc-dump/path/to/file.wrr\n ```\n\n- Get first 4 characters of a hex digest of sha256 hash computed on the URL without the fragment/hash part:\n ```\n wrrarms get -e \"net_url|to_ascii|sha256|take_prefix 4\" ../dumb_server/pwebarc-dump/path/to/file.wrr\n ```\n\n- Pipe response body from a given WRR file to stdout, but less efficiently, by generating a temporary file and giving it to `cat`:\n ```\n wrrarms run cat ../dumb_server/pwebarc-dump/path/to/file.wrr\n ```\n\n Thus `wrrarms run` can be used to do almost anything you want, e.g.\n\n ```\n wrrarms run less ../dumb_server/pwebarc-dump/path/to/file.wrr\n ```\n\n ```\n wrrarms run -- sort -R ../dumb_server/pwebarc-dump/path/to/file.wrr\n ```\n\n ```\n wrrarms run -n 2 -- diff -u ../dumb_server/pwebarc-dump/path/to/file-v1.wrr ../dumb_server/pwebarc-dump/path/to/file-v2.wrr\n ```\n\n- List paths of all WRR files from `../dumb_server/pwebarc-dump` that contain only complete `200 OK` responses with bodies larger than 1K:\n ```\n wrrarms find --and \"status|== 200C\" --and \"response.body|len|> 1024\" ../dumb_server/pwebarc-dump\n ```\n\n- Rename all WRR files in `../dumb_server/pwebarc-dump/default` according to their metadata using `--output default` (see the `wrrarms organize` section for its definition, the `default` format is designed to be human-readable while causing almost no collisions, thus making `num` substitution parameter to almost always stay equal to `0`, making things nice and deterministic):\n ```\n wrrarms organize ../dumb_server/pwebarc-dump/default\n ```\n\n alternatively, just show what would be done\n\n ```\n wrrarms organize --dry-run ../dumb_server/pwebarc-dump/default\n ```\n\n- The output of `wrrarms organize --zero-terminated` can be piped into `wrrarms organize --stdin0` to perform complex updates. E.g. the following will rename new reqres from `../dumb_server/pwebarc-dump` to `~/pwebarc/raw` renaming them with `--output default`, the `for` loop is there to preserve profiles:\n ```\n for arg in ../dumb_server/pwebarc-dump/* ; do\n wrrarms organize --zero-terminated --to ~/pwebarc/raw/\"$(basename \"$arg\")\" \"$arg\"\n done > changes\n ```\n\n then, we can reuse `changes` to symlink all new files from `~/pwebarc/raw` to `~/pwebarc/all` using `--output hupq_msn`, which would show most of the URL in the file name:\n\n ```\n wrrarms organize --stdin0 --symlink --to ~/pwebarc/all --output hupq_msn < changes\n ```\n\n and then, we can reuse `changes` again and use them to update `~/pwebarc/latest`, filling it with symlinks pointing to the latest `200 OK` complete reqres from `~/pwebarc/raw`, similar to what `wget -r` would produce (except `wget` would do network requests and produce responce bodies, while this will build a file system tree of symlinks to WRR files in `/pwebarc/raw`):\n\n ```\n wrrarms organize --stdin0 --symlink --latest --to ~/pwebarc/latest --output hupq --and \"status|== 200C\" < changes\n ```\n\n- `wrrarms organize --move` is de-duplicating when possible, while `--copy`, `--hardlink`, and `--symlink` are non-duplicating when possible, i.e.:\n ```\n wrrarms organize --copy --to ~/pwebarc/copy1 ~/pwebarc/original\n wrrarms organize --copy --to ~/pwebarc/copy2 ~/pwebarc/original\n wrrarms organize --hardlink --to ~/pwebarc/copy3 ~/pwebarc/original\n\n # noops\n wrrarms organize --copy --to ~/pwebarc/copy1 ~/pwebarc/original\n wrrarms organize --hardlink --to ~/pwebarc/copy1 ~/pwebarc/original\n wrrarms organize --copy --to ~/pwebarc/copy2 ~/pwebarc/original\n wrrarms organize --hardlink --to ~/pwebarc/copy2 ~/pwebarc/original\n wrrarms organize --copy --to ~/pwebarc/copy3 ~/pwebarc/original\n wrrarms organize --hardlink --to ~/pwebarc/copy3 ~/pwebarc/original\n\n # de-duplicate\n wrrarms organize --move --to ~/pwebarc/all ~/pwebarc/original ~/pwebarc/copy1 ~/pwebarc/copy2 ~/pwebarc/copy3\n ```\n\n will produce `~/pwebarc/all` which has each duplicated file stored only once. Similarly,\n\n ```\n wrrarms organize --symlink --output hupq_msn --to ~/pwebarc/pointers ~/pwebarc/original\n wrrarms organize --symlink --output shupq_msn --to ~/pwebarc/schemed ~/pwebarc/original\n\n # noop\n wrrarms organize --symlink --output hupq_msn --to ~/pwebarc/pointers ~/pwebarc/original ~/pwebarc/schemed\n ```\n\n will produce `~/pwebarc/pointers` which has each symlink only once.\n\n## Advanced examples\n\n- Pretty-print all reqres in `../dumb_server/pwebarc-dump` by dumping their whole structure into an abridged Pythonic Object Representation (repr):\n ```\n wrrarms stream --expr . ../dumb_server/pwebarc-dump\n ```\n\n ```\n wrrarms stream -e . ../dumb_server/pwebarc-dump\n ```\n\n- Pretty-print all reqres in `../dumb_server/pwebarc-dump` using the unabridged verbose textual representation:\n ```\n wrrarms pprint --unabridged ../dumb_server/pwebarc-dump\n ```\n\n ```\n wrrarms pprint -u ../dumb_server/pwebarc-dump\n ```\n\n- Pretty-print all reqres in `../dumb_server/pwebarc-dump` by dumping their whole structure into the unabridged Pythonic Object Representation (repr) format:\n ```\n wrrarms stream --unabridged --expr . ../dumb_server/pwebarc-dump\n ```\n\n ```\n wrrarms stream -ue . ../dumb_server/pwebarc-dump\n ```\n\n- Produce a JSON list of `[<file path>, <time it finished loading in milliseconds since UNIX epoch>, <URL>]` tuples (one per reqres) and pipe it into `jq` for indented and colored output:\n ```\n wrrarms stream --format=json -ue fs_path -e finished_at -e request.url ../dumb_server/pwebarc-dump | jq .\n ```\n\n- Similarly, but produce a CBOR output:\n ```\n wrrarms stream --format=cbor -ue fs_path -e finished_at -e request.url ../dumb_server/pwebarc-dump | less\n ```\n\n- Concatenate all response bodies of all the requests in `../dumb_server/pwebarc-dump`:\n ```\n wrrarms stream --format=raw --not-terminated -ue \"response.body|es\" ../dumb_server/pwebarc-dump | less\n ```\n\n- Print all unique visited URLs, one per line:\n ```\n wrrarms stream --format=raw --lf-terminated -ue request.url ../dumb_server/pwebarc-dump | sort | uniq\n ```\n\n- Same idea, but using NUL bytes while processing, and prints two URLs per line:\n ```\n wrrarms stream --format=raw --zero-terminated -ue request.url ../dumb_server/pwebarc-dump | sort -z | uniq -z | xargs -0 -n2 echo\n ```\n\n### How to handle binary data\n\nTrying to use response bodies produced by `wrrarms stream --format=json` is likely to result garbled data as JSON can't represent raw sequences of bytes, thus binary data will have to be encoded into UNICODE using replacement characters:\n\n```\nwrrarms stream --format=json -ue . ../dumb_server/pwebarc-dump/path/to/file.wrr | jq .\n```\n\nThe most generic solution to this is to use `--format=cbor` instead, which would produce a verbose CBOR representation equivalent to the one used by `--format=json` but with binary data preserved as-is:\n\n```\nwrrarms stream --format=cbor -ue . ../dumb_server/pwebarc-dump/path/to/file.wrr | less\n```\n\nOr you could just dump raw response bodies separately:\n\n```\nwrrarms stream --format=raw -ue response.body ../dumb_server/pwebarc-dump/path/to/file.wrr | less\n```\n\n```\nwrrarms get ../dumb_server/pwebarc-dump/path/to/file.wrr | less\n```\n\n",
"bugtrack_url": null,
"license": "GPL-3.0-or-later",
"summary": "A tool for displaying and manipulating Web Request+Response (WRR) files of Private Passive Web Archive (pwebarc) project",
"version": "0.11.0",
"project_urls": {
"GitHub": "https://github.com/Own-Data-Privateer/pwebarc",
"Homepage": "https://oxij.org/software/pwebarc/",
"Support Development": "https://oxij.org/#support"
},
"split_keywords": [
"http",
" https",
" archive",
" wayback machine",
" download"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8eed2a662dacbe206a3a22687ac8ae600644fd0d3ffb069a130d089549b7f438",
"md5": "a4359bf0a599cab6faa564efccf37111",
"sha256": "dde24e5343ed4a72dbe7258ad37ebc18835aa5438961239b0039761da56e2703"
},
"downloads": -1,
"filename": "pwebarc_wrrarms-0.11.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a4359bf0a599cab6faa564efccf37111",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 73674,
"upload_time": "2024-04-03T14:26:48",
"upload_time_iso_8601": "2024-04-03T14:26:48.485327Z",
"url": "https://files.pythonhosted.org/packages/8e/ed/2a662dacbe206a3a22687ac8ae600644fd0d3ffb069a130d089549b7f438/pwebarc_wrrarms-0.11.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "41111f43c3e03423d6acdbbafcf81b2ccdf6e7e613443332d6ea1b4e451c28ed",
"md5": "ed2778e6b451f74e1f06a2200ab04131",
"sha256": "7be0df03ad20d071ebed39be580e75daa2bd7b56ea691cdf90e7af087b4bc275"
},
"downloads": -1,
"filename": "pwebarc-wrrarms-0.11.0.tar.gz",
"has_sig": false,
"md5_digest": "ed2778e6b451f74e1f06a2200ab04131",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 98464,
"upload_time": "2024-04-03T14:26:51",
"upload_time_iso_8601": "2024-04-03T14:26:51.256991Z",
"url": "https://files.pythonhosted.org/packages/41/11/1f43c3e03423d6acdbbafcf81b2ccdf6e7e613443332d6ea1b4e451c28ed/pwebarc-wrrarms-0.11.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-03 14:26:51",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Own-Data-Privateer",
"github_project": "pwebarc",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "pwebarc-wrrarms"
}
JSON
