| Name | pawk JSON |
| Version |
0.8.0
JSON |
| download |
| home_page | https://github.com/alecthomas/pawk |
| Summary | PAWK - A Python line processor (like AWK) |
| upload_time | 2023-01-29 19:23:37 |
| maintainer | |
| docs_url | None |
| author | Alec Thomas |
| requires_python | >=3.7 |
| license | MIT |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# PAWK - A Python line processor (like AWK)
PAWK aims to bring the full power of Python to AWK-like line-processing.
Here are some quick examples to show some of the advantages of pawk over AWK.
The first example transforms `/etc/hosts` into a JSON map of host to IP:
cat /etc/hosts | pawk -B 'd={}' -E 'json.dumps(d)' '!/^#/ d[f[1]] = f[0]'
Breaking this down:
1. `-B 'd={}'` is a begin statement initializing a dictionary, executed once before processing begins.
2. `-E 'json.dumps(d)'` is an end statement expression, producing the JSON representation of the dictionary `d`.
3. `!/^#/` tells pawk to match any line *not* beginning with `#`.
4. `d[f[1]] = f[0]` adds a dictionary entry where the key is the second field in the line (the first hostname) and the value is the first field (the IP address).
And another example showing how to bzip2-compress + base64-encode a file:
cat pawk.py | pawk -E 'base64.encodestring(bz2.compress(t))'
### AWK example translations
Most basic AWK constructs are available. You can find more idiomatic examples below in the example section, but here are a bunch of awk commands and their equivalent pawk commands to get started with:
Print lines matching a pattern:
ls -l / | awk '/etc/'
ls -l / | pawk '/etc/'
Print lines *not* matching a pattern:
ls -l / | awk '!/etc/'
ls -l / | pawk '!/etc/'
Field slicing and dicing (here pawk wins because of Python's array slicing):
ls -l / | awk '/etc/ {print $5, $6, $7, $8, $9}'
ls -l / | pawk '/etc/ f[4:]'
Begin and end end actions (in this case, summing the sizes of all files):
ls -l | awk 'BEGIN {c = 0} {c += $5} END {print c}'
ls -l | pawk -B 'c = 0' -E 'c' 'c += int(f[4])'
Print files where a field matches a numeric expression (in this case where files are > 1024 bytes):
ls -l | awk '$5 > 1024'
ls -l | pawk 'int(f[4]) > 1024'
Matching a single field (any filename with "t" in it):
ls -l | awk '$NF ~/t/'
ls -l | pawk '"t" in f[-1]'
## Installation
It should be as simple as:
```
pip install pawk
```
But if that doesn't work, just download the `pawk.py`, make it executable, and place it somewhere in your path.
## Expression evaluation
PAWK evaluates a Python expression or statement against each line in stdin. The following variables are available in local context:
- `line` - Current line text, including newline.
- `l` - Current line text, excluding newline.
- `n` - The current 1-based line number.
- `f` - Fields of the line (split by the field separator `-F`).
- `nf` - Number of fields in this line.
- `m` - Tuple of match regular expression capture groups, if any.
In the context of the `-E` block:
- `t` - The entire input text up to the current cursor position.
If the flag `-H, --header` is provided, each field in the first row of the input will be treated as field variable names in subsequent rows. The header is not output. For example, given the input:
```
count name
12 bob
34 fred
```
We could do:
```
$ pawk -H '"%s is %s" % (name, count)' < input.txt
bob is 12
fred is 34
```
To output a header as well, use `-B`:
```
$ pawk -H -B '"name is count"' '"%s is %s" % (name, count)' < input.txt
name is count
bob is 12
fred is 34
```
Module references will be automatically imported if possible. Additionally, the `--import <module>[,<module>,...]` flag can be used to import symbols from a set of modules into the evaluation context.
eg. `--import os.path` will import all symbols from `os.path`, such as `os.path.isfile()`, into the context.
## Output
### Line actions
The type of the evaluated expression determines how output is displayed:
- `tuple` or `list`: the elements are converted to strings and joined with the output delimiter (`-O`).
- `None` or `False`: nothing is output for that line.
- `True`: the original line is output.
- Any other value is converted to a string.
### Start/end blocks
The rules are the same as for line actions with one difference. Because there is no "line" that corresponds to them, an expression returning True is ignored.
$ echo -ne 'foo\nbar' | pawk -E t
foo
bar
## Command-line usage
```
Usage: cat input | pawk [<options>] <expr>
A Python line-processor (like awk).
See https://github.com/alecthomas/pawk for details. Based on
http://code.activestate.com/recipes/437932/.
Options:
-h, --help show this help message and exit
-I <filename>, --in_place=<filename>
modify given input file in-place
-i <modules>, --import=<modules>
comma-separated list of modules to "from x import *"
from
-F <delim> input delimiter
-O <delim> output delimiter
-L <delim> output line separator
-B <statement>, --begin=<statement>
begin statement
-E <statement>, --end=<statement>
end statement
-s, --statement DEPRECATED. retained for backward compatibility
-H, --header use first row as field variable names in subsequent
rows
--strict abort on exceptions
```
## Examples
### Line processing
Print the name and size of every file from stdin:
find . -type f | pawk 'f[0], os.stat(f[0]).st_size'
> **Note:** this example also shows how pawk automatically imports referenced modules, in this case `os`.
Print the sum size of all files from stdin:
find . -type f | \
pawk \
--begin 'c=0' \
--end c \
'c += os.stat(f[0]).st_size'
Short-flag version:
find . -type f | pawk -B c=0 -E c 'c += os.stat(f[0]).st_size'
### Whole-file processing
If you do not provide a line expression, but do provide an end statement, pawk will accumulate each line, and the entire file's text will be available in the end statement as `t`. This is useful for operations on entire files, like the following example of converting a file from markdown to HTML:
cat README.md | \
pawk --end 'markdown.markdown(t)'
Short-flag version:
cat README.md | pawk -E 'markdown.markdown(t)'
Raw data
{
"_id": null,
"home_page": "https://github.com/alecthomas/pawk",
"name": "pawk",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "",
"author": "Alec Thomas",
"author_email": "alec@swapoff.org",
"download_url": "https://files.pythonhosted.org/packages/be/19/b8c7847050c68f58c32e24f0424509b5999aa715cb00517e003c2382ce63/pawk-0.8.0.tar.gz",
"platform": null,
"description": "# PAWK - A Python line processor (like AWK)\n\nPAWK aims to bring the full power of Python to AWK-like line-processing.\n\nHere are some quick examples to show some of the advantages of pawk over AWK.\n\nThe first example transforms `/etc/hosts` into a JSON map of host to IP:\n\n\tcat /etc/hosts | pawk -B 'd={}' -E 'json.dumps(d)' '!/^#/ d[f[1]] = f[0]'\n\nBreaking this down:\n\n1. `-B 'd={}'` is a begin statement initializing a dictionary, executed once before processing begins.\n2. `-E 'json.dumps(d)'` is an end statement expression, producing the JSON representation of the dictionary `d`.\n3. `!/^#/` tells pawk to match any line *not* beginning with `#`.\n4. `d[f[1]] = f[0]` adds a dictionary entry where the key is the second field in the line (the first hostname) and the value is the first field (the IP address).\n\nAnd another example showing how to bzip2-compress + base64-encode a file:\n\n\tcat pawk.py | pawk -E 'base64.encodestring(bz2.compress(t))'\n\n### AWK example translations\n\nMost basic AWK constructs are available. You can find more idiomatic examples below in the example section, but here are a bunch of awk commands and their equivalent pawk commands to get started with:\n\nPrint lines matching a pattern:\n\n\tls -l / | awk '/etc/'\n\tls -l / | pawk '/etc/'\n\nPrint lines *not* matching a pattern:\n\n\tls -l / | awk '!/etc/'\n\tls -l / | pawk '!/etc/'\n\nField slicing and dicing (here pawk wins because of Python's array slicing):\n\n\tls -l / | awk '/etc/ {print $5, $6, $7, $8, $9}'\n\tls -l / | pawk '/etc/ f[4:]'\n\nBegin and end end actions (in this case, summing the sizes of all files):\n\n\tls -l | awk 'BEGIN {c = 0} {c += $5} END {print c}'\n\tls -l | pawk -B 'c = 0' -E 'c' 'c += int(f[4])'\n\nPrint files where a field matches a numeric expression (in this case where files are > 1024 bytes):\n\n\tls -l | awk '$5 > 1024'\n\tls -l | pawk 'int(f[4]) > 1024'\n\nMatching a single field (any filename with \"t\" in it):\n\n\tls -l | awk '$NF ~/t/'\n\tls -l | pawk '\"t\" in f[-1]'\n\n## Installation\n\nIt should be as simple as:\n\n```\npip install pawk\n```\n\nBut if that doesn't work, just download the `pawk.py`, make it executable, and place it somewhere in your path.\n\n## Expression evaluation\n\nPAWK evaluates a Python expression or statement against each line in stdin. The following variables are available in local context:\n\n- `line` - Current line text, including newline.\n- `l` - Current line text, excluding newline.\n- `n` - The current 1-based line number.\n- `f` - Fields of the line (split by the field separator `-F`).\n- `nf` - Number of fields in this line.\n- `m` - Tuple of match regular expression capture groups, if any.\n\n\nIn the context of the `-E` block:\n\n- `t` - The entire input text up to the current cursor position.\n\nIf the flag `-H, --header` is provided, each field in the first row of the input will be treated as field variable names in subsequent rows. The header is not output. For example, given the input:\n\n```\ncount name\n12 bob\n34 fred\n```\n\nWe could do:\n\n```\n$ pawk -H '\"%s is %s\" % (name, count)' < input.txt\nbob is 12\nfred is 34\n```\n\nTo output a header as well, use `-B`:\n\n```\n$ pawk -H -B '\"name is count\"' '\"%s is %s\" % (name, count)' < input.txt\nname is count\nbob is 12\nfred is 34\n```\n\nModule references will be automatically imported if possible. Additionally, the `--import <module>[,<module>,...]` flag can be used to import symbols from a set of modules into the evaluation context.\n\neg. `--import os.path` will import all symbols from `os.path`, such as `os.path.isfile()`, into the context.\n\n## Output\n\n### Line actions\n\nThe type of the evaluated expression determines how output is displayed:\n\n- `tuple` or `list`: the elements are converted to strings and joined with the output delimiter (`-O`).\n- `None` or `False`: nothing is output for that line.\n- `True`: the original line is output.\n- Any other value is converted to a string.\n\n### Start/end blocks\n\nThe rules are the same as for line actions with one difference. Because there is no \"line\" that corresponds to them, an expression returning True is ignored.\n\n\t$ echo -ne 'foo\\nbar' | pawk -E t\n foo\n bar\n\n\n## Command-line usage\n\n```\nUsage: cat input | pawk [<options>] <expr>\n\nA Python line-processor (like awk).\n\nSee https://github.com/alecthomas/pawk for details. Based on\nhttp://code.activestate.com/recipes/437932/.\n\nOptions:\n -h, --help show this help message and exit\n -I <filename>, --in_place=<filename>\n modify given input file in-place\n -i <modules>, --import=<modules>\n comma-separated list of modules to \"from x import *\"\n from\n -F <delim> input delimiter\n -O <delim> output delimiter\n -L <delim> output line separator\n -B <statement>, --begin=<statement>\n begin statement\n -E <statement>, --end=<statement>\n end statement\n -s, --statement DEPRECATED. retained for backward compatibility\n -H, --header use first row as field variable names in subsequent\n rows\n --strict abort on exceptions\n```\n\n## Examples\n\n### Line processing\n\nPrint the name and size of every file from stdin:\n\n\tfind . -type f | pawk 'f[0], os.stat(f[0]).st_size'\n\n> **Note:** this example also shows how pawk automatically imports referenced modules, in this case `os`.\n\nPrint the sum size of all files from stdin:\n\n\tfind . -type f | \\\n\t\tpawk \\\n\t\t\t--begin 'c=0' \\\n\t\t\t--end c \\\n\t\t\t'c += os.stat(f[0]).st_size'\n\nShort-flag version:\n\n\tfind . -type f | pawk -B c=0 -E c 'c += os.stat(f[0]).st_size'\n\n\n### Whole-file processing\n\nIf you do not provide a line expression, but do provide an end statement, pawk will accumulate each line, and the entire file's text will be available in the end statement as `t`. This is useful for operations on entire files, like the following example of converting a file from markdown to HTML:\n\n\tcat README.md | \\\n\t\tpawk --end 'markdown.markdown(t)'\n\nShort-flag version:\n\n\tcat README.md | pawk -E 'markdown.markdown(t)'\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "PAWK - A Python line processor (like AWK)",
"version": "0.8.0",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "60e202cdc272639f3e044679045a11f4fc130f96828038be361ea5a18e8aa280",
"md5": "d32858d59b4f5d8586fbd51db33a6f7d",
"sha256": "374a67627a12a67c6521c85db5a79b49567a4204ccf1b3aa9dc28f75a810d771"
},
"downloads": -1,
"filename": "pawk-0.8.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d32858d59b4f5d8586fbd51db33a6f7d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 7400,
"upload_time": "2023-01-29T19:23:35",
"upload_time_iso_8601": "2023-01-29T19:23:35.412448Z",
"url": "https://files.pythonhosted.org/packages/60/e2/02cdc272639f3e044679045a11f4fc130f96828038be361ea5a18e8aa280/pawk-0.8.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "be19b8c7847050c68f58c32e24f0424509b5999aa715cb00517e003c2382ce63",
"md5": "96c99151b325e986d028588d331e054c",
"sha256": "cfcfb9dd087a9da8e282ef4780c59d780d6c72c56592665d1d0b8667e6e432fa"
},
"downloads": -1,
"filename": "pawk-0.8.0.tar.gz",
"has_sig": false,
"md5_digest": "96c99151b325e986d028588d331e054c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 7067,
"upload_time": "2023-01-29T19:23:37",
"upload_time_iso_8601": "2023-01-29T19:23:37.194757Z",
"url": "https://files.pythonhosted.org/packages/be/19/b8c7847050c68f58c32e24f0424509b5999aa715cb00517e003c2382ce63/pawk-0.8.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-01-29 19:23:37",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "alecthomas",
"github_project": "pawk",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "pawk"
}
