# Pakettic
Pakettic is a command-line tool for minifying and compressing
[TIC-80](http://tic80.com/) fantasy console carts. The tool is written
in Python (3.9+) and used especially for
[sizecoding](http://www.sizecoding.org/wiki/TIC-80). It compresses
existing carts approximately ~1.2% better than best alternatives, and by
using its [magic comments](#magic-comments), pakettic might find code
that compresses even better.
## Installation
Installing with pip:
```bash
$ pip install pakettic
```
Installing the latest main branch from GitHub:
```bash
$ pip install git+https://github.com/vsariola/pakettic.git@main
```
Installing a checked out version of the repository:
```bash
$ pip install -e path/to/pakettic
```
Installing a checked out version of the repository using
[poetry](https://python-poetry.org/) for a nice virtual environment with
locked dependencies (run inside the pakettic folder):
```bash
$ poetry install
```
## Usage
To compress a cart, run:
```bash
$ pakettic path/to/cart.tic
```
If your PATH is not configured to include pip installed executables, you
can use
```bash
$ python -m pakettic path/to/cart.tic
```
If you installed using poetry into a virtual environment, you need to
prepend `poetry run` before every command e.g.
```bash
$ poetry run pakettic path/to/cart.tic
```
Pakettic supports both .tic and .lua carts. Multiple input files may be
defined. Input files are globbed, so `?`, `*`, and `**` work as
wildcards for a single character, multiple characters and a directory,
respectively.
For a full list of command line options, see:
```bash
$ pakettic --help
```
See also [tips for command line arguments](#tips-for-command-line-arguments)
Running all tests:
```bash
$ poetry run python -m unittest discover -s tests
```
## Features
Pakettic first parses the LUA-script to an abstract syntax tree, and
then uses a local optimization algorithm
([simulated annealing](https://en.wikipedia.org/wiki/Simulated_annealing),
[late acceptance hill climbing](https://arxiv.org/pdf/1806.09328.pdf) or
its variant diversified late acceptance search) to randomly mutate the
syntax tree & see if it compresses better. Implemented mutations
include:
- shortening variable names
- flipping comparisons `>`, `<`, `>=`, `<=`, `~=`, and `==`
- reordering arithmetic operators `+`, `-`, `*` and `/` and bit logic
operators `&`, `~` and `|`
- converting `a^2` into `a*a` and vice versa
- using either single or double quotes for all strings
- converting whole hexadecimals into decimals
- convert `for a,b,1 do` into `for a,b do` and vice versa
- reordering statements: statements that can be reordered are marked
with [magic comments](#magic-comments)
- alternative expressions: alternatives are marked with
[magic comments](#magic-comments)
- folding constant expressions
Internally, pakettic uses [zopfli](https://github.com/google/zopfli) for
the compression.
`load'<code>'` is parsed as `function(...)<code>end` so you can easily
recompress already compressed carts. Conversely, `function()<code>end`
or `function(...)<code>end` is replaced with `load'<code>'` during
compression.
Note that `function(...)<code>end` and `load'<code>'` are not 100%
semantically identical: the load version cannot access locals in the outer
scope. For example: `local x="hello" function f()print(x)end` works but
`local x="hello" f=load'print(x)'` does not. Since locals are rarely
used in size-coding, we default to using the load-trick, but you can
disable it with the command-line parameter `--no-load`.
However, pakettic does not convert functions with parameters. In
particular, pakettic does not automatically convert
`function SCN(x)<code>end` into `SCN=load'x=...<code>'`, because they
are not semantically identical: in the load version, `x` is now global
and thus could trash a global variable, unintentionally breaking the
cart. To make `SCN` compress nicely, you have to write it as
`function SCN(...)x=...<code>end`, taking responsibility for `x` not
overwriting anything important.
Unnecessary parentheses are removed from expressions so you do not have
to worry about those.
## Magic comments
### Reorderable statements
The algorithm will try to reorder statements between `--{` and `--}`.
For example:
```lua
--{
a="hello"
b="world"
--}
```
will try both `a="hello"b="world"` and `b="world"a="hello"` to see which
compresses better.
Notice that only complete statements can be reordered. Thus, this will
NOT work:
```lua
--{
for x=0,239 do
for y=0,135 do
--}
end
end
```
A good rule of thumb is that you should be able to replace `--{` and
`--}` with `do` and `end`, respectively, and still have valid code.
Statements between `--{!` and `--}` are not ordered, so you can make
blocks of statements that are kept in order within a pair of `--{` and
`--}` tags.
### Alternative expressions
There is a special `--|` operator that allows alternative expressions to
be tested, to see if they compress better. For example: `5--|4--|6`
means that the algorithm will try 4 and 6 in place of the 5. This will
naturally show up as a comment in LUA so you will have to continue the
expression on next line if this is in the middle of an expression. `--|`
has the lowest precedence, even lower than `^`, so put parentheses if
you want to try more complicated expressions e.g. `(x//256)--|(x>>8)`
### Debug code
Pakettic treats `--![` and `--!]` as multiline comment tags, while LUA
treats these as single line comments. Useful for including debug code in
the unpacked intro: the code will not be included in the packed cart.
## Tips for command line arguments
- If pakettic complains about CODE_ZIP chunk size, the code is just too
big after compression. In TIC-80, CODE_ZIP chunks do not support
multiple banks (and likely never will, as the feature is already
deprecated), and thus are unfortunately limited to 65535 bytes.
`--uncompressed` is a temporary fix, but code will be uncompressed and
thus the size much larger.
- The Zopfli compression level can be set with `-z<level>`, with level
ranging from 0 to 5. When developing, start with `-z0` for fast
optimization, and only increase when necessary e.g. when you are just
a few bytes over the limit. The default Zopfli-level is 0.
- The algorithm uses a pseudorandom generator. Sometimes using a
different seed finds a few byte better or worse solution. Use command
line argument `--seed` to try different seeds.
- Similarly, different optimization heuristics produce slightly
different results. Try different heuristics e.g. with `-alahc`,
`-adlas` or `-aanneal`.
- To avoid re-optimizing all the expressions every time, do a long
optimization run, study the results and change your expressions to the
forms that pack well. Set the number of steps with `-s`. Use
command-line argument `-p` to always print a reasonably readable
version of the best solution when one is found.
- By default, pakettic only includes CODE and DEFAULT chunks. DEFAULT
indicates that before loading the cart, TIC-80 loads the default cart,
setting default palette, waveforms etc. If you don't need the default
values (e.g. you set the palette yourself), save one byte by only
including CODE chunk in the cart: `-ccode`
- Working on a tweet-cart? Use `-l` to output LUA carts, which are
uncompressed. The optimization algorithm then just optimizes the
uncompressed size of the code.
- If the packed cart is missing sprites, music, map etc., try adding
`-call` (or something more specific) to include necessary chunks.
- Do you want to use the TIC-80 sprites or the tracker, but don't like
the fact that the data chunks are uncompressed? Use `-d` to have
pakettic automatically convert all data chunks into hexadecimal
strings in the code, along with a small stub placed at the beginning
of the code that interprets the string and loads the data at correct
address. For example,
`-d -cCODE,MUSIC,PATTERNS,WAVEFORM,SAMPLES,DEFAULT` would include the
necessary chunks for the music.
## Known issues
- At the moment, all the branches of swappable operators are assumed to
be without side effects. If they have side-effects, the swapping might
inadvertedly swap the execution order of the two branches.
- The parser can crash with large carts. Carts in the size coding range
(few thousand characters) do not seem to cause problems, but crashes
have been observed parsing carts with tens of thousands of code
characters. This may be related to how the pyparsing grammar is
defined, which could result in highly recursive parsing and eventually
stack overflows.
## Credits
Code contributors: [Veikko Sariola/pestis](https://github.com/vsariola), [wojciech-graj](https://github.com/wojciech-graj),
[koorogi](https://github.com/koorogi), [dezgeg](https://github.com/dezgeg)
Test corpus contributors: [psenough](corpus/psenough/), [ilmenit](corpus/ilmenit/),
[gigabates](corpus/gigabates/), [gasman](corpus/gasman/), [pellicus](corpus/pellicus/),
[luchak](corpus/psenough/fabracid.lua).
## License
[MIT](https://choosealicense.com/licenses/mit/)
The test corpus carts have their own licenses, see the license files in
the subdirectories of the [corpus](corpus/) directory.
Raw data
{
"_id": null,
"home_page": "https://github.com/vsariola/pakettic",
"name": "pakettic",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.9,<4.0",
"maintainer_email": "",
"keywords": "TIC-80,Fantasy Console,Compression,LUA",
"author": "Veikko Sariola",
"author_email": "5684185+vsariola@users.noreply.github.com",
"download_url": "https://files.pythonhosted.org/packages/72/a0/54c33920d4df7189b6cc6e1c8770bc453fed0ac5a94d6ec17ce30825f662/pakettic-1.3.1.tar.gz",
"platform": null,
"description": "# Pakettic\n\nPakettic is a command-line tool for minifying and compressing\n[TIC-80](http://tic80.com/) fantasy console carts. The tool is written\nin Python (3.9+) and used especially for\n[sizecoding](http://www.sizecoding.org/wiki/TIC-80). It compresses\nexisting carts approximately ~1.2% better than best alternatives, and by\nusing its [magic comments](#magic-comments), pakettic might find code\nthat compresses even better.\n\n## Installation\n\nInstalling with pip:\n\n```bash\n$ pip install pakettic\n```\n\nInstalling the latest main branch from GitHub:\n\n```bash\n$ pip install git+https://github.com/vsariola/pakettic.git@main\n```\n\nInstalling a checked out version of the repository:\n\n```bash\n$ pip install -e path/to/pakettic\n```\n\nInstalling a checked out version of the repository using\n[poetry](https://python-poetry.org/) for a nice virtual environment with\nlocked dependencies (run inside the pakettic folder):\n\n```bash\n$ poetry install\n```\n\n## Usage\n\nTo compress a cart, run:\n\n```bash\n$ pakettic path/to/cart.tic\n```\n\nIf your PATH is not configured to include pip installed executables, you\ncan use\n\n```bash\n$ python -m pakettic path/to/cart.tic\n```\n\nIf you installed using poetry into a virtual environment, you need to\nprepend `poetry run` before every command e.g.\n\n```bash\n$ poetry run pakettic path/to/cart.tic\n```\n\nPakettic supports both .tic and .lua carts. Multiple input files may be\ndefined. Input files are globbed, so `?`, `*`, and `**` work as\nwildcards for a single character, multiple characters and a directory,\nrespectively.\n\nFor a full list of command line options, see:\n\n```bash\n$ pakettic --help\n```\n\nSee also [tips for command line arguments](#tips-for-command-line-arguments)\n\nRunning all tests:\n\n```bash\n$ poetry run python -m unittest discover -s tests\n```\n\n## Features\n\nPakettic first parses the LUA-script to an abstract syntax tree, and\nthen uses a local optimization algorithm\n([simulated annealing](https://en.wikipedia.org/wiki/Simulated_annealing),\n[late acceptance hill climbing](https://arxiv.org/pdf/1806.09328.pdf) or\nits variant diversified late acceptance search) to randomly mutate the\nsyntax tree & see if it compresses better. Implemented mutations\ninclude:\n - shortening variable names\n - flipping comparisons `>`, `<`, `>=`, `<=`, `~=`, and `==`\n - reordering arithmetic operators `+`, `-`, `*` and `/` and bit logic\n operators `&`, `~` and `|`\n - converting `a^2` into `a*a` and vice versa\n - using either single or double quotes for all strings\n - converting whole hexadecimals into decimals\n - convert `for a,b,1 do` into `for a,b do` and vice versa\n - reordering statements: statements that can be reordered are marked\n with [magic comments](#magic-comments)\n - alternative expressions: alternatives are marked with\n [magic comments](#magic-comments)\n - folding constant expressions\n\nInternally, pakettic uses [zopfli](https://github.com/google/zopfli) for\nthe compression.\n\n`load'<code>'` is parsed as `function(...)<code>end` so you can easily\nrecompress already compressed carts. Conversely, `function()<code>end`\nor `function(...)<code>end` is replaced with `load'<code>'` during\ncompression.\n\nNote that `function(...)<code>end` and `load'<code>'` are not 100%\nsemantically identical: the load version cannot access locals in the outer\nscope. For example: `local x=\"hello\" function f()print(x)end` works but\n`local x=\"hello\" f=load'print(x)'` does not. Since locals are rarely\nused in size-coding, we default to using the load-trick, but you can\ndisable it with the command-line parameter `--no-load`.\n\nHowever, pakettic does not convert functions with parameters. In\nparticular, pakettic does not automatically convert\n`function SCN(x)<code>end` into `SCN=load'x=...<code>'`, because they\nare not semantically identical: in the load version, `x` is now global\nand thus could trash a global variable, unintentionally breaking the\ncart. To make `SCN` compress nicely, you have to write it as\n`function SCN(...)x=...<code>end`, taking responsibility for `x` not\noverwriting anything important.\n\nUnnecessary parentheses are removed from expressions so you do not have\nto worry about those.\n\n## Magic comments\n\n### Reorderable statements\n\nThe algorithm will try to reorder statements between `--{` and `--}`.\nFor example:\n\n```lua\n --{\n a=\"hello\"\n b=\"world\"\n --}\n```\n\nwill try both `a=\"hello\"b=\"world\"` and `b=\"world\"a=\"hello\"` to see which\ncompresses better.\n\nNotice that only complete statements can be reordered. Thus, this will\nNOT work:\n\n```lua\n --{\n for x=0,239 do\n for y=0,135 do\n --}\n end\n end\n```\n\nA good rule of thumb is that you should be able to replace `--{` and\n`--}` with `do` and `end`, respectively, and still have valid code.\n\nStatements between `--{!` and `--}` are not ordered, so you can make\nblocks of statements that are kept in order within a pair of `--{` and\n`--}` tags.\n\n### Alternative expressions\n\nThere is a special `--|` operator that allows alternative expressions to\nbe tested, to see if they compress better. For example: `5--|4--|6`\nmeans that the algorithm will try 4 and 6 in place of the 5. This will\nnaturally show up as a comment in LUA so you will have to continue the\nexpression on next line if this is in the middle of an expression. `--|`\nhas the lowest precedence, even lower than `^`, so put parentheses if\nyou want to try more complicated expressions e.g. `(x//256)--|(x>>8)`\n\n### Debug code\n\nPakettic treats `--![` and `--!]` as multiline comment tags, while LUA\ntreats these as single line comments. Useful for including debug code in\nthe unpacked intro: the code will not be included in the packed cart.\n\n## Tips for command line arguments\n\n- If pakettic complains about CODE_ZIP chunk size, the code is just too\n big after compression. In TIC-80, CODE_ZIP chunks do not support\n multiple banks (and likely never will, as the feature is already\n deprecated), and thus are unfortunately limited to 65535 bytes.\n `--uncompressed` is a temporary fix, but code will be uncompressed and\n thus the size much larger.\n- The Zopfli compression level can be set with `-z<level>`, with level\n ranging from 0 to 5. When developing, start with `-z0` for fast\n optimization, and only increase when necessary e.g. when you are just\n a few bytes over the limit. The default Zopfli-level is 0.\n- The algorithm uses a pseudorandom generator. Sometimes using a\n different seed finds a few byte better or worse solution. Use command\n line argument `--seed` to try different seeds.\n- Similarly, different optimization heuristics produce slightly\n different results. Try different heuristics e.g. with `-alahc`,\n `-adlas` or `-aanneal`.\n- To avoid re-optimizing all the expressions every time, do a long\n optimization run, study the results and change your expressions to the\n forms that pack well. Set the number of steps with `-s`. Use\n command-line argument `-p` to always print a reasonably readable\n version of the best solution when one is found.\n- By default, pakettic only includes CODE and DEFAULT chunks. DEFAULT\n indicates that before loading the cart, TIC-80 loads the default cart,\n setting default palette, waveforms etc. If you don't need the default\n values (e.g. you set the palette yourself), save one byte by only\n including CODE chunk in the cart: `-ccode`\n- Working on a tweet-cart? Use `-l` to output LUA carts, which are\n uncompressed. The optimization algorithm then just optimizes the\n uncompressed size of the code.\n- If the packed cart is missing sprites, music, map etc., try adding\n `-call` (or something more specific) to include necessary chunks.\n- Do you want to use the TIC-80 sprites or the tracker, but don't like\n the fact that the data chunks are uncompressed? Use `-d` to have\n pakettic automatically convert all data chunks into hexadecimal\n strings in the code, along with a small stub placed at the beginning\n of the code that interprets the string and loads the data at correct\n address. For example,\n `-d -cCODE,MUSIC,PATTERNS,WAVEFORM,SAMPLES,DEFAULT` would include the\n necessary chunks for the music.\n\n## Known issues\n\n- At the moment, all the branches of swappable operators are assumed to\n be without side effects. If they have side-effects, the swapping might\n inadvertedly swap the execution order of the two branches.\n- The parser can crash with large carts. Carts in the size coding range\n (few thousand characters) do not seem to cause problems, but crashes\n have been observed parsing carts with tens of thousands of code\n characters. This may be related to how the pyparsing grammar is\n defined, which could result in highly recursive parsing and eventually\n stack overflows.\n\n## Credits\n\nCode contributors: [Veikko Sariola/pestis](https://github.com/vsariola), [wojciech-graj](https://github.com/wojciech-graj),\n[koorogi](https://github.com/koorogi), [dezgeg](https://github.com/dezgeg)\n\nTest corpus contributors: [psenough](corpus/psenough/), [ilmenit](corpus/ilmenit/),\n[gigabates](corpus/gigabates/), [gasman](corpus/gasman/), [pellicus](corpus/pellicus/),\n[luchak](corpus/psenough/fabracid.lua).\n\n## License\n\n[MIT](https://choosealicense.com/licenses/mit/)\n\nThe test corpus carts have their own licenses, see the license files in\nthe subdirectories of the [corpus](corpus/) directory.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A tool for minifying and compressing TIC-80 fantasy console carts",
"version": "1.3.1",
"project_urls": {
"Homepage": "https://github.com/vsariola/pakettic",
"Repository": "https://github.com/vsariola/pakettic"
},
"split_keywords": [
"tic-80",
"fantasy console",
"compression",
"lua"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e7a7eaf642383a60e7580a2455e65797d8d75a89636187d01b77471ee8fa89bf",
"md5": "1d7897b14ffab8f77a782284e3e96503",
"sha256": "f34b50075af390980a6e1ef3ac03520d08c098ef323b8abe929fb344bafca48d"
},
"downloads": -1,
"filename": "pakettic-1.3.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "1d7897b14ffab8f77a782284e3e96503",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9,<4.0",
"size": 28586,
"upload_time": "2024-01-09T08:24:34",
"upload_time_iso_8601": "2024-01-09T08:24:34.475771Z",
"url": "https://files.pythonhosted.org/packages/e7/a7/eaf642383a60e7580a2455e65797d8d75a89636187d01b77471ee8fa89bf/pakettic-1.3.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "72a054c33920d4df7189b6cc6e1c8770bc453fed0ac5a94d6ec17ce30825f662",
"md5": "b312d51e93015e5eb4d67f67a07ccb61",
"sha256": "c79f7de26c7659c69f67c71af6f192fee71259d9d7f40eb794f1beaaa35e5a2b"
},
"downloads": -1,
"filename": "pakettic-1.3.1.tar.gz",
"has_sig": false,
"md5_digest": "b312d51e93015e5eb4d67f67a07ccb61",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9,<4.0",
"size": 27236,
"upload_time": "2024-01-09T08:24:36",
"upload_time_iso_8601": "2024-01-09T08:24:36.337338Z",
"url": "https://files.pythonhosted.org/packages/72/a0/54c33920d4df7189b6cc6e1c8770bc453fed0ac5a94d6ec17ce30825f662/pakettic-1.3.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-09 08:24:36",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "vsariola",
"github_project": "pakettic",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "pakettic"
}
JSON
