# It's Not Just Unicode, It's Hemi-Semi-Demicode!
Demicode is a Python command line tool to explore the current, broken state of
fixed-width rendering for [Unicode](https://home.unicode.org) in terminals and
code editors. However, because terminals support styling a program's output with
[ANSI escape sequences](https://en.wikipedia.org/wiki/ANSI_escape_code), they
also are more amenable to helpful visualization than code editors.
- [Fixed-Width Character Blots](#fixed-width-character-blots)
- [Features](#features)
- [Installation](#installation)
- [Versions](#versions)
- [Etc](#etc)
## Fixed-Width Character Blots
Demicode's core functionality is the **fixed-width character blot**, which
visualizes a single grapheme cluster's fixed-width rendering. Since the current
state-of-the-art uses two fixed-width columns at most, each blot is one more
column, that is, three columns wide. That extra padding makes glaringly obvious
when theoretical and actual width diverge. For terminals, said padding comes in
two forms, with the first using ` ` U+0020 space in a different color to
highlight any overlap and the second using `█` U+2588 full block to obstruct
those same bits.
The following screenshot shows an example for **demicode's output
`--with-curation`** when running in Terminal.app on macOS. Out of the nine
terminals I have been testing—[Alacritty](https://alacritty.org),
[Hyper](https://hyper.is), [iTerm2](https://iterm2.com),
[Kitty](https://sw.kovidgoyal.net/kitty/), [Rio](https://raphamorim.io/rio/),
[Terminal.app](https://en.wikipedia.org/wiki/Terminal_(macOS)), [Visual Studio
Code's terminal](https://code.visualstudio.com/docs/terminal/basics),
[Warp](https://www.warp.dev), and
[wezTerm](https://wezfurlong.org/wezterm/index.html)—I find Terminal.app's and
iTerm2's handling of overly wide glyphs the least bad. However, even with
demicode using ANSI escape codes to line up columns, Terminal.app still manages
to distort the column grid, as the lines for the technologist, person: red hair,
and rainbow flag emoji in the screenshot below illustrate. I haven't found an
effective work-around, despite trying several alternatives such as rendering
character information first and blots second.
## Features
Demicode supports the following features:
* Display fixed-width **character blots together with helpful metadata** one
grapheme per line.
* Or, **display `--in-grid`/`-g`** to fit many more graphemes into the same
window, albeit without metadata.
* For code points that combine with variation selectors, **automatically show
the code point without and with applicable variation selectors**.
* Optionally **display blots `--in-more-color`/`-c` and
`--in-dark-mode`/`-d`**. The first option may be given twice for even more
color. The second option usually is superfluous because demicode
automatically detects dark mode. See screenshot below.
* Run `--with-curation` and `--with-…` other **carefully selected groups of
graphemes**. Or provide your own graphemes as regular command line
arguments. Both literal strings and Unicode's `U+…` notation are acceptable.
Quote several `U+…` forms to group them into a grapheme.
* **Automatically download necessary files** from the [Unicode Character
Database](https://unicode.org/ucd/) (UCD) and [Common Locale Data
Repository](https://cldr.unicode.org/) (CLDR) and then cache them locally.
* **Automatically detect the most recent version of the UCD and the CLDR**.
Since CLDR data serves one, non-normative purpose only, emoji sequence
names, demicode always utilizes the latest version. But `--ucd-version` lets
you pick older UCD versions at will.
* In interactive mode, **page the output**. Let user control whether to **go
backward or forward** while also **automatically adjusting to terminal
window size**.
* On Linux and macOS, **page backward and forward with the left and right
arrow keys**. On other operating systems, use `b` or `p` followed by
`‹return›` to page backward; just `‹return›` or alternatively `f` or `n`
followed by `‹return›` to page forward; and just `‹control-c›` or
alternatively `q` or `x` followed by `‹return›` to terminate demicode. All
of these, no `‹return›` required, work on Linux and macOS, too. Plus
`‹delete›` or `‹shift-tab›` to page backward; `‹space›` or `‹tab›` to page
forward; and `‹escape›` to terminate. So which triple is yours?
* In batch mode, i.e., with standard in or out redirected, **emit all
character blots at once and consecutively**.
## Installation
Demicode is written in Python and distributed through
[PyPI](https://pypi.org/project/demicode/), the Python Packaging Index. Since it
utilizes recent language and library features, it **requires Python 3.11** or
later. The best option for installing demicode is using
[pipx](https://pypa.github.io/pipx/). If you haven't installed `pipx` yet,
`brew` makes that easy on Linux or macOS:
```sh
% brew install pipx
==> Fetching pipx
==> Downloading https://ghcr.io/v2/homebrew/core/pipx/manifests/1.2.0
...
🍺 /usr/local/Cellar/pipx/1.2.0: 885 files, 11.2MB
==> Running `brew cleanup pipx`...
Disable this behavior by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
%
```
Once you have `pipx` installed, installing `demicode` is trivial:
```sh
% python --version
Python 3.11.1
% pipx install demicode
installed package demicode 0.5.0, installed using Python 3.11.5
These apps are now globally available
- demicode
done! ✨ 🌟 ✨
% demicode --with-curation
...
```
The output of the last command should look something like the first screenshot.
## Versions
- **v1.3.0** (2024/01/07):
- Automate running demicode in popular terminals on macOS and collecting
either performance data or screenshots.
- Fix crash when mirror directory has not yet been created. Thanks to
[yohann84L](https://github.com/yohann84L) for reporting this bug.
- **v1.2.0** (2023/10/30):
- Mirroring of UCD files has been refactored and now uses an [explicit
manifest](ucd.manifest.json) for tracking locally available versions.
`--ucd-mirror-all` causes demicode to eagerly download all files for all
versions, enabling fully disconnected operation. `--ucd-list-versions`
lists all versions contained in mirror directory.
- With `--inspect-latency`/`-T`, demicode now measures page rendering
latency. [Initial results](perf.json) suggest that the nine measured
terminals are reasonable fast rendering styled text, taking between
4–9 ms on a four-year-old macOS laptop. But when demicode also
queries the terminal for the current column, the spread of average
latencies explodes to 10–946 ms.
- To better track provenance of experimental results, demicode gains the
ability to determine terminal name and version—based on environment
variables, ANSI escape codes, and, on macOS, bundle identifiers.
- Demicode now uses GitHub actions for CI.
- **v1.1.0** (2023/10/17):
- Improve terminal intput/output, notably by `--incrementally`/`-i`
displaying character blots, which is significantly slower but allows for
measuring the size of blots.
- Fix crashing bug in path handling for mirrored CLDR files.
- Make internal handling of UCD data more uniform, with an eye towards
evolving demicode's UCD abstractions into a more generally useful
library.
- Switch from mypy to pyright, address pyright's improved diagnostics,
integrate type checking into `runtest.py`, and improve test script
output.
- **v1.0.0** (2023/09/19):
- Support grapheme cluster segmentation according to Unicode 15.1 and 15.0.
- Tabulate bit size of Unicode properties, alternative groups of required
properties for `--stats`.
- Update internal interface for UCD data to favor generic access to
properties.
- **v1.0.0b1** (2023/09/12):
- In interactive mode, render every page from scratch, taking terminal
size into account. This enables paging forward *and* backward. On Linux
and macOS, use left and right arrow keys to control paging.
- In batch mode, i.e., when standard input or output are redirected, emit
all character blots without paging.
- Test file loading and property look up for every supported UCD version
to squash any remaining crashing bugs. Nonetheless, advise in tool help
that default, i.e., latest version produces best results.
- In preparation of Unicode 15.1, add support for the
Canonical_Combining_Class, Indic_Syllabic_Category, and Script
properties. Remove support for unused Dash, Noncharacter_Code_Point,
Variation_Selector, and White_Space properties again.
- Clean up UCD file loading. Eliminate most boilerplate and private helper
functions in `demicode.ucd`.
- Eliminate global instance of `UnicodeCharacterDatabase`. Leverage
independent instance for collecting statistics, eliminating need for two
tool runs to collect all data.
- **v0.7.0** (2023/09/06) Clearly distinguish between user errors and
unexpected exceptions; print traceback only for the latter. Modularize test
script using `unittest`. In preparation of Unicode 15.1, specify which
versions to use for code generation.
- **v0.6.0** (2023/09/05) Fix handling of emoji data for early versions of
Unicode. Suppress blot for unassigned code points or sequences that are more
than one grapheme cluster; add explanatory note.
- **v0.5.0** (2023/09/04) Optimize range-based Unicode data for space and
bisection speed. Improve built-in selections of graphemes; notably, the
Unicode version oracle now displays exactly one emoji per detectable Unicode
version.
- **v0.4.0** (2023/09/01) Fix bug in URL creation for UCD files and move local
cache to the OS-specific application cache directory. Restructure and
simplify code to compute `width()`, renamed from `wcwidth()` due to changes.
- **v0.3.0** (2023/09/01) Add support for grapheme clusters in addition to
individual code points; account for emoji when calculating width; expose
binary emoji properties; log server accesses; add tests; and improve
property count statistics.
- **v0.2.0–0.2.3** (2023/08/13) First advertised release, with more robust UCD
mirroring, more elaborate output, and support for dark mode. Alas,
screenshot links and README still needed some TLC.
- **v0.1.0** (2023/08/06) First, downlow release
## Etc
The **project name is a play on the name Unicode**: Fixed-width rendering of
Unicode can't get by with a single *uni*-column—from the Latin *unus* for
one—but requires at the very least a *demi*-view—from the Latin *dimidius* for
half via the French *demi* also for half. As so happens, *hemi* and *semi* mean
half as well, tracing back to Greek and Latin origin, respectively.
Alas, the real question is whether **hemisemidemi-anything is cumulative, i.e.,
<sup>1</sup>⁄<sub>8</sub>, or just reinforcing, i.e., still
<sup>1</sup>⁄<sub>2</sub>**.
I am **working on a technical blog post** to provide more on motivation,
technical background, and first findings after blotting far too many Unicode
code points. One unexpected outcome is a test that should identify the Unicode
version supported by a terminal just by displaying a bunch of emoji. 😳
I 💖 Unicode!
---
Demicode is © 2023 [Robert Grimm](https://apparebit.com) and has been released
under the Apache 2.0 license.
Raw data
{
"_id": null,
"home_page": null,
"name": "demicode",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.11",
"maintainer_email": null,
"keywords": "Unicode,fixed-width,East Asian width,wcwidth,hot mess",
"author": null,
"author_email": "Robert Grimm <rgrimm@alum.mit.edu>",
"download_url": null,
"platform": null,
"description": "# It's Not Just Unicode, It's Hemi-Semi-Demicode!\n\nDemicode is a Python command line tool to explore the current, broken state of\nfixed-width rendering for [Unicode](https://home.unicode.org) in terminals and\ncode editors. However, because terminals support styling a program's output with\n[ANSI escape sequences](https://en.wikipedia.org/wiki/ANSI_escape_code), they\nalso are more amenable to helpful visualization than code editors.\n\n - [Fixed-Width Character Blots](#fixed-width-character-blots)\n - [Features](#features)\n - [Installation](#installation)\n - [Versions](#versions)\n - [Etc](#etc)\n\n\n## Fixed-Width Character Blots\n\nDemicode's core functionality is the **fixed-width character blot**, which\nvisualizes a single grapheme cluster's fixed-width rendering. Since the current\nstate-of-the-art uses two fixed-width columns at most, each blot is one more\ncolumn, that is, three columns wide. That extra padding makes glaringly obvious\nwhen theoretical and actual width diverge. For terminals, said padding comes in\ntwo forms, with the first using ` ` U+0020 space in a different color to\nhighlight any overlap and the second using `\u2588` U+2588 full block to obstruct\nthose same bits.\n\nThe following screenshot shows an example for **demicode's output\n`--with-curation`** when running in Terminal.app on macOS. Out of the nine\nterminals I have been testing\u2014[Alacritty](https://alacritty.org),\n[Hyper](https://hyper.is), [iTerm2](https://iterm2.com),\n[Kitty](https://sw.kovidgoyal.net/kitty/), [Rio](https://raphamorim.io/rio/),\n[Terminal.app](https://en.wikipedia.org/wiki/Terminal_(macOS)), [Visual Studio\nCode's terminal](https://code.visualstudio.com/docs/terminal/basics),\n[Warp](https://www.warp.dev), and\n[wezTerm](https://wezfurlong.org/wezterm/index.html)\u2014I find Terminal.app's and\niTerm2's handling of overly wide glyphs the least bad. However, even with\ndemicode using ANSI escape codes to line up columns, Terminal.app still manages\nto distort the column grid, as the lines for the technologist, person: red hair,\nand rainbow flag emoji in the screenshot below illustrate. I haven't found an\neffective work-around, despite trying several alternatives such as rendering\ncharacter information first and blots second.\n\n\n\n\n\n## Features\n\nDemicode supports the following features:\n\n * Display fixed-width **character blots together with helpful metadata** one\n grapheme per line.\n * Or, **display `--in-grid`/`-g`** to fit many more graphemes into the same\n window, albeit without metadata.\n * For code points that combine with variation selectors, **automatically show\n the code point without and with applicable variation selectors**.\n * Optionally **display blots `--in-more-color`/`-c` and\n `--in-dark-mode`/`-d`**. The first option may be given twice for even more\n color. The second option usually is superfluous because demicode\n automatically detects dark mode. See screenshot below.\n * Run `--with-curation` and `--with-\u2026` other **carefully selected groups of\n graphemes**. Or provide your own graphemes as regular command line\n arguments. Both literal strings and Unicode's `U+\u2026` notation are acceptable.\n Quote several `U+\u2026` forms to group them into a grapheme.\n * **Automatically download necessary files** from the [Unicode Character\n Database](https://unicode.org/ucd/) (UCD) and [Common Locale Data\n Repository](https://cldr.unicode.org/) (CLDR) and then cache them locally.\n * **Automatically detect the most recent version of the UCD and the CLDR**.\n Since CLDR data serves one, non-normative purpose only, emoji sequence\n names, demicode always utilizes the latest version. But `--ucd-version` lets\n you pick older UCD versions at will.\n * In interactive mode, **page the output**. Let user control whether to **go\n backward or forward** while also **automatically adjusting to terminal\n window size**.\n * On Linux and macOS, **page backward and forward with the left and right\n arrow keys**. On other operating systems, use `b` or `p` followed by\n `\u2039return\u203a` to page backward; just `\u2039return\u203a` or alternatively `f` or `n`\n followed by `\u2039return\u203a` to page forward; and just `\u2039control-c\u203a` or\n alternatively `q` or `x` followed by `\u2039return\u203a` to terminate demicode. All\n of these, no `\u2039return\u203a` required, work on Linux and macOS, too. Plus\n `\u2039delete\u203a` or `\u2039shift-tab\u203a` to page backward; `\u2039space\u203a` or `\u2039tab\u203a` to page\n forward; and `\u2039escape\u203a` to terminate. So which triple is yours?\n * In batch mode, i.e., with standard in or out redirected, **emit all\n character blots at once and consecutively**.\n\n\n\n\n\n## Installation\n\nDemicode is written in Python and distributed through\n[PyPI](https://pypi.org/project/demicode/), the Python Packaging Index. Since it\nutilizes recent language and library features, it **requires Python 3.11** or\nlater. The best option for installing demicode is using\n[pipx](https://pypa.github.io/pipx/). If you haven't installed `pipx` yet,\n`brew` makes that easy on Linux or macOS:\n\n```sh\n% brew install pipx\n==> Fetching pipx\n==> Downloading https://ghcr.io/v2/homebrew/core/pipx/manifests/1.2.0\n...\n\ud83c\udf7a /usr/local/Cellar/pipx/1.2.0: 885 files, 11.2MB\n==> Running `brew cleanup pipx`...\nDisable this behavior by setting HOMEBREW_NO_INSTALL_CLEANUP.\nHide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).\n%\n```\n\nOnce you have `pipx` installed, installing `demicode` is trivial:\n\n```sh\n% python --version\nPython 3.11.1\n% pipx install demicode\n installed package demicode 0.5.0, installed using Python 3.11.5\n These apps are now globally available\n - demicode\ndone! \u2728 \ud83c\udf1f \u2728\n% demicode --with-curation\n...\n```\n\nThe output of the last command should look something like the first screenshot.\n\n\n## Versions\n\n - **v1.3.0** (2024/01/07):\n - Automate running demicode in popular terminals on macOS and collecting\n either performance data or screenshots.\n - Fix crash when mirror directory has not yet been created. Thanks to\n [yohann84L](https://github.com/yohann84L) for reporting this bug.\n - **v1.2.0** (2023/10/30):\n - Mirroring of UCD files has been refactored and now uses an [explicit\n manifest](ucd.manifest.json) for tracking locally available versions.\n `--ucd-mirror-all` causes demicode to eagerly download all files for all\n versions, enabling fully disconnected operation. `--ucd-list-versions`\n lists all versions contained in mirror directory.\n - With `--inspect-latency`/`-T`, demicode now measures page rendering\n latency. [Initial results](perf.json) suggest that the nine measured\n terminals are reasonable fast rendering styled text, taking between\n 4\u20139 ms on a four-year-old macOS laptop. But when demicode also\n queries the terminal for the current column, the spread of average\n latencies explodes to 10\u2013946 ms.\n - To better track provenance of experimental results, demicode gains the\n ability to determine terminal name and version\u2014based on environment\n variables, ANSI escape codes, and, on macOS, bundle identifiers.\n - Demicode now uses GitHub actions for CI.\n - **v1.1.0** (2023/10/17):\n - Improve terminal intput/output, notably by `--incrementally`/`-i`\n displaying character blots, which is significantly slower but allows for\n measuring the size of blots.\n - Fix crashing bug in path handling for mirrored CLDR files.\n - Make internal handling of UCD data more uniform, with an eye towards\n evolving demicode's UCD abstractions into a more generally useful\n library.\n - Switch from mypy to pyright, address pyright's improved diagnostics,\n integrate type checking into `runtest.py`, and improve test script\n output.\n - **v1.0.0** (2023/09/19):\n - Support grapheme cluster segmentation according to Unicode 15.1 and 15.0.\n - Tabulate bit size of Unicode properties, alternative groups of required\n properties for `--stats`.\n - Update internal interface for UCD data to favor generic access to\n properties.\n - **v1.0.0b1** (2023/09/12):\n - In interactive mode, render every page from scratch, taking terminal\n size into account. This enables paging forward *and* backward. On Linux\n and macOS, use left and right arrow keys to control paging.\n - In batch mode, i.e., when standard input or output are redirected, emit\n all character blots without paging.\n - Test file loading and property look up for every supported UCD version\n to squash any remaining crashing bugs. Nonetheless, advise in tool help\n that default, i.e., latest version produces best results.\n - In preparation of Unicode 15.1, add support for the\n Canonical_Combining_Class, Indic_Syllabic_Category, and Script\n properties. Remove support for unused Dash, Noncharacter_Code_Point,\n Variation_Selector, and White_Space properties again.\n - Clean up UCD file loading. Eliminate most boilerplate and private helper\n functions in `demicode.ucd`.\n - Eliminate global instance of `UnicodeCharacterDatabase`. Leverage\n independent instance for collecting statistics, eliminating need for two\n tool runs to collect all data.\n - **v0.7.0** (2023/09/06) Clearly distinguish between user errors and\n unexpected exceptions; print traceback only for the latter. Modularize test\n script using `unittest`. In preparation of Unicode 15.1, specify which\n versions\u00a0to use for code generation.\n - **v0.6.0** (2023/09/05) Fix handling of emoji data for early versions of\n Unicode. Suppress blot for unassigned code points or sequences that are more\n than one grapheme cluster; add explanatory note.\n - **v0.5.0** (2023/09/04) Optimize range-based Unicode data for space and\n bisection speed. Improve built-in selections of graphemes; notably, the\n Unicode version oracle now displays exactly one emoji per detectable Unicode\n version.\n - **v0.4.0** (2023/09/01) Fix bug in URL creation for UCD files and move local\n cache to the OS-specific application cache directory. Restructure and\n simplify code to compute `width()`, renamed from `wcwidth()` due to changes.\n - **v0.3.0** (2023/09/01) Add support for grapheme clusters in addition to\n individual code points; account for emoji when calculating width; expose\n binary emoji properties; log server accesses; add tests; and improve\n property count statistics.\n - **v0.2.0\u20130.2.3** (2023/08/13) First advertised release, with more robust UCD\n mirroring, more elaborate output, and support for dark mode. Alas,\n screenshot links and README still needed some TLC.\n - **v0.1.0** (2023/08/06) First, downlow release\n\n\n## Etc\n\nThe **project name is a play on the name Unicode**: Fixed-width rendering of\nUnicode can't get by with a single *uni*-column\u2014from the Latin *unus* for\none\u2014but requires at the very least a *demi*-view\u2014from the Latin *dimidius* for\nhalf via the French *demi* also for half. As so happens, *hemi* and *semi* mean\nhalf as well, tracing back to Greek and Latin origin, respectively.\n\nAlas, the real question is whether **hemisemidemi-anything is cumulative, i.e.,\n<sup>1</sup>⁄<sub>8</sub>, or just reinforcing, i.e., still\n<sup>1</sup>⁄<sub>2</sub>**.\n\nI am **working on a technical blog post** to provide more on motivation,\ntechnical background, and first findings after blotting far too many Unicode\ncode points. One unexpected outcome is a test that should identify the Unicode\nversion supported by a terminal just by displaying a bunch of emoji. \ud83d\ude33\n\nI \ud83d\udc96 Unicode!\n\n---\n\nDemicode is \u00a9 2023 [Robert Grimm](https://apparebit.com) and has been released\nunder the Apache 2.0 license.\n",
"bugtrack_url": null,
"license": null,
"summary": "Fixed-width display of Unicode is deeply broken",
"version": "1.3.0",
"project_urls": {
"repository": "https://github.com/apparebit/demicode"
},
"split_keywords": [
"unicode",
"fixed-width",
"east asian width",
"wcwidth",
"hot mess"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "565b96b21578e915d8d918f1da6dc3d928518f4cd969d83abf7a3de4b2c2c715",
"md5": "2d6bb236467faedb9a934e14b6716397",
"sha256": "3ae555a5d9e18d48519883cf23cf6e29ed02023701154f48ba47eade921553f7"
},
"downloads": -1,
"filename": "demicode-1.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "2d6bb236467faedb9a934e14b6716397",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.11",
"size": 77309,
"upload_time": "2024-01-07T22:27:00",
"upload_time_iso_8601": "2024-01-07T22:27:00.409878Z",
"url": "https://files.pythonhosted.org/packages/56/5b/96b21578e915d8d918f1da6dc3d928518f4cd969d83abf7a3de4b2c2c715/demicode-1.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-07 22:27:00",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "apparebit",
"github_project": "demicode",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "demicode"
}
JSON
