# pyunormalize
A pure Python implementation of the **Unicode normalization algorithm** independent of Python’s core Unicode database. This package conforms to version 16.0 of the Unicode standard, released in September 2024, and has been rigorously tested for accuracy using the official [Unicode test file](https://www.unicode.org/Public/16.0.0/ucd/NormalizationTest.txt).
### Installation and updates
To install the package, run:
```shell
pip install pyunormalize
```
To upgrade to the latest version, run:
```shell
pip install pyunormalize --upgrade
```
### Unicode character database (UCD) version
To retrieve the version of the Unicode character database in use:
```python
>>> from pyunormalize import UCD_VERSION
>>> UCD_VERSION
'16.0.0'
```
### Example usage
```python
>>> from pyunormalize import NFC, NFD, NFKC, NFKD
>>> s = "élève" # "\u00E9\u006C\u00E8\u0076\u0065"
>>> nfc = NFC(s)
>>> nfd = NFD(s)
>>> nfc == s
True
>>> nfd == nfc
False
>>> " ".join([f"{ord(x):04X}" for x in nfc])
'00E9 006C 00E8 0076 0065'
>>> " ".join([f"{ord(x):04X}" for x in nfd])
'0065 0301 006C 0065 0300 0076 0065'
>>>
>>> s = "⑴ ffi ²"
>>> NFC(s), NFKC(s), NFD(s), NFKD(s)
('⑴ ffi ²', '(1) ffi 2', '⑴ ffi ²', '(1) ffi 2')
>>> from pyunormalize import normalize
>>> normalize("NFKD", "⑴ ffi ²")
'(1) ffi 2'
>>> forms = ["NFC", "NFD", "NFKC", "NFKD"]
>>> [normalize(f, "\u017F\u0307\u0323") for f in forms]
['ẛ̣', 'ẛ̣', 'ṩ', 'ṩ']
```
### Related resources
This implementation is based on the following resources:
- [Section 3.11, “Normalization Forms,” in the Unicode core specification, version 16.0.0](https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-3/#G49537)
- [Unicode Standard Annex #15: Unicode Normalization Forms, revision 56](https://www.unicode.org/reports/tr15/tr15-56.html)
### Licenses
The code is licensed under the [MIT license](https://github.com/mlodewijck/pyunormalize/blob/master/LICENSE).
Usage of Unicode data files is subject to the [UNICODE TERMS OF USE](https://www.unicode.org/copyright.html). Additional rights and restrictions regarding Unicode data files and software are outlined in the [Unicode Data Files and Software License](https://www.unicode.org/license.txt), a copy of which is included as [UNICODE-LICENSE](https://github.com/mlodewijck/pyunormalize/blob/master/UNICODE-LICENSE).
Raw data
{
"_id": null,
"home_page": "https://github.com/mlodewijck/pyunormalize",
"name": "pyunormalize",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": null,
"keywords": "Unicode, Unicode data, Unicode normalization, normalization, NFC, NFD, NFKC, NFKD, Unicode Normalization Forms, Canonical Ordering Algorithm, Canonical Composition Algorithm, canonical ordering, canonical composition, Hangul Syllable Composition Algorithm, Hangul Syllable Decomposition Algorithm, Hangul syllables, Hangul jamo characters",
"author": "Marc Lodewijck",
"author_email": "mlodewijck@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/b3/08/568036c725dac746ecb267bb749ef930fb7907454fe69fce83c8557287fb/pyunormalize-16.0.0.tar.gz",
"platform": null,
"description": "# pyunormalize\r\nA pure Python implementation of the **Unicode normalization algorithm** independent of Python\u2019s core Unicode database. This package conforms to version 16.0 of the Unicode standard, released in September 2024, and has been rigorously tested for accuracy using the official [Unicode test file](https://www.unicode.org/Public/16.0.0/ucd/NormalizationTest.txt).\r\n\r\n### Installation and updates\r\nTo install the package, run:\r\n```shell\r\npip install pyunormalize\r\n```\r\n\r\nTo upgrade to the latest version, run:\r\n```shell\r\npip install pyunormalize --upgrade\r\n```\r\n\r\n### Unicode character database (UCD) version\r\nTo retrieve the version of the Unicode character database in use:\r\n```python\r\n>>> from pyunormalize import UCD_VERSION\r\n>>> UCD_VERSION\r\n'16.0.0'\r\n```\r\n\r\n### Example usage\r\n```python\r\n>>> from pyunormalize import NFC, NFD, NFKC, NFKD\r\n>>> s = \"\u00e9l\u00e8ve\" # \"\\u00E9\\u006C\\u00E8\\u0076\\u0065\"\r\n>>> nfc = NFC(s)\r\n>>> nfd = NFD(s)\r\n>>> nfc == s\r\nTrue\r\n>>> nfd == nfc\r\nFalse\r\n>>> \" \".join([f\"{ord(x):04X}\" for x in nfc])\r\n'00E9 006C 00E8 0076 0065'\r\n>>> \" \".join([f\"{ord(x):04X}\" for x in nfd])\r\n'0065 0301 006C 0065 0300 0076 0065'\r\n>>>\r\n>>> s = \"\u2474 \ufb03 \u00b2\"\r\n>>> NFC(s), NFKC(s), NFD(s), NFKD(s)\r\n('\u2474 \ufb03 \u00b2', '(1) ffi 2', '\u2474 \ufb03 \u00b2', '(1) ffi 2')\r\n\r\n>>> from pyunormalize import normalize\r\n>>> normalize(\"NFKD\", \"\u2474 \ufb03 \u00b2\")\r\n'(1) ffi 2'\r\n>>> forms = [\"NFC\", \"NFD\", \"NFKC\", \"NFKD\"]\r\n>>> [normalize(f, \"\\u017F\\u0307\\u0323\") for f in forms]\r\n['\u1e9b\u0323', '\u017f\u0323\u0307', '\u1e69', 's\u0323\u0307']\r\n```\r\n\r\n### Related resources\r\nThis implementation is based on the following resources:\r\n- [Section 3.11, \u201cNormalization Forms,\u201d in the Unicode core specification, version 16.0.0](https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-3/#G49537)\r\n- [Unicode Standard Annex #15: Unicode Normalization Forms, revision 56](https://www.unicode.org/reports/tr15/tr15-56.html)\r\n\r\n### Licenses\r\nThe code is licensed under the [MIT license](https://github.com/mlodewijck/pyunormalize/blob/master/LICENSE).\r\n\r\nUsage of Unicode data files is subject to the [UNICODE TERMS OF USE](https://www.unicode.org/copyright.html). Additional rights and restrictions regarding Unicode data files and software are outlined in the [Unicode Data Files and Software License](https://www.unicode.org/license.txt), a copy of which is included as [UNICODE-LICENSE](https://github.com/mlodewijck/pyunormalize/blob/master/UNICODE-LICENSE).\r\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Unicode normalization forms (NFC, NFKC, NFD, NFKD). A library independent of the Python core Unicode database.",
"version": "16.0.0",
"project_urls": {
"Bug Reports": "https://github.com/mlodewijck/pyunormalize/issues",
"Homepage": "https://github.com/mlodewijck/pyunormalize",
"Source": "https://github.com/mlodewijck/pyunormalize/"
},
"split_keywords": [
"unicode",
" unicode data",
" unicode normalization",
" normalization",
" nfc",
" nfd",
" nfkc",
" nfkd",
" unicode normalization forms",
" canonical ordering algorithm",
" canonical composition algorithm",
" canonical ordering",
" canonical composition",
" hangul syllable composition algorithm",
" hangul syllable decomposition algorithm",
" hangul syllables",
" hangul jamo characters"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "39f99d86e56f716e0651194a5ad58be9c146fcaf1de6901ac6f3cd3affeeb74e",
"md5": "a7a9fa8837a611d03529dbec4254fb32",
"sha256": "c647d95e5d1e2ea9a2f448d1d95d8518348df24eab5c3fd32d2b5c3300a49152"
},
"downloads": -1,
"filename": "pyunormalize-16.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a7a9fa8837a611d03529dbec4254fb32",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 49173,
"upload_time": "2024-09-17T17:08:17",
"upload_time_iso_8601": "2024-09-17T17:08:17.078621Z",
"url": "https://files.pythonhosted.org/packages/39/f9/9d86e56f716e0651194a5ad58be9c146fcaf1de6901ac6f3cd3affeeb74e/pyunormalize-16.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b308568036c725dac746ecb267bb749ef930fb7907454fe69fce83c8557287fb",
"md5": "5936760e58090bb7a54366ccf31be79f",
"sha256": "2e1dfbb4a118154ae26f70710426a52a364b926c9191f764601f5a8cb12761f7"
},
"downloads": -1,
"filename": "pyunormalize-16.0.0.tar.gz",
"has_sig": false,
"md5_digest": "5936760e58090bb7a54366ccf31be79f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 49968,
"upload_time": "2024-09-17T17:08:18",
"upload_time_iso_8601": "2024-09-17T17:08:18.245828Z",
"url": "https://files.pythonhosted.org/packages/b3/08/568036c725dac746ecb267bb749ef930fb7907454fe69fce83c8557287fb/pyunormalize-16.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-17 17:08:18",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mlodewijck",
"github_project": "pyunormalize",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"tox": true,
"lcname": "pyunormalize"
}
JSON
