# pyunormalize
A pure Python implementation of the **Unicode normalization algorithm** independent from the Python core Unicode database. This package supports version 15.1 of the Unicode standard (released in September 2023). It has been thoroughly tested against the [Unicode test file](https://www.unicode.org/Public/15.1.0/ucd/NormalizationTest.txt).
For the formal specification of the Unicode normalization algorithm, see [Section 3.11, Normalization Forms](https://www.unicode.org/versions/Unicode15.1.0/ch03.pdf#G49537), in the Unicode core specification.
### Installation
The easiest method to install is using pip:
```shell
pip install pyunormalize
```
### UCD version
To get the version of the Unicode character database currently used:
```python
>>> from pyunormalize import UCD_VERSION
>>> UCD_VERSION
'15.1.0'
```
### Example usage
```python
>>> from pyunormalize import NFC, NFD, NFKC, NFKD
>>> s = "élève" # "\u00E9\u006C\u00E8\u0076\u0065"
>>> nfc = NFC(s)
>>> nfd = NFD(s)
>>> nfc == s
True
>>> nfd == nfc
False
>>> " ".join([f"{ord(x):04X}" for x in nfc])
'00E9 006C 00E8 0076 0065'
>>> " ".join([f"{ord(x):04X}" for x in nfd])
'0065 0301 006C 0065 0300 0076 0065'
>>>
>>> s = "⑴ ffi ²"
>>> NFC(s), NFKC(s), NFD(s), NFKD(s)
('⑴ ffi ²', '(1) ffi 2', '⑴ ffi ²', '(1) ffi 2')
>>> from pyunormalize import normalize
>>> normalize("NFKD", "⑴ ffi ²")
'(1) ffi 2'
>>> forms = ["NFC", "NFD", "NFKC", "NFKD"]
>>> [normalize(f, "\u017F\u0307\u0323") for f in forms]
['ẛ̣', 'ẛ̣', 'ṩ', 'ṩ']
```
### Related resources
This implementation is based on the following resources:
- [Section 3.11, Normalization Forms, in the Unicode core specification, version 15.1.0](https://www.unicode.org/versions/Unicode15.1.0/ch03.pdf#G49537)
- [Unicode Standard Annex #15: Unicode Normalization Forms, version 54](https://www.unicode.org/reports/tr15/tr15-54.html)
### Licenses
The code is available under the [MIT license](https://github.com/mlodewijck/pyunormalize/blob/master/LICENSE).
Usage of Unicode data files is governed by the [UNICODE TERMS OF USE](https://www.unicode.org/copyright.html). Further specifications of rights and restrictions pertaining to the use of the Unicode data files and software can be found in the [Unicode Data Files and Software License](https://www.unicode.org/license.txt), a copy of which is included as [UNICODE-LICENSE](https://github.com/mlodewijck/pyunormalize/blob/master/UNICODE-LICENSE).
Raw data
{
"_id": null,
"home_page": "https://github.com/mlodewijck/pyunormalize",
"name": "pyunormalize",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": "",
"keywords": "Unicode,Unicode data,Unicode normalization,normalization,NFC,NFD,NFKC,NFKD,Unicode Normalization Forms,Canonical Ordering Algorithm,Canonical Composition Algorithm,canonical ordering,canonical composition,Hangul Syllable Composition Algorithm,Hangul Syllable Decomposition Algorithm,Hangul syllables,Hangul jamo characters",
"author": "Marc Lodewijck",
"author_email": "mlodewijck@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/dc/5b/ddc89263363422c0d52fdc0a4d88a126621d5cb60359cd45679d3c0447fc/pyunormalize-15.1.0.tar.gz",
"platform": null,
"description": "# pyunormalize\r\nA pure Python implementation of the **Unicode normalization algorithm** independent from the Python core Unicode database. This package supports version 15.1 of the Unicode standard (released in September 2023). It has been thoroughly tested against the [Unicode test file](https://www.unicode.org/Public/15.1.0/ucd/NormalizationTest.txt).\r\n\r\nFor the formal specification of the Unicode normalization algorithm, see [Section 3.11, Normalization Forms](https://www.unicode.org/versions/Unicode15.1.0/ch03.pdf#G49537), in the Unicode core specification.\r\n\r\n### Installation\r\nThe easiest method to install is using pip:\r\n```shell\r\npip install pyunormalize\r\n```\r\n\r\n### UCD version\r\nTo get the version of the Unicode character database currently used:\r\n```python\r\n>>> from pyunormalize import UCD_VERSION\r\n>>> UCD_VERSION\r\n'15.1.0'\r\n```\r\n\r\n### Example usage\r\n```python\r\n>>> from pyunormalize import NFC, NFD, NFKC, NFKD\r\n>>> s = \"\u00e9l\u00e8ve\" # \"\\u00E9\\u006C\\u00E8\\u0076\\u0065\"\r\n>>> nfc = NFC(s)\r\n>>> nfd = NFD(s)\r\n>>> nfc == s\r\nTrue\r\n>>> nfd == nfc\r\nFalse\r\n>>> \" \".join([f\"{ord(x):04X}\" for x in nfc])\r\n'00E9 006C 00E8 0076 0065'\r\n>>> \" \".join([f\"{ord(x):04X}\" for x in nfd])\r\n'0065 0301 006C 0065 0300 0076 0065'\r\n>>>\r\n>>> s = \"\u2474 \ufb03 \u00b2\"\r\n>>> NFC(s), NFKC(s), NFD(s), NFKD(s)\r\n('\u2474 \ufb03 \u00b2', '(1) ffi 2', '\u2474 \ufb03 \u00b2', '(1) ffi 2')\r\n\r\n>>> from pyunormalize import normalize\r\n>>> normalize(\"NFKD\", \"\u2474 \ufb03 \u00b2\")\r\n'(1) ffi 2'\r\n>>> forms = [\"NFC\", \"NFD\", \"NFKC\", \"NFKD\"]\r\n>>> [normalize(f, \"\\u017F\\u0307\\u0323\") for f in forms]\r\n['\u1e9b\u0323', '\u017f\u0323\u0307', '\u1e69', 's\u0323\u0307']\r\n```\r\n\r\n### Related resources\r\nThis implementation is based on the following resources:\r\n- [Section 3.11, Normalization Forms, in the Unicode core specification, version 15.1.0](https://www.unicode.org/versions/Unicode15.1.0/ch03.pdf#G49537)\r\n- [Unicode Standard Annex #15: Unicode Normalization Forms, version 54](https://www.unicode.org/reports/tr15/tr15-54.html)\r\n\r\n### Licenses\r\nThe code is available under the [MIT license](https://github.com/mlodewijck/pyunormalize/blob/master/LICENSE).\r\n\r\nUsage of Unicode data files is governed by the [UNICODE TERMS OF USE](https://www.unicode.org/copyright.html). Further specifications of rights and restrictions pertaining to the use of the Unicode data files and software can be found in the [Unicode Data Files and Software License](https://www.unicode.org/license.txt), a copy of which is included as [UNICODE-LICENSE](https://github.com/mlodewijck/pyunormalize/blob/master/UNICODE-LICENSE).\r\n\r\n\r\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Unicode normalization forms (NFC, NFKC, NFD, NFKD). A library independent from the Python core Unicode database.",
"version": "15.1.0",
"project_urls": {
"Bug Reports": "https://github.com/mlodewijck/pyunormalize/issues",
"Homepage": "https://github.com/mlodewijck/pyunormalize",
"Source": "https://github.com/mlodewijck/pyunormalize/"
},
"split_keywords": [
"unicode",
"unicode data",
"unicode normalization",
"normalization",
"nfc",
"nfd",
"nfkc",
"nfkd",
"unicode normalization forms",
"canonical ordering algorithm",
"canonical composition algorithm",
"canonical ordering",
"canonical composition",
"hangul syllable composition algorithm",
"hangul syllable decomposition algorithm",
"hangul syllables",
"hangul jamo characters"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "dc5bddc89263363422c0d52fdc0a4d88a126621d5cb60359cd45679d3c0447fc",
"md5": "31aaa8cbca6e409dec59b07d4ea24419",
"sha256": "cf4a87451a0f1cb76911aa97f432f4579e1f564a2f0c84ce488c73a73901b6c1"
},
"downloads": -1,
"filename": "pyunormalize-15.1.0.tar.gz",
"has_sig": false,
"md5_digest": "31aaa8cbca6e409dec59b07d4ea24419",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 515497,
"upload_time": "2023-11-11T13:43:59",
"upload_time_iso_8601": "2023-11-11T13:43:59.895062Z",
"url": "https://files.pythonhosted.org/packages/dc/5b/ddc89263363422c0d52fdc0a4d88a126621d5cb60359cd45679d3c0447fc/pyunormalize-15.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-11 13:43:59",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mlodewijck",
"github_project": "pyunormalize",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "pyunormalize"
}
JSON
