# indicparser
Grapheme Parser for indic languages
# Installaton
```python
pip install indicparser
```
# Useage
* initializing the parser
```python
from indicparser import graphemeParser
gp=graphemeParser("bangla")
```
* extracting graphemes
```python
text=" শাটিকাপ মার"
graphemes=gp.process(text)
print("Graphemes:",graphemes)
```
> Graphemes: [' ', ' ', 'শা', 'টি', 'কা', 'প', ' ', ' ', ' ', 'মা', 'র']
* extracting graphemes but merging spaces and clearing initial and ending space
```python
graphemes=gp.process(text,merge_spaces=True)
print("Graphemes (space corrected):",graphemes)
```
> Graphemes (space corrected): ['শা', 'টি', 'কা', 'প', ' ', 'মা', 'র']
* treatment of numbers and puntucation and english is also available by default
```python
text="এটাকি 2441139 ? না ভাই wrong number"
graphemes=gp.process(text,merge_spaces=True)
print("Graphemes:",graphemes)
```
> Graphemes: ['এ', 'টা', 'কি', ' ', '2', '4', '4', '1', '1', '3', '9', ' ', '?', ' ', 'না', ' ', 'ভা', 'ই', ' ', 'w', 'r', 'o', 'n', 'g', ' ', 'n', 'u', 'm', 'b', 'e', 'r']
* available languages
```python
from indicparser import languages
languages.keys()
```
> dict_keys(['bangla', 'malyalam', 'tamil', 'gujrati', 'panjabi', 'odiya', 'hindi','nagri'])
# Normalization
* For best results use normalized text before parsing
* An example bangla unicode normalizer can be found [here](https://pypi.org/project/bnunicodenormalizer/)
# ABOUT
* Authors: [Bengali.AI](https://bengali.ai/)
* **Cite our work**
```bibtext
@inproceedings{ansary-etal-2024-unicode-normalization,
title = "{U}nicode Normalization and Grapheme Parsing of {I}ndic Languages",
author = "Ansary, Nazmuddoha and
Adib, Quazi Adibur Rahman and
Reasat, Tahsin and
Sushmit, Asif Shahriyar and
Humayun, Ahmed Imtiaz and
Mehnaz, Sazia and
Fatema, Kanij and
Rashid, Mohammad Mamun Or and
Sadeque, Farig",
editor = "Calzolari, Nicoletta and
Kan, Min-Yen and
Hoste, Veronique and
Lenci, Alessandro and
Sakti, Sakriani and
Xue, Nianwen",
booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
month = may,
year = "2024",
address = "Torino, Italia",
publisher = "ELRA and ICCL",
url = "https://aclanthology.org/2024.lrec-main.1479",
pages = "17019--17030",
abstract = "Writing systems of Indic languages have orthographic syllables, also known as complex graphemes, as unique horizontal units. A prominent feature of these languages is these complex grapheme units that comprise consonants/consonant conjuncts, vowel diacritics, and consonant diacritics, which, together make a unique Language. Unicode-based writing schemes of these languages often disregard this feature of these languages and encode words as linear sequences of Unicode characters using an intricate scheme of connector characters and font interpreters. Due to this way of using a few dozen Unicode glyphs to write thousands of different unique glyphs (complex graphemes), there are serious ambiguities that lead to malformed words. In this paper, we are proposing two libraries: i) a normalizer for normalizing inconsistencies caused by a Unicode-based encoding scheme for Indic languages and ii) a grapheme parser for Abugida text. It deconstructs words into visually distinct orthographic syllables or complex graphemes and their constituents. Our proposed normalizer is a more efficient and effective tool than the previously used IndicNLP normalizer. Moreover, our parser and normalizer are also suitable tools for general Abugida text processing as they performed well in our robust word-based and NLP experiments. We report the pipeline for the scripts of 7 languages in this work and develop the framework for the integration of more scripts.",
}
```
Change Log
===========
0.0.1 (12/02/2022)
-------------------
- First Release
0.0.2 (12/02/2022)
-------------------
- Basic Documentation
- Modifier removal
- space correction
- text mode parser
0.0.3 (14/02/2022)
-------------------
- Connector ending
- Exception case for component construction in bangla
- Added test
0.0.4 (14/02/2022)
-------------------
- pip test stable
- added malformed word detection
0.0.5 (19/02/2022)
-------------------
- encoding correction
- no space char handling
0.0.6 (15/04/2022)
-------------------
- removed malformed word detection [not useful]
- removed component calculation [not consistent]
0.0.7 (26/04/2022)
-------------------
- addition order correction
0.0.8 (21/10/2022)
-------------------
- allow middle Connector
0.0.9 (31/12/2022)
-------------------
- added sylheti nagri
0.0.10 (31/12/2022)
-------------------
- added sylheti nagri alternate hosonto
0.1.0 (26/05/2024)
-------------------
- added proper citation
Raw data
{
"_id": null,
"home_page": "https://github.com/mnansary/indicparser",
"name": "indicparser",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "grapheme, indic languages",
"author": "Bengali.AI",
"author_email": "research.bengaliai@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/62/25/a2da000406fdae162f1811e73236a56ed2f5e8f25a2d95a2892ecb4c6e2b/indicparser-0.1.0.tar.gz",
"platform": null,
"description": "# indicparser\nGrapheme Parser for indic languages \n\n# Installaton\n\n```python\npip install indicparser\n```\n# Useage\n* initializing the parser\n\n```python\nfrom indicparser import graphemeParser\ngp=graphemeParser(\"bangla\")\n```\n* extracting graphemes\n\n```python\ntext=\" \u09b6\u09be\u099f\u09bf\u0995\u09be\u09aa \u09ae\u09be\u09b0\"\ngraphemes=gp.process(text)\nprint(\"Graphemes:\",graphemes)\n```\n> Graphemes: [' ', ' ', '\u09b6\u09be', '\u099f\u09bf', '\u0995\u09be', '\u09aa', ' ', ' ', ' ', '\u09ae\u09be', '\u09b0']\n\n* extracting graphemes but merging spaces and clearing initial and ending space\n```python\ngraphemes=gp.process(text,merge_spaces=True)\nprint(\"Graphemes (space corrected):\",graphemes)\n```\n> Graphemes (space corrected): ['\u09b6\u09be', '\u099f\u09bf', '\u0995\u09be', '\u09aa', ' ', '\u09ae\u09be', '\u09b0']\n\n* treatment of numbers and puntucation and english is also available by default\n\n```python\ntext=\"\u098f\u099f\u09be\u0995\u09bf 2441139 ? \u09a8\u09be \u09ad\u09be\u0987 wrong number\"\ngraphemes=gp.process(text,merge_spaces=True)\nprint(\"Graphemes:\",graphemes)\n```\n> Graphemes: ['\u098f', '\u099f\u09be', '\u0995\u09bf', ' ', '2', '4', '4', '1', '1', '3', '9', ' ', '?', ' ', '\u09a8\u09be', ' ', '\u09ad\u09be', '\u0987', ' ', 'w', 'r', 'o', 'n', 'g', ' ', 'n', 'u', 'm', 'b', 'e', 'r']\n\n* available languages\n\n```python\nfrom indicparser import languages\nlanguages.keys()\n```\n> dict_keys(['bangla', 'malyalam', 'tamil', 'gujrati', 'panjabi', 'odiya', 'hindi','nagri'])\n\n\n# Normalization\n* For best results use normalized text before parsing\n* An example bangla unicode normalizer can be found [here](https://pypi.org/project/bnunicodenormalizer/)\n\n# ABOUT\n* Authors: [Bengali.AI](https://bengali.ai/)\n\n* **Cite our work**\n```bibtext\n@inproceedings{ansary-etal-2024-unicode-normalization,\n title = \"{U}nicode Normalization and Grapheme Parsing of {I}ndic Languages\",\n author = \"Ansary, Nazmuddoha and\n Adib, Quazi Adibur Rahman and\n Reasat, Tahsin and\n Sushmit, Asif Shahriyar and\n Humayun, Ahmed Imtiaz and\n Mehnaz, Sazia and\n Fatema, Kanij and\n Rashid, Mohammad Mamun Or and\n Sadeque, Farig\",\n editor = \"Calzolari, Nicoletta and\n Kan, Min-Yen and\n Hoste, Veronique and\n Lenci, Alessandro and\n Sakti, Sakriani and\n Xue, Nianwen\",\n booktitle = \"Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)\",\n month = may,\n year = \"2024\",\n address = \"Torino, Italia\",\n publisher = \"ELRA and ICCL\",\n url = \"https://aclanthology.org/2024.lrec-main.1479\",\n pages = \"17019--17030\",\n abstract = \"Writing systems of Indic languages have orthographic syllables, also known as complex graphemes, as unique horizontal units. A prominent feature of these languages is these complex grapheme units that comprise consonants/consonant conjuncts, vowel diacritics, and consonant diacritics, which, together make a unique Language. Unicode-based writing schemes of these languages often disregard this feature of these languages and encode words as linear sequences of Unicode characters using an intricate scheme of connector characters and font interpreters. Due to this way of using a few dozen Unicode glyphs to write thousands of different unique glyphs (complex graphemes), there are serious ambiguities that lead to malformed words. In this paper, we are proposing two libraries: i) a normalizer for normalizing inconsistencies caused by a Unicode-based encoding scheme for Indic languages and ii) a grapheme parser for Abugida text. It deconstructs words into visually distinct orthographic syllables or complex graphemes and their constituents. Our proposed normalizer is a more efficient and effective tool than the previously used IndicNLP normalizer. Moreover, our parser and normalizer are also suitable tools for general Abugida text processing as they performed well in our robust word-based and NLP experiments. We report the pipeline for the scripts of 7 languages in this work and develop the framework for the integration of more scripts.\",\n}\n```\n\n\nChange Log\n===========\n\n0.0.1 (12/02/2022)\n-------------------\n- First Release\n\n0.0.2 (12/02/2022)\n-------------------\n- Basic Documentation \n- Modifier removal\n- space correction\n- text mode parser\n\n0.0.3 (14/02/2022)\n-------------------\n- Connector ending \n- Exception case for component construction in bangla\n- Added test\n\n0.0.4 (14/02/2022)\n-------------------\n- pip test stable\n- added malformed word detection\n\n0.0.5 (19/02/2022)\n-------------------\n- encoding correction\n- no space char handling\n\n0.0.6 (15/04/2022)\n-------------------\n- removed malformed word detection [not useful]\n- removed component calculation [not consistent]\n\n0.0.7 (26/04/2022)\n-------------------\n- addition order correction\n\n0.0.8 (21/10/2022)\n-------------------\n- allow middle Connector\n\n0.0.9 (31/12/2022)\n-------------------\n- added sylheti nagri\n\n0.0.10 (31/12/2022)\n-------------------\n- added sylheti nagri alternate hosonto\n\n0.1.0 (26/05/2024)\n-------------------\n- added proper citation\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Grapheme Parser for indic languages",
"version": "0.1.0",
"project_urls": {
"Homepage": "https://github.com/mnansary/indicparser"
},
"split_keywords": [
"grapheme",
" indic languages"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "940c8ea9d88eb89ff4375600ac0d1a272d644fa68f9ae58c440d92580c9b5194",
"md5": "fcf39509b6e6c07a1cd4d56a24cf97fb",
"sha256": "32ba30a2594f8a52cf96f38c8375084e4c2fb961f76d6379e39ae7150c486f4c"
},
"downloads": -1,
"filename": "indicparser-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "fcf39509b6e6c07a1cd4d56a24cf97fb",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 7354,
"upload_time": "2024-05-26T17:01:17",
"upload_time_iso_8601": "2024-05-26T17:01:17.301486Z",
"url": "https://files.pythonhosted.org/packages/94/0c/8ea9d88eb89ff4375600ac0d1a272d644fa68f9ae58c440d92580c9b5194/indicparser-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6225a2da000406fdae162f1811e73236a56ed2f5e8f25a2d95a2892ecb4c6e2b",
"md5": "fbfba10b560b226676f651fa335752c0",
"sha256": "0f66e4bd38fcb511e0fbc2d0b78237527faea9f358925a56dad4d4b2bff08d0c"
},
"downloads": -1,
"filename": "indicparser-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "fbfba10b560b226676f651fa335752c0",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 432112,
"upload_time": "2024-05-26T17:01:21",
"upload_time_iso_8601": "2024-05-26T17:01:21.117868Z",
"url": "https://files.pythonhosted.org/packages/62/25/a2da000406fdae162f1811e73236a56ed2f5e8f25a2d95a2892ecb4c6e2b/indicparser-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-26 17:01:21",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mnansary",
"github_project": "indicparser",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "indicparser"
}
JSON
