# whisper_normalizer
<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->
## Installation of package
``` sh
pip install whisper_normalizer
```
or from github repository
``` sh
pip install git+https://github.com/kurianbenoy/whisper_normalizer.git
```
## Why should we normalize/standardize text?
- In ASR systems it’s important to normalize the text to reduce
unintentional penalties in metrics like WER, CER etc.
- Text normalization/standardization is process of converting texts in
different styles into a standardized form, which is a best-effort
attempt to penalize only when a word error is caused by actually
mistranscribing a word, and not by formatting or punctuation
differences.(from [Whisper
paper](https://cdn.openai.com/papers/whisper.pdf))
## Why use this python package?
This package is a python implementation of the text
standardisation/normalization approach which is being used in OpenAI
whisper text normalizer. If you want to use just text normalization
alone, it’s better to use this instead reimplementing the same thing.
OpenAI approach of text normalization is very helpful and is being used
as normalization step when evaluating competitive models like
[AssemblyAI Conformer-1
model](https://www.assemblyai.com/blog/conformer-1/).
## Models evaluated using Whisper normalization
- Massively Multilingual Speech (MMS) models by Meta
- Conformer 1 by AssemblyAI
- Conformer 2 by AssemblyAI
## How to use
OpenAI open source approach of text normalization/standardization is
mentioned in detail Appendix Section C pp.21 the paper [Robust Speech
Recognition via Large-Scale Weak
Supervision](https://cdn.openai.com/papers/whisper.pdf).
Whisper Normalizer by default comes with two classes
[`BasicTextNormalizer`](https://kurianbenoy.github.io/whisper_normalizer/basic.html#basictextnormalizer)
and
[`EnglishTextNormalizer`](https://kurianbenoy.github.io/whisper_normalizer/english.html#englishtextnormalizer)
You can use the same thing in this package as follows:
``` python
from whisper_normalizer.basic import BasicTextNormalizer
normalizer = BasicTextNormalizer()
normalizer("I'm a little teapot, short and stout. Tip me over and pour me out!")
```
'i m a little teapot short and stout tip me over and pour me out '
``` python
from whisper_normalizer.english import EnglishTextNormalizer
english_normalizer = EnglishTextNormalizer()
english_normalizer("I'm a little teapot, short and stout. Tip me over and pour me out!")
```
'i am a little teapot short and stout tip me over and pour me out'
### This model extends Whisper_normalizer to support Indic languages as well.
The logic for normalization in Indic languages is derived from
[indic-nlp-library](https://github.com/anoopkunchukuttan/indic_nlp_library).
The logic for Malayalam normalization is expanded beyond the Indic NLP
library by
[`MalayalamNormalizer`](https://kurianbenoy.github.io/whisper_normalizer/1b.indic_normalizer.html#malayalamnormalizer).
``` python
from whisper_normalizer.indic_normalizer import MalayalamNormalizer
normalizer = MalayalamNormalizer()
normalizer("എന്റെ കമ്പ്യൂട്ടറിനു് എന്റെ ഭാഷ.")
```
'എന്റെ കമ്പ്യൂട്ടറിനു് എന്റെ ഭാഷ.'
Raw data
{
"_id": null,
"home_page": "https://github.com/kurianbenoy/whisper_normalizer",
"name": "whisper-normalizer",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "nbdev jupyter notebook openai whisper normalizer",
"author": "kurianbenoy",
"author_email": "kurian.bkk@gmail.com",
"download_url": "",
"platform": null,
"description": "# whisper_normalizer\n\n<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->\n\n## Installation of package\n\n``` sh\npip install whisper_normalizer\n```\n\nor from github repository\n\n``` sh\npip install git+https://github.com/kurianbenoy/whisper_normalizer.git\n```\n\n## Why should we normalize/standardize text?\n\n- In ASR systems it\u2019s important to normalize the text to reduce\n unintentional penalties in metrics like WER, CER etc.\n- Text normalization/standardization is process of converting texts in\n different styles into a standardized form, which is a best-effort\n attempt to penalize only when a word error is caused by actually\n mistranscribing a word, and not by formatting or punctuation\n differences.(from [Whisper\n paper](https://cdn.openai.com/papers/whisper.pdf))\n\n## Why use this python package?\n\nThis package is a python implementation of the text\nstandardisation/normalization approach which is being used in OpenAI\nwhisper text normalizer. If you want to use just text normalization\nalone, it\u2019s better to use this instead reimplementing the same thing.\nOpenAI approach of text normalization is very helpful and is being used\nas normalization step when evaluating competitive models like\n[AssemblyAI Conformer-1\nmodel](https://www.assemblyai.com/blog/conformer-1/).\n\n## Models evaluated using Whisper normalization\n\n- Massively Multilingual Speech (MMS) models by Meta\n- Conformer 1 by AssemblyAI\n- Conformer 2 by AssemblyAI\n\n## How to use\n\nOpenAI open source approach of text normalization/standardization is\nmentioned in detail Appendix Section C pp.21 the paper [Robust Speech\nRecognition via Large-Scale Weak\nSupervision](https://cdn.openai.com/papers/whisper.pdf).\n\nWhisper Normalizer by default comes with two classes\n[`BasicTextNormalizer`](https://kurianbenoy.github.io/whisper_normalizer/basic.html#basictextnormalizer)\nand\n[`EnglishTextNormalizer`](https://kurianbenoy.github.io/whisper_normalizer/english.html#englishtextnormalizer)\n\nYou can use the same thing in this package as follows:\n\n``` python\nfrom whisper_normalizer.basic import BasicTextNormalizer\n\nnormalizer = BasicTextNormalizer()\nnormalizer(\"I'm a little teapot, short and stout. Tip me over and pour me out!\")\n```\n\n 'i m a little teapot short and stout tip me over and pour me out '\n\n``` python\nfrom whisper_normalizer.english import EnglishTextNormalizer\n\nenglish_normalizer = EnglishTextNormalizer()\nenglish_normalizer(\"I'm a little teapot, short and stout. Tip me over and pour me out!\")\n```\n\n 'i am a little teapot short and stout tip me over and pour me out'\n\n### This model extends Whisper_normalizer to support Indic languages as well.\n\nThe logic for normalization in Indic languages is derived from\n[indic-nlp-library](https://github.com/anoopkunchukuttan/indic_nlp_library).\nThe logic for Malayalam normalization is expanded beyond the Indic NLP\nlibrary by\n[`MalayalamNormalizer`](https://kurianbenoy.github.io/whisper_normalizer/1b.indic_normalizer.html#malayalamnormalizer).\n\n``` python\nfrom whisper_normalizer.indic_normalizer import MalayalamNormalizer\n\nnormalizer = MalayalamNormalizer()\nnormalizer(\"\u0d0e\u0d28\u0d4d\u0d31\u0d46 \u0d15\u0d2e\u0d4d\u0d2a\u0d4d\u0d2f\u0d42\u0d1f\u0d4d\u0d1f\u0d31\u0d3f\u0d28\u0d41\u0d4d \u0d0e\u0d28\u0d4d\u0d31\u0d46 \u0d2d\u0d3e\u0d37.\")\n```\n\n '\u0d0e\u0d28\u0d4d\u0d31\u0d46 \u0d15\u0d2e\u0d4d\u0d2a\u0d4d\u0d2f\u0d42\u0d1f\u0d4d\u0d1f\u0d31\u0d3f\u0d28\u0d41\u0d4d \u0d0e\u0d28\u0d4d\u0d31\u0d46 \u0d2d\u0d3e\u0d37.'\n",
"bugtrack_url": null,
"license": "MIT License",
"summary": "A python package for whisper normalizer",
"version": "0.0.8",
"project_urls": {
"Homepage": "https://github.com/kurianbenoy/whisper_normalizer"
},
"split_keywords": [
"nbdev",
"jupyter",
"notebook",
"openai",
"whisper",
"normalizer"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "35f73d04dfe7d5524542f99ace028a6f05abe4f8610ce045cbf9662625664694",
"md5": "71c1780cae0babb6572296f26de035cb",
"sha256": "f61295b7ff7c637529f39f51537ef89a9d067f837f82eac218f5bf3818fdd03a"
},
"downloads": -1,
"filename": "whisper_normalizer-0.0.8-py3-none-any.whl",
"has_sig": false,
"md5_digest": "71c1780cae0babb6572296f26de035cb",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 22761,
"upload_time": "2024-02-10T16:41:28",
"upload_time_iso_8601": "2024-02-10T16:41:28.609459Z",
"url": "https://files.pythonhosted.org/packages/35/f7/3d04dfe7d5524542f99ace028a6f05abe4f8610ce045cbf9662625664694/whisper_normalizer-0.0.8-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-10 16:41:28",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "kurianbenoy",
"github_project": "whisper_normalizer",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "whisper-normalizer"
}
JSON
