[English](README.md) | [Español](./docs/README.es.md) | [Français](./docs/README.fr.md) | [Deutsch](./docs/README.de.md) | [中文](./docs/README.zh.md) | [Türkçe](./docs/README.tr.md) | [日本語](./docs/README.ja.md) | [한국어](./docs/README.ko.md)
## visuals_tokenization
A library to view how different tokenization performss
Developed by Marcel Tino (c) 2024
## Examples of How To Use the library
You can use this to alter according to your requirements
```
##syntax
from visual_tokenization import view_tokenize
view_tokenize(text)
```
```python
text="Strong economic growth in the first quarter of FY23 helped India overcome the UK to become the fifth-largest economy after it recovered from the COVID-19 pandemic shock. Nominal GDP or GDP at Current Prices in the year 2023-24 is estimated at Rs. 295.36 lakh crores (US$ 3.54 trillion), against the First Revised Estimates (FRE) of GDP for the year 2022-23 of Rs. 269.50 lakh crores (US$ 3.23 trillion). The growth in nominal GDP during 2023-24 is estimated at 9.6% as compared to 14.2% in 2022-23. Strong domestic demand for consumption and investment, along with Government’s continued emphasis on capital expenditure are seen as among the key driver of the GDP in the second half of FY24. During the period April-June 2025, India’s exports stood at US$ 109.11 billion, with Engineering Goods (25.35%), Petroleum Products (18.33%) and electronic goods (7.73%) being the top three exported commodity. Rising employment and increasing private consumption, supported by rising consumer sentiment, will support GDP growth in the coming months."
from visual_tokenization import view_tokenize
view_tokenize(text)
```
+ Share visual_tokenizations on these social media platforms if you like it!
[](https://reddit.com/submit?url=https://github.com/Kanaries/pygwalker&title=Say%20Hello%20to%20pygwalker%3A%20Combining%20Jupyter%20Notebook%20with%20a%20Tableau-like%20UI)
[](https://news.ycombinator.com/submitlink?u=https://github.com/Kanaries/pygwalker)
[](https://twitter.com/share?url=https://github.com/Kanaries/pygwalker&text=Say%20Hello%20to%20pygwalker%3A%20Combining%20Jupyter%20Notebook%20with%20a%20Tableau-alternative%20UI)
[](https://www.facebook.com/sharer/sharer.php?u=https://github.com/Kanaries/pygwalker)
[](https://www.linkedin.com/shareArticle?url=https://github.com/Kanaries/pygwalker&&title=Say%20Hello%20to%20pygwalker%3A%20Combining%20Jupyter%20Notebook%20with%20a%20Tableau-alternative%20UI)
Raw data
{
"_id": null,
"home_page": null,
"name": "visuals-tokenization",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "tokens, visual_tokenizations",
"author": "Marcel Tino",
"author_email": "<marceltino92@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/49/f4/b0d6e51be20013f5d19da96ec2c9e2709ebf79e6f5a150598cc73b76c014/visuals_tokenization-0.1.2.tar.gz",
"platform": null,
"description": "\r\n[English](README.md) | [Espa\u00f1ol](./docs/README.es.md) | [Fran\u00e7ais](./docs/README.fr.md) | [Deutsch](./docs/README.de.md) | [\u4e2d\u6587](./docs/README.zh.md) | [T\u00fcrk\u00e7e](./docs/README.tr.md) | [\u65e5\u672c\u8a9e](./docs/README.ja.md) | [\ud55c\uad6d\uc5b4](./docs/README.ko.md)\r\n\r\n## visuals_tokenization\r\n\r\nA library to view how different tokenization performss\r\n\r\nDeveloped by Marcel Tino (c) 2024\r\n\r\n## Examples of How To Use the library \r\n\r\nYou can use this to alter according to your requirements\r\n\r\n\r\n```\r\n##syntax\r\nfrom visual_tokenization import view_tokenize\r\n\r\nview_tokenize(text)\r\n```\r\n\r\n\r\n```python\r\n\r\ntext=\"Strong economic growth in the first quarter of FY23 helped India overcome the UK to become the fifth-largest economy after it recovered from the COVID-19 pandemic shock. Nominal GDP or GDP at Current Prices in the year 2023-24 is estimated at Rs. 295.36 lakh crores (US$ 3.54 trillion), against the First Revised Estimates (FRE) of GDP for the year 2022-23 of Rs. 269.50 lakh crores (US$ 3.23 trillion). The growth in nominal GDP during 2023-24 is estimated at 9.6% as compared to 14.2% in 2022-23. Strong domestic demand for consumption and investment, along with Government\u2019s continued emphasis on capital expenditure are seen as among the key driver of the GDP in the second half of FY24. During the period April-June 2025, India\u2019s exports stood at US$ 109.11 billion, with Engineering Goods (25.35%), Petroleum Products (18.33%) and electronic goods (7.73%) being the top three exported commodity. Rising employment and increasing private consumption, supported by rising consumer sentiment, will support GDP growth in the coming months.\"\r\n\r\nfrom visual_tokenization import view_tokenize\r\n\r\nview_tokenize(text)\r\n\r\n```\r\n\r\n\r\n+ Share visual_tokenizations on these social media platforms if you like it!\r\n[](https://reddit.com/submit?url=https://github.com/Kanaries/pygwalker&title=Say%20Hello%20to%20pygwalker%3A%20Combining%20Jupyter%20Notebook%20with%20a%20Tableau-like%20UI)\r\n[](https://news.ycombinator.com/submitlink?u=https://github.com/Kanaries/pygwalker)\r\n[](https://twitter.com/share?url=https://github.com/Kanaries/pygwalker&text=Say%20Hello%20to%20pygwalker%3A%20Combining%20Jupyter%20Notebook%20with%20a%20Tableau-alternative%20UI)\r\n[](https://www.facebook.com/sharer/sharer.php?u=https://github.com/Kanaries/pygwalker)\r\n[](https://www.linkedin.com/shareArticle?url=https://github.com/Kanaries/pygwalker&&title=Say%20Hello%20to%20pygwalker%3A%20Combining%20Jupyter%20Notebook%20with%20a%20Tableau-alternative%20UI)\r\n",
"bugtrack_url": null,
"license": null,
"summary": "Library to visualize different tokenization methords",
"version": "0.1.2",
"project_urls": null,
"split_keywords": [
"tokens",
" visual_tokenizations"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8ea6c1e1ae0d8c40e45c0dfb361d5716bf0f56655ba9eaf144148c92cec6d142",
"md5": "0dd2c68538263eefc6eb02e870a7ca76",
"sha256": "4560641aef1dd80d0beae0120381084f88bb236954d9e55cf5d862d6497d12c8"
},
"downloads": -1,
"filename": "visuals_tokenization-0.1.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0dd2c68538263eefc6eb02e870a7ca76",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 3801,
"upload_time": "2024-09-21T05:31:49",
"upload_time_iso_8601": "2024-09-21T05:31:49.661096Z",
"url": "https://files.pythonhosted.org/packages/8e/a6/c1e1ae0d8c40e45c0dfb361d5716bf0f56655ba9eaf144148c92cec6d142/visuals_tokenization-0.1.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "49f4b0d6e51be20013f5d19da96ec2c9e2709ebf79e6f5a150598cc73b76c014",
"md5": "b8534b91817d0ed7b35e7d8b8782e930",
"sha256": "eb97db535b751f9e29ea5aa4f48c33f242c38ca86bdf112456c3cb73394ad2d4"
},
"downloads": -1,
"filename": "visuals_tokenization-0.1.2.tar.gz",
"has_sig": false,
"md5_digest": "b8534b91817d0ed7b35e7d8b8782e930",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 3644,
"upload_time": "2024-09-21T05:31:51",
"upload_time_iso_8601": "2024-09-21T05:31:51.134356Z",
"url": "https://files.pythonhosted.org/packages/49/f4/b0d6e51be20013f5d19da96ec2c9e2709ebf79e6f5a150598cc73b76c014/visuals_tokenization-0.1.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-21 05:31:51",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "visuals-tokenization"
}
JSON