# rs-bytepiece
## Install
```bash
pip install rs_bytepiece
```
## Usage
```python
from rs_bytepiece import Tokenizer
tokenizer = Tokenizer()
# a custom model
tokenizer = Tokenizer("/path/to/model")
ids = tokenizer.encode("今天天气不错")
text = tokenizer.decode(ids)
```
## Performance
The performance is a bit faster than the original implementation. I've tested (on my M2 16G) the《鲁迅全集》which has 625890 chars. The time unit is millisecond.
| length | jieba | aho_py | aho_cy | aho_rs |
| ------ | -------- | ------- | ------ | ------ |
| 100 | 17062.12 | 1404.37 | 564.31 | 112.94 |
| 1000 | 17104.38 | 1424.6 | 573.32 | 113.18 |
| 10000 | 17432.58 | 1429.0 | 574.93 | 110.03 |
| 100000 | 17228.17 | 1401.01 | 574.5 | 110.44 |
| 625890 | 17305.95 | 1419.79 | 567.78 | 108.54 |
Raw data
{
"_id": null,
"home_page": null,
"name": "rs-bytepiece",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": null,
"keywords": "NLP,tokenizer,bytepiece,Deep Learning",
"author": "Yam(\u957f\u7434) <haoshaochun@gmail.com>",
"author_email": "Yam(\u957f\u7434) <haoshaochun@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/8a/22/b6bbac87677550e256e049e97b2cc4a75aa33bd12bb81ac7c4709d407178/rs_bytepiece-0.2.2.tar.gz",
"platform": null,
"description": "# rs-bytepiece\n\n## Install\n\n```bash\npip install rs_bytepiece\n```\n\n## Usage\n\n```python\nfrom rs_bytepiece import Tokenizer\n\ntokenizer = Tokenizer()\n# a custom model\ntokenizer = Tokenizer(\"/path/to/model\")\nids = tokenizer.encode(\"\u4eca\u5929\u5929\u6c14\u4e0d\u9519\")\ntext = tokenizer.decode(ids)\n```\n\n## Performance\n\nThe performance is a bit faster than the original implementation. I've tested (on my M2 16G) the\u300a\u9c81\u8fc5\u5168\u96c6\u300bwhich has 625890 chars. The time unit is millisecond.\n\n| length | jieba | aho_py | aho_cy | aho_rs |\n| ------ | -------- | ------- | ------ | ------ |\n| 100 | 17062.12 | 1404.37 | 564.31 | 112.94 |\n| 1000 | 17104.38 | 1424.6 | 573.32 | 113.18 |\n| 10000 | 17432.58 | 1429.0 | 574.93 | 110.03 |\n| 100000 | 17228.17 | 1401.01 | 574.5 | 110.44 |\n| 625890 | 17305.95 | 1419.79 | 567.78 | 108.54 |\n\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "bytepiece-rs Python binding",
"version": "0.2.2",
"project_urls": {
"documentation": "https://github.com/hscspring/bytepiece-rs",
"homepage": "https://github.com/hscspring/bytepiece-rs",
"repository": "https://github.com/hscspring/bytepiece-rs"
},
"split_keywords": [
"nlp",
"tokenizer",
"bytepiece",
"deep learning"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "4863809ac38242cf82098144777c8f82a94290d15aa04f9c45dbb57da7a78be8",
"md5": "485620da2afd64f4758720e3ab7704aa",
"sha256": "14338a8c8573df2ac4dc83567a58722d18984f678ce3426237d58e689288df1d"
},
"downloads": -1,
"filename": "rs_bytepiece-0.2.2-cp37-abi3-macosx_10_7_x86_64.whl",
"has_sig": false,
"md5_digest": "485620da2afd64f4758720e3ab7704aa",
"packagetype": "bdist_wheel",
"python_version": "cp37",
"requires_python": ">=3.7",
"size": 2295816,
"upload_time": "2023-11-12T08:52:19",
"upload_time_iso_8601": "2023-11-12T08:52:19.642411Z",
"url": "https://files.pythonhosted.org/packages/48/63/809ac38242cf82098144777c8f82a94290d15aa04f9c45dbb57da7a78be8/rs_bytepiece-0.2.2-cp37-abi3-macosx_10_7_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "14610a22cb90c845829383640d2a9ef17ab0286e9cef9189b7346194b5df2b72",
"md5": "52a51aebe3216eadb80bafb8150db11a",
"sha256": "2405a38e0a03985fabeb025eae65fcafcf5aae7c5f9b065c10cc17b72ae5e723"
},
"downloads": -1,
"filename": "rs_bytepiece-0.2.2-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "52a51aebe3216eadb80bafb8150db11a",
"packagetype": "bdist_wheel",
"python_version": "cp37",
"requires_python": ">=3.7",
"size": 3463328,
"upload_time": "2023-11-12T08:52:26",
"upload_time_iso_8601": "2023-11-12T08:52:26.347012Z",
"url": "https://files.pythonhosted.org/packages/14/61/0a22cb90c845829383640d2a9ef17ab0286e9cef9189b7346194b5df2b72/rs_bytepiece-0.2.2-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "6230dea969abe55a7cf936c69b32b5ddda7eb09687300be9fa8923535da0d9f6",
"md5": "a20df1d801d72386db90a6e23dfc9efe",
"sha256": "2e6a9f8b78bc24d4856b00240c82392ebe6008b110ba482e22c00f48f81b4e6c"
},
"downloads": -1,
"filename": "rs_bytepiece-0.2.2-cp37-abi3-win_amd64.whl",
"has_sig": false,
"md5_digest": "a20df1d801d72386db90a6e23dfc9efe",
"packagetype": "bdist_wheel",
"python_version": "cp37",
"requires_python": ">=3.7",
"size": 3819501,
"upload_time": "2023-11-12T08:52:32",
"upload_time_iso_8601": "2023-11-12T08:52:32.319579Z",
"url": "https://files.pythonhosted.org/packages/62/30/dea969abe55a7cf936c69b32b5ddda7eb09687300be9fa8923535da0d9f6/rs_bytepiece-0.2.2-cp37-abi3-win_amd64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "8a22b6bbac87677550e256e049e97b2cc4a75aa33bd12bb81ac7c4709d407178",
"md5": "5b7c4bd2fb2e597f4088d5b1a33bfb9f",
"sha256": "f86206004808f118d7581fe314387c6d0c45566cd34fda346c439c3392e912cd"
},
"downloads": -1,
"filename": "rs_bytepiece-0.2.2.tar.gz",
"has_sig": false,
"md5_digest": "5b7c4bd2fb2e597f4088d5b1a33bfb9f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 1241456,
"upload_time": "2023-11-12T08:52:37",
"upload_time_iso_8601": "2023-11-12T08:52:37.611552Z",
"url": "https://files.pythonhosted.org/packages/8a/22/b6bbac87677550e256e049e97b2cc4a75aa33bd12bb81ac7c4709d407178/rs_bytepiece-0.2.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-12 08:52:37",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "hscspring",
"github_project": "bytepiece-rs",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "rs-bytepiece"
}
JSON
