# 🥢 Curated Tokenizers
This Python library provides word-/sentencepiece tokenizers. The following
types of tokenizers are currenty supported:
| Tokenizer | Binding | Example model |
| --------- | ------------- | ------------- |
| BPE | sentencepiece | |
| Byte BPE | Native | RoBERTa/GPT-2 |
| Unigram | sentencepiece | XLM-RoBERTa |
| Wordpiece | Native | BERT |
## ⚠️ Warning: experimental package
This package is experimental and it is likely that the APIs will change in
incompatible ways.
## ⏳ Install
Curated tokenizers is availble through PyPI:
```bash
pip install curated_tokenizers
```
## 🚀 Quickstart
The best way to get started with curated tokenizers is through the
[`curated-transformers`](https://github.com/explosion/curated-transformers)
library. `curated-transformers` also provides functionality to load tokenization
models from Huggingface Hub.
Raw data
{
"_id": null,
"home_page": "https://github.com/explosion/curated-tokenizers",
"name": "curated-tokenizers",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": null,
"author": "Explosion",
"author_email": "contact@explosion.ai",
"download_url": "https://files.pythonhosted.org/packages/73/22/23b6fe4e15a788405a64304067ca46ff104283f9c3ff9608b4b5be7fac48/curated_tokenizers-2.0.0.tar.gz",
"platform": null,
"description": "# \ud83e\udd62 Curated Tokenizers\n\nThis Python library provides word-/sentencepiece tokenizers. The following\ntypes of tokenizers are currenty supported:\n\n| Tokenizer | Binding | Example model |\n| --------- | ------------- | ------------- |\n| BPE | sentencepiece | |\n| Byte BPE | Native | RoBERTa/GPT-2 |\n| Unigram | sentencepiece | XLM-RoBERTa |\n| Wordpiece | Native | BERT |\n\n## \u26a0\ufe0f Warning: experimental package\n\nThis package is experimental and it is likely that the APIs will change in\nincompatible ways.\n\n## \u23f3 Install\n\nCurated tokenizers is availble through PyPI:\n\n```bash\npip install curated_tokenizers\n```\n\n## \ud83d\ude80 Quickstart\n\nThe best way to get started with curated tokenizers is through the\n[`curated-transformers`](https://github.com/explosion/curated-transformers)\nlibrary. `curated-transformers` also provides functionality to load tokenization\nmodels from Huggingface Hub.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Lightweight piece tokenization library",
"version": "2.0.0",
"project_urls": {
"Homepage": "https://github.com/explosion/curated-tokenizers"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "9d412e2523b258b98755a911171434878dfc61cac54b13abc6943e417b5542ba",
"md5": "4793d47b137d73c7f31935a4fa5d07de",
"sha256": "55a565b40ddd04a127e7367ebdcb560a61699350e41586c2642481ec1d6b1964"
},
"downloads": -1,
"filename": "curated_tokenizers-2.0.0-cp310-cp310-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "4793d47b137d73c7f31935a4fa5d07de",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.9",
"size": 773936,
"upload_time": "2024-04-15T17:18:50",
"upload_time_iso_8601": "2024-04-15T17:18:50.982825Z",
"url": "https://files.pythonhosted.org/packages/9d/41/2e2523b258b98755a911171434878dfc61cac54b13abc6943e417b5542ba/curated_tokenizers-2.0.0-cp310-cp310-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "026b2b8774b6a86fb7be3a52a96d23ba1cb751a82adad0a2c0e71a2acd3ae7ff",
"md5": "af5651532cc4d3a0a73e57752b1f39e7",
"sha256": "73a77d9ef4619f51fc043b76a8ff6845e7fc908d09fbdbd68043ca9dcfe8a597"
},
"downloads": -1,
"filename": "curated_tokenizers-2.0.0-cp310-cp310-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "af5651532cc4d3a0a73e57752b1f39e7",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.9",
"size": 741825,
"upload_time": "2024-04-15T17:18:53",
"upload_time_iso_8601": "2024-04-15T17:18:53.425324Z",
"url": "https://files.pythonhosted.org/packages/02/6b/2b8774b6a86fb7be3a52a96d23ba1cb751a82adad0a2c0e71a2acd3ae7ff/curated_tokenizers-2.0.0-cp310-cp310-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "274b7ac743f1cdf94bfa99bff9d4d2873bfa997c881ba00f9e77ec0405a083b0",
"md5": "5eb8cc64dec047e5b0415bfa9caf0b8c",
"sha256": "a04525e89b00e163a03caa942b2a2ac8c7b97258057f51f0bb16f8f3c652bae8"
},
"downloads": -1,
"filename": "curated_tokenizers-2.0.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
"has_sig": false,
"md5_digest": "5eb8cc64dec047e5b0415bfa9caf0b8c",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.9",
"size": 745685,
"upload_time": "2024-04-15T17:18:54",
"upload_time_iso_8601": "2024-04-15T17:18:54.812835Z",
"url": "https://files.pythonhosted.org/packages/27/4b/7ac743f1cdf94bfa99bff9d4d2873bfa997c881ba00f9e77ec0405a083b0/curated_tokenizers-2.0.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "69fff66937ceb5bae34da4f119f260e35066bc683989075252127187599691f6",
"md5": "e5e43b31ca4b327249edb6a965958f46",
"sha256": "c15dbf7bdf1fe0f4d2d94c8a10c7cf36c420e195d32022b1b26062fd4c61c28e"
},
"downloads": -1,
"filename": "curated_tokenizers-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "e5e43b31ca4b327249edb6a965958f46",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.9",
"size": 772765,
"upload_time": "2024-04-15T17:18:56",
"upload_time_iso_8601": "2024-04-15T17:18:56.921424Z",
"url": "https://files.pythonhosted.org/packages/69/ff/f66937ceb5bae34da4f119f260e35066bc683989075252127187599691f6/curated_tokenizers-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "385210d6f8f3ec94ee8d27dc9cf028c792de12a4466beb1eb8c1cb750b32ef0f",
"md5": "a84aaf6713efd35d5bc8d930d5cebf5f",
"sha256": "d21af1ec57faac43f3c0214878a6805abb6a56123e75ab5f011babf4cc9cfd9c"
},
"downloads": -1,
"filename": "curated_tokenizers-2.0.0-cp310-cp310-win_amd64.whl",
"has_sig": false,
"md5_digest": "a84aaf6713efd35d5bc8d930d5cebf5f",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.9",
"size": 760716,
"upload_time": "2024-04-15T17:18:59",
"upload_time_iso_8601": "2024-04-15T17:18:59.112969Z",
"url": "https://files.pythonhosted.org/packages/38/52/10d6f8f3ec94ee8d27dc9cf028c792de12a4466beb1eb8c1cb750b32ef0f/curated_tokenizers-2.0.0-cp310-cp310-win_amd64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c480360e34d47922eb70226eeb02b946b48194c1bf41721427880be5b520aa74",
"md5": "4e71a6b5680f08d196b370b9f3a24f3b",
"sha256": "fef8dcc273845ca24e3f3eb8ac53a4b4a45cf1a4e64d1a89370bdc95a02e8fb0"
},
"downloads": -1,
"filename": "curated_tokenizers-2.0.0-cp311-cp311-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "4e71a6b5680f08d196b370b9f3a24f3b",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.9",
"size": 774599,
"upload_time": "2024-04-15T17:19:01",
"upload_time_iso_8601": "2024-04-15T17:19:01.195544Z",
"url": "https://files.pythonhosted.org/packages/c4/80/360e34d47922eb70226eeb02b946b48194c1bf41721427880be5b520aa74/curated_tokenizers-2.0.0-cp311-cp311-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b37358be1d08b7738ba76db1f1cd499d9bb8e72c06a89e7d2d9dac2b80605b4e",
"md5": "4b45d5244c2cfccf25dab048f3ab2586",
"sha256": "dbad2c2f40f81e9d720f8c272843b786f906712582f39d338a42a361f6b4856e"
},
"downloads": -1,
"filename": "curated_tokenizers-2.0.0-cp311-cp311-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "4b45d5244c2cfccf25dab048f3ab2586",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.9",
"size": 741957,
"upload_time": "2024-04-15T17:19:03",
"upload_time_iso_8601": "2024-04-15T17:19:03.465285Z",
"url": "https://files.pythonhosted.org/packages/b3/73/58be1d08b7738ba76db1f1cd499d9bb8e72c06a89e7d2d9dac2b80605b4e/curated_tokenizers-2.0.0-cp311-cp311-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "7cca0e606ecb19e6938416815ce5a9e0090fe508cb0a1c6c1389686fc2272b19",
"md5": "517a72be3fdbb15899b33f8235c06770",
"sha256": "563fc87e930aa466295cc0dc44697e369559b896db94453702693133c5abbbb8"
},
"downloads": -1,
"filename": "curated_tokenizers-2.0.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
"has_sig": false,
"md5_digest": "517a72be3fdbb15899b33f8235c06770",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.9",
"size": 749625,
"upload_time": "2024-04-15T17:19:06",
"upload_time_iso_8601": "2024-04-15T17:19:06.047188Z",
"url": "https://files.pythonhosted.org/packages/7c/ca/0e606ecb19e6938416815ce5a9e0090fe508cb0a1c6c1389686fc2272b19/curated_tokenizers-2.0.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3b5eecc43ae53535add247556b4768c78ab1895b0f2712901744c851b3ad3dca",
"md5": "e29561a8cfff9433dc183b4f0facb199",
"sha256": "51d0d86094eb863995c6eb5d0990a064e840c9a7e273358fa4596f7958593d93"
},
"downloads": -1,
"filename": "curated_tokenizers-2.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "e29561a8cfff9433dc183b4f0facb199",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.9",
"size": 776886,
"upload_time": "2024-04-15T17:19:07",
"upload_time_iso_8601": "2024-04-15T17:19:07.986217Z",
"url": "https://files.pythonhosted.org/packages/3b/5e/ecc43ae53535add247556b4768c78ab1895b0f2712901744c851b3ad3dca/curated_tokenizers-2.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "2b6141dc572a610e84b0f87a335f146acd21da6eed646f93bd9c8a27cc99fbfe",
"md5": "62997a2d6da014ee01236faeeefc63d8",
"sha256": "6d56eb806a791b5818cbccdf22fffe39c07eff2f2dd1a32b94f11792c646d343"
},
"downloads": -1,
"filename": "curated_tokenizers-2.0.0-cp311-cp311-win_amd64.whl",
"has_sig": false,
"md5_digest": "62997a2d6da014ee01236faeeefc63d8",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.9",
"size": 760906,
"upload_time": "2024-04-15T17:19:09",
"upload_time_iso_8601": "2024-04-15T17:19:09.823750Z",
"url": "https://files.pythonhosted.org/packages/2b/61/41dc572a610e84b0f87a335f146acd21da6eed646f93bd9c8a27cc99fbfe/curated_tokenizers-2.0.0-cp311-cp311-win_amd64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "4c135788a0b18c126c85b6a44102df6d61e9f991ac15491b3922b73f3447516e",
"md5": "365e9ee0c8c97b49c4e6c5bf253544ff",
"sha256": "480f702afd1d0c53d44f8aaef08f379d769d2894c469777d5d0f9030179f21d4"
},
"downloads": -1,
"filename": "curated_tokenizers-2.0.0-cp312-cp312-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "365e9ee0c8c97b49c4e6c5bf253544ff",
"packagetype": "bdist_wheel",
"python_version": "cp312",
"requires_python": ">=3.9",
"size": 775049,
"upload_time": "2024-04-15T17:19:11",
"upload_time_iso_8601": "2024-04-15T17:19:11.791997Z",
"url": "https://files.pythonhosted.org/packages/4c/13/5788a0b18c126c85b6a44102df6d61e9f991ac15491b3922b73f3447516e/curated_tokenizers-2.0.0-cp312-cp312-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "90e84b5a4f8774f59badf1bc4fadc3ba2aa27b07308556cabe10bd7ab70a6de3",
"md5": "b413b112a69cdb4f6dad2f9b3b8127e9",
"sha256": "99562bbb36e6ec54f6a6cc41c73a2adb8002ec723c2670dcfcb2eee7f384f8a5"
},
"downloads": -1,
"filename": "curated_tokenizers-2.0.0-cp312-cp312-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "b413b112a69cdb4f6dad2f9b3b8127e9",
"packagetype": "bdist_wheel",
"python_version": "cp312",
"requires_python": ">=3.9",
"size": 741438,
"upload_time": "2024-04-15T17:19:13",
"upload_time_iso_8601": "2024-04-15T17:19:13.837146Z",
"url": "https://files.pythonhosted.org/packages/90/e8/4b5a4f8774f59badf1bc4fadc3ba2aa27b07308556cabe10bd7ab70a6de3/curated_tokenizers-2.0.0-cp312-cp312-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a74ff39fb03f8d43e1c7c7dd16523cce711d7b92f83ba2772ceb7a34e235fad6",
"md5": "901c2520f3adf59a6f53dd1d997dc186",
"sha256": "6122c8db5f239189fd4703047315adf3d623a45922bf15cbeb9970bf5833b8f4"
},
"downloads": -1,
"filename": "curated_tokenizers-2.0.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
"has_sig": false,
"md5_digest": "901c2520f3adf59a6f53dd1d997dc186",
"packagetype": "bdist_wheel",
"python_version": "cp312",
"requires_python": ">=3.9",
"size": 746414,
"upload_time": "2024-04-15T17:19:15",
"upload_time_iso_8601": "2024-04-15T17:19:15.884157Z",
"url": "https://files.pythonhosted.org/packages/a7/4f/f39fb03f8d43e1c7c7dd16523cce711d7b92f83ba2772ceb7a34e235fad6/curated_tokenizers-2.0.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "fe330ce2cad39bdcaa8eeda2a7de7ce4613ea6f5c540d5accf0fbe51c9f5253c",
"md5": "1cb2dfa4c793a67344145741ac54e81b",
"sha256": "9b948d299157c317474d7c2c325b073187c6123c5c520b299603548781b8b98a"
},
"downloads": -1,
"filename": "curated_tokenizers-2.0.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "1cb2dfa4c793a67344145741ac54e81b",
"packagetype": "bdist_wheel",
"python_version": "cp312",
"requires_python": ">=3.9",
"size": 775230,
"upload_time": "2024-04-15T17:19:17",
"upload_time_iso_8601": "2024-04-15T17:19:17.874420Z",
"url": "https://files.pythonhosted.org/packages/fe/33/0ce2cad39bdcaa8eeda2a7de7ce4613ea6f5c540d5accf0fbe51c9f5253c/curated_tokenizers-2.0.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "bdf26e9b2915e031f7b6e377bccc8399dab151cc90ef2f1c6f14cce743321fc2",
"md5": "3f8d7e8430f442861218977cb57e9c1e",
"sha256": "32a6e98498f694bcc3c8b3a752f8cb3327b9001d605974d399ceacf66920f37d"
},
"downloads": -1,
"filename": "curated_tokenizers-2.0.0-cp312-cp312-win_amd64.whl",
"has_sig": false,
"md5_digest": "3f8d7e8430f442861218977cb57e9c1e",
"packagetype": "bdist_wheel",
"python_version": "cp312",
"requires_python": ">=3.9",
"size": 761307,
"upload_time": "2024-04-15T17:19:19",
"upload_time_iso_8601": "2024-04-15T17:19:19.746234Z",
"url": "https://files.pythonhosted.org/packages/bd/f2/6e9b2915e031f7b6e377bccc8399dab151cc90ef2f1c6f14cce743321fc2/curated_tokenizers-2.0.0-cp312-cp312-win_amd64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "cbcd583cd4de0990ec9c48d070338b84e96304425f9789a6a78a26b759004277",
"md5": "12c6310c3e2d05b1e3d4c6e9f27dd212",
"sha256": "7375dd55bc72b0e4ff43b547bf369f758f710220c0887e76c483842bcbcd24a1"
},
"downloads": -1,
"filename": "curated_tokenizers-2.0.0-cp39-cp39-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "12c6310c3e2d05b1e3d4c6e9f27dd212",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.9",
"size": 775915,
"upload_time": "2024-04-15T17:19:22",
"upload_time_iso_8601": "2024-04-15T17:19:22.086707Z",
"url": "https://files.pythonhosted.org/packages/cb/cd/583cd4de0990ec9c48d070338b84e96304425f9789a6a78a26b759004277/curated_tokenizers-2.0.0-cp39-cp39-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "0519c269314f931b444b214a3593d694943e382258d766cdd37254465afd791f",
"md5": "33ff639cf9ba2a637d5690eaa9e74260",
"sha256": "a63a9cf96a8dc1737c3b5c9f018f0a8d185b1e7e14631b5daf22c3b3420db9c4"
},
"downloads": -1,
"filename": "curated_tokenizers-2.0.0-cp39-cp39-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "33ff639cf9ba2a637d5690eaa9e74260",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.9",
"size": 743303,
"upload_time": "2024-04-15T17:19:23",
"upload_time_iso_8601": "2024-04-15T17:19:23.898609Z",
"url": "https://files.pythonhosted.org/packages/05/19/c269314f931b444b214a3593d694943e382258d766cdd37254465afd791f/curated_tokenizers-2.0.0-cp39-cp39-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "57a9e011134ff4f9286646d68cb242275148c3a86dea6d02d06d7c4b9a40ec71",
"md5": "994d293b76881d8c9d49206eaab9b6cb",
"sha256": "999963a5854f7cf2c90923ea1613a852e6b4474ecb4f34785a9d49805e37856e"
},
"downloads": -1,
"filename": "curated_tokenizers-2.0.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
"has_sig": false,
"md5_digest": "994d293b76881d8c9d49206eaab9b6cb",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.9",
"size": 747577,
"upload_time": "2024-04-15T17:19:26",
"upload_time_iso_8601": "2024-04-15T17:19:26.024373Z",
"url": "https://files.pythonhosted.org/packages/57/a9/e011134ff4f9286646d68cb242275148c3a86dea6d02d06d7c4b9a40ec71/curated_tokenizers-2.0.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "e1eb1cce7280a389cccdf7cb0b105ebd224a6acb43dc971121f71e08e1dd7f10",
"md5": "3ddccaddb09d131c66204ee5e952eab5",
"sha256": "6acd7931ff5ff620a6f84ea8279312ad1d1e87a1e2014d4367c9e58b8ef23e0d"
},
"downloads": -1,
"filename": "curated_tokenizers-2.0.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "3ddccaddb09d131c66204ee5e952eab5",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.9",
"size": 774766,
"upload_time": "2024-04-15T17:19:27",
"upload_time_iso_8601": "2024-04-15T17:19:27.963038Z",
"url": "https://files.pythonhosted.org/packages/e1/eb/1cce7280a389cccdf7cb0b105ebd224a6acb43dc971121f71e08e1dd7f10/curated_tokenizers-2.0.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6fa31969c4ca7c2cd7069ce32e0bec1d6da0a71a85636d894e1655a357901daa",
"md5": "d130b6b2b201f884c1a346d303080466",
"sha256": "571ee5087323e9478fbf1a3c73e68392fae5908ad12a65d57f985565b7c73f1b"
},
"downloads": -1,
"filename": "curated_tokenizers-2.0.0-cp39-cp39-win_amd64.whl",
"has_sig": false,
"md5_digest": "d130b6b2b201f884c1a346d303080466",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.9",
"size": 762274,
"upload_time": "2024-04-15T17:19:29",
"upload_time_iso_8601": "2024-04-15T17:19:29.750926Z",
"url": "https://files.pythonhosted.org/packages/6f/a3/1969c4ca7c2cd7069ce32e0bec1d6da0a71a85636d894e1655a357901daa/curated_tokenizers-2.0.0-cp39-cp39-win_amd64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "732223b6fe4e15a788405a64304067ca46ff104283f9c3ff9608b4b5be7fac48",
"md5": "42f6216ca15f18cd509001c21d9ba417",
"sha256": "0a8f8c527bc93a6404ec0ba8f0df215b8c205edc81a05617fbbe705438804094"
},
"downloads": -1,
"filename": "curated_tokenizers-2.0.0.tar.gz",
"has_sig": false,
"md5_digest": "42f6216ca15f18cd509001c21d9ba417",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 2262541,
"upload_time": "2024-04-15T17:19:32",
"upload_time_iso_8601": "2024-04-15T17:19:32.449894Z",
"url": "https://files.pythonhosted.org/packages/73/22/23b6fe4e15a788405a64304067ca46ff104283f9c3ff9608b4b5be7fac48/curated_tokenizers-2.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-15 17:19:32",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "explosion",
"github_project": "curated-tokenizers",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "cython",
"specs": [
[
">=",
"0.25"
]
]
},
{
"name": "wheel",
"specs": []
},
{
"name": "regex",
"specs": [
[
">=",
"2022"
]
]
},
{
"name": "pytest",
"specs": [
[
">=",
"5.2.0"
]
]
},
{
"name": "mypy",
"specs": [
[
"<",
"1.6.0"
],
[
">=",
"1.5.0"
]
]
}
],
"lcname": "curated-tokenizers"
}
JSON
