| Name | tiktoken-async JSON |
| Version |
0.3.2
JSON |
| download |
| home_page | |
| Summary | tiktoken-async is a fast BPE tokeniser for use with OpenAI's models, with added support for asynchronous processing. |
| upload_time | 2023-03-28 02:09:19 |
| maintainer | |
| docs_url | None |
| author | Shantanu Jain |
| requires_python | >=3.8 |
| license | MIT License Copyright (c) 2022 OpenAI, Shantanu Jain Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# ⏳ tiktoken-async
tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with
OpenAI's models.
```python
import asyncio
import tiktoken_async
enc = asyncio.run(tiktoken_async.get_encoding("cl100k_base"))
assert enc.decode(enc.encode("hello world")) == "hello world"
# To get the tokeniser corresponding to a specific model in the OpenAI API:
enc = asyncio.run(tiktoken_async.encoding_for_model("gpt-4"))
```
The open source version of `tiktoken-async` can be installed from PyPI:
```
pip install tiktoken-async
```
The tokeniser API is documented in `tiktoken_async/core.py`.
Example code using `tiktoken` can be found in the
[OpenAI Cookbook](https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb).
## Performance
`tiktoken` is between 3-6x faster than a comparable open source tokeniser:
Performance measured on 1GB of text using the GPT-2 tokeniser, using `GPT2TokenizerFast` from
`tokenizers==0.13.2`, `transformers==4.24.0` and `tiktoken==0.2.0`.
## Getting help
Please post questions in the [issue tracker](https://github.com/openai/tiktoken/issues).
If you work at OpenAI, make sure to check the internal documentation or feel free to contact
@shantanu.
## Extending tiktoken
You may wish to extend `tiktoken-async` to support new encodings. There are two ways to do this.
**Create your `Encoding` object exactly the way you want and simply pass it around.**
```python
import asyncio
cl100k_base = asyncio.run(tiktoken.get_encoding("cl100k_base"))
# In production, load the arguments directly instead of accessing private attributes
# See openai_public.py for examples of arguments for specific encodings
enc = tiktoken_async.Encoding(
# If you're changing the set of special tokens, make sure to use a different name
# It should be clear from the name what behaviour to expect.
name="cl100k_im",
pat_str=cl100k_base._pat_str,
mergeable_ranks=cl100k_base._mergeable_ranks,
special_tokens={
**cl100k_base._special_tokens,
"<|im_start|>": 100264,
"<|im_end|>": 100265,
}
)
```
**Use the `tiktoken_async_ext` plugin mechanism to register your `Encoding` objects with `tiktoken_async`.**
This is only useful if you need `tiktoken_async.get_encoding` to find your encoding, otherwise prefer
option 1.
To do this, you'll need to create a namespace package under `tiktoken_async_ext`.
Layout your project like this, making sure to omit the `tiktoken_ext/__init__.py` file:
```
my_tiktoken_extension
├── tiktoken_async_ext
│ └── my_encodings.py
└── setup.py
```
`my_encodings.py` should be a module that contains a variable named `ENCODING_CONSTRUCTORS`.
This is a dictionary from an encoding name to a function that takes no arguments and returns
arguments that can be passed to `tiktoken_async.Encoding` to construct that encoding. For an example, see
`tiktoken_async_ext/openai_public.py`. For precise details, see `tiktoken_async/registry.py`.
Your `setup.py` should look something like this:
```python
from setuptools import setup, find_namespace_packages
setup(
name="my_tiktoken_extension",
packages=find_namespace_packages(include=['tiktoken_async_ext*']),
install_requires=["tiktoken_async"],
...
)
```
Then simply `pip install ./my_tiktoken_extension` and you should be able to use your
custom encodings! Make sure **not** to use an editable install.
Raw data
{
"_id": null,
"home_page": "",
"name": "tiktoken-async",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "",
"author": "Shantanu Jain",
"author_email": "shantanu@openai.com",
"download_url": "https://files.pythonhosted.org/packages/30/3a/f446013b23c59b948e11ead63fc439bf45efbe32bd271cbc5e77a92cd432/tiktoken-async-0.3.2.tar.gz",
"platform": null,
"description": "# \u23f3 tiktoken-async\n\ntiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with\nOpenAI's models.\n\n```python\nimport asyncio\nimport tiktoken_async\nenc = asyncio.run(tiktoken_async.get_encoding(\"cl100k_base\"))\nassert enc.decode(enc.encode(\"hello world\")) == \"hello world\"\n\n# To get the tokeniser corresponding to a specific model in the OpenAI API:\nenc = asyncio.run(tiktoken_async.encoding_for_model(\"gpt-4\"))\n```\n\nThe open source version of `tiktoken-async` can be installed from PyPI:\n\n```\npip install tiktoken-async\n```\n\nThe tokeniser API is documented in `tiktoken_async/core.py`.\n\nExample code using `tiktoken` can be found in the\n[OpenAI Cookbook](https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb).\n\n## Performance\n\n`tiktoken` is between 3-6x faster than a comparable open source tokeniser:\n\n\n\nPerformance measured on 1GB of text using the GPT-2 tokeniser, using `GPT2TokenizerFast` from\n`tokenizers==0.13.2`, `transformers==4.24.0` and `tiktoken==0.2.0`.\n\n## Getting help\n\nPlease post questions in the [issue tracker](https://github.com/openai/tiktoken/issues).\n\nIf you work at OpenAI, make sure to check the internal documentation or feel free to contact\n@shantanu.\n\n## Extending tiktoken\n\nYou may wish to extend `tiktoken-async` to support new encodings. There are two ways to do this.\n\n**Create your `Encoding` object exactly the way you want and simply pass it around.**\n\n```python\nimport asyncio\n\ncl100k_base = asyncio.run(tiktoken.get_encoding(\"cl100k_base\"))\n\n# In production, load the arguments directly instead of accessing private attributes\n# See openai_public.py for examples of arguments for specific encodings\nenc = tiktoken_async.Encoding(\n # If you're changing the set of special tokens, make sure to use a different name\n # It should be clear from the name what behaviour to expect.\n name=\"cl100k_im\",\n pat_str=cl100k_base._pat_str,\n mergeable_ranks=cl100k_base._mergeable_ranks,\n special_tokens={\n **cl100k_base._special_tokens,\n \"<|im_start|>\": 100264,\n \"<|im_end|>\": 100265,\n }\n)\n```\n\n**Use the `tiktoken_async_ext` plugin mechanism to register your `Encoding` objects with `tiktoken_async`.**\n\nThis is only useful if you need `tiktoken_async.get_encoding` to find your encoding, otherwise prefer\noption 1.\n\nTo do this, you'll need to create a namespace package under `tiktoken_async_ext`.\n\nLayout your project like this, making sure to omit the `tiktoken_ext/__init__.py` file:\n\n```\nmy_tiktoken_extension\n\u251c\u2500\u2500 tiktoken_async_ext\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 my_encodings.py\n\u2514\u2500\u2500 setup.py\n```\n\n`my_encodings.py` should be a module that contains a variable named `ENCODING_CONSTRUCTORS`.\nThis is a dictionary from an encoding name to a function that takes no arguments and returns\narguments that can be passed to `tiktoken_async.Encoding` to construct that encoding. For an example, see\n`tiktoken_async_ext/openai_public.py`. For precise details, see `tiktoken_async/registry.py`.\n\nYour `setup.py` should look something like this:\n\n```python\nfrom setuptools import setup, find_namespace_packages\n\nsetup(\n name=\"my_tiktoken_extension\",\n packages=find_namespace_packages(include=['tiktoken_async_ext*']),\n install_requires=[\"tiktoken_async\"],\n ...\n)\n```\n\nThen simply `pip install ./my_tiktoken_extension` and you should be able to use your\ncustom encodings! Make sure **not** to use an editable install.\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) 2022 OpenAI, Shantanu Jain Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
"summary": "tiktoken-async is a fast BPE tokeniser for use with OpenAI's models, with added support for asynchronous processing.",
"version": "0.3.2",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "42b2c430f350199551d394f729bab7697e53fa869c51faa18dd0707930ea6e8c",
"md5": "2805d15696a192b100990093f79b29f3",
"sha256": "f1938b39f4de8424400eb5074e8cec22f812c5a7e0d58aec632a76d337fa4820"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp310-cp310-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "2805d15696a192b100990093f79b29f3",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.8",
"size": 735617,
"upload_time": "2023-03-28T02:08:09",
"upload_time_iso_8601": "2023-03-28T02:08:09.450815Z",
"url": "https://files.pythonhosted.org/packages/42/b2/c430f350199551d394f729bab7697e53fa869c51faa18dd0707930ea6e8c/tiktoken_async-0.3.2-cp310-cp310-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "5044aff7552ddae3f0608ff4bfc58db3d4caadb24de3d0672e4cc347d902d45b",
"md5": "59f9fa813009eca13828c6927cb13ab3",
"sha256": "5f99d02259570446f9f96138d3ef4ffe51f7ab5ec934027703c435e2e33d4cba"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp310-cp310-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "59f9fa813009eca13828c6927cb13ab3",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.8",
"size": 705510,
"upload_time": "2023-03-28T02:08:11",
"upload_time_iso_8601": "2023-03-28T02:08:11.867793Z",
"url": "https://files.pythonhosted.org/packages/50/44/aff7552ddae3f0608ff4bfc58db3d4caadb24de3d0672e4cc347d902d45b/tiktoken_async-0.3.2-cp310-cp310-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "4ea572a8c93fe8e4bf535e9d8a79d43fda7dd7c23a329a014101a170be522a66",
"md5": "276dff837e302c79960f7e1571d27ddb",
"sha256": "97e1cffe734e4a168c57e66a4fccb8a0a6c82b7be763efa6c781b2ce5fe00812"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
"has_sig": false,
"md5_digest": "276dff837e302c79960f7e1571d27ddb",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.8",
"size": 1616819,
"upload_time": "2023-03-28T02:08:13",
"upload_time_iso_8601": "2023-03-28T02:08:13.796400Z",
"url": "https://files.pythonhosted.org/packages/4e/a5/72a8c93fe8e4bf535e9d8a79d43fda7dd7c23a329a014101a170be522a66/tiktoken_async-0.3.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "82e6d739fbfb7afd567fdeff07a3d75f942c89f9b007c1de1644aac83813c273",
"md5": "d52a734cf1d8f68948ae144fa02c00a0",
"sha256": "520bb5144021336f071c6609a9c524011d4b8de8ece0efabd99848490138def4"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "d52a734cf1d8f68948ae144fa02c00a0",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.8",
"size": 1661181,
"upload_time": "2023-03-28T02:08:16",
"upload_time_iso_8601": "2023-03-28T02:08:16.298589Z",
"url": "https://files.pythonhosted.org/packages/82/e6/d739fbfb7afd567fdeff07a3d75f942c89f9b007c1de1644aac83813c273/tiktoken_async-0.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "00ac574c42b77ed1d7cb5ea4b5952b85120ee40a7d1df6920523abb4da770aae",
"md5": "c19d7cbed8545cd4673c7aac9603137c",
"sha256": "1b1943c55a41daec559f3bf811c8a5622d0e3d350a5ea85099ace6c473cb169d"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp310-cp310-musllinux_1_1_aarch64.whl",
"has_sig": false,
"md5_digest": "c19d7cbed8545cd4673c7aac9603137c",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.8",
"size": 1647042,
"upload_time": "2023-03-28T02:08:18",
"upload_time_iso_8601": "2023-03-28T02:08:18.751472Z",
"url": "https://files.pythonhosted.org/packages/00/ac/574c42b77ed1d7cb5ea4b5952b85120ee40a7d1df6920523abb4da770aae/tiktoken_async-0.3.2-cp310-cp310-musllinux_1_1_aarch64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "75dd613e4a3e828458c69358af71ad6a29767caed679ae11f9653f0f23a12721",
"md5": "b83025b3715be303dc3aa321cc781081",
"sha256": "4f5d14f10c48b27edaf00833b402f301136060c8d610d949f22c2d427b3faa81"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp310-cp310-musllinux_1_1_x86_64.whl",
"has_sig": false,
"md5_digest": "b83025b3715be303dc3aa321cc781081",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.8",
"size": 1688808,
"upload_time": "2023-03-28T02:08:21",
"upload_time_iso_8601": "2023-03-28T02:08:21.102769Z",
"url": "https://files.pythonhosted.org/packages/75/dd/613e4a3e828458c69358af71ad6a29767caed679ae11f9653f0f23a12721/tiktoken_async-0.3.2-cp310-cp310-musllinux_1_1_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "7ff1fd33b0e0e8d64332888557f136225e57a722c4d50fd39efd5c73bc9aba0c",
"md5": "caa8d707fec63755b72d4465dfeb4708",
"sha256": "c2fc609ebd993c806035618b38a608090c7ec6b3c36b59c550bf0db9685c337c"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp310-cp310-win_amd64.whl",
"has_sig": false,
"md5_digest": "caa8d707fec63755b72d4465dfeb4708",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.8",
"size": 579485,
"upload_time": "2023-03-28T02:08:22",
"upload_time_iso_8601": "2023-03-28T02:08:22.959720Z",
"url": "https://files.pythonhosted.org/packages/7f/f1/fd33b0e0e8d64332888557f136225e57a722c4d50fd39efd5c73bc9aba0c/tiktoken_async-0.3.2-cp310-cp310-win_amd64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1250c7d41a0200b7e64a9e6856f846e8524de42d9f3b2188c72115675158cdd6",
"md5": "ba9822c931cbca9c9dc76b09ff2b3424",
"sha256": "466d73c6bbfc360154d144b2995ab4a50957b56a5872ab76248ec68819bda191"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp311-cp311-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "ba9822c931cbca9c9dc76b09ff2b3424",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.8",
"size": 735620,
"upload_time": "2023-03-28T02:08:25",
"upload_time_iso_8601": "2023-03-28T02:08:25.402758Z",
"url": "https://files.pythonhosted.org/packages/12/50/c7d41a0200b7e64a9e6856f846e8524de42d9f3b2188c72115675158cdd6/tiktoken_async-0.3.2-cp311-cp311-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "18727b7c682ad896229d175fbe5e1acd8a64e91fb83b544622c2f272129e0736",
"md5": "60852c95c7014dabbe1236b3903026d1",
"sha256": "245f6b3f39c294dcfdfd936a9bb6a0aa324f42729426c3f874907ebc22668487"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp311-cp311-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "60852c95c7014dabbe1236b3903026d1",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.8",
"size": 705512,
"upload_time": "2023-03-28T02:08:27",
"upload_time_iso_8601": "2023-03-28T02:08:27.674444Z",
"url": "https://files.pythonhosted.org/packages/18/72/7b7c682ad896229d175fbe5e1acd8a64e91fb83b544622c2f272129e0736/tiktoken_async-0.3.2-cp311-cp311-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "11ebdd69591a174d7d67624bfe24796284a5d2fbf6a2158893fa0706878218b7",
"md5": "64a1c4160a7f707dfb0375d745d43d60",
"sha256": "0e5f9fff300ae5a81d3ca8a330f2b37b3f93cc9eece8a9aa5fd4e5f2f2df047c"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
"has_sig": false,
"md5_digest": "64a1c4160a7f707dfb0375d745d43d60",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.8",
"size": 1616820,
"upload_time": "2023-03-28T02:08:29",
"upload_time_iso_8601": "2023-03-28T02:08:29.574327Z",
"url": "https://files.pythonhosted.org/packages/11/eb/dd69591a174d7d67624bfe24796284a5d2fbf6a2158893fa0706878218b7/tiktoken_async-0.3.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "87ef2d891eff86271440c88ddc0540f0c43fe05e09a335e5f001783c1e0075d6",
"md5": "3ca6485505bc9919f220758209ea043d",
"sha256": "bcaef0cf8782898477bfc5f5bb20171fc8a1f2be3ce6765c85d2994ade95a78a"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "3ca6485505bc9919f220758209ea043d",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.8",
"size": 1661182,
"upload_time": "2023-03-28T02:08:32",
"upload_time_iso_8601": "2023-03-28T02:08:32.067465Z",
"url": "https://files.pythonhosted.org/packages/87/ef/2d891eff86271440c88ddc0540f0c43fe05e09a335e5f001783c1e0075d6/tiktoken_async-0.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "4121e2696138e9d29c2af510031a5223a8240d49be0c252267df7fc3fe8ec9f3",
"md5": "736c82a2e35ce52246ea5a1af98696de",
"sha256": "0fc630bd6da9e8c91c725aad48298e13509ef8028ab23f1338e67b51a4181583"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp311-cp311-musllinux_1_1_aarch64.whl",
"has_sig": false,
"md5_digest": "736c82a2e35ce52246ea5a1af98696de",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.8",
"size": 1647047,
"upload_time": "2023-03-28T02:08:34",
"upload_time_iso_8601": "2023-03-28T02:08:34.637964Z",
"url": "https://files.pythonhosted.org/packages/41/21/e2696138e9d29c2af510031a5223a8240d49be0c252267df7fc3fe8ec9f3/tiktoken_async-0.3.2-cp311-cp311-musllinux_1_1_aarch64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "01b3a394f7e49769c3d91d2bc4ccfbb27ab9f047cee7bc731c1bb3f4448cc4b4",
"md5": "00280b3441b425e002da3929f4a9f92c",
"sha256": "9ec71580ba58f61bd0db24a96dffa43136cdaf88966026ebbf564abcf4d88b63"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp311-cp311-musllinux_1_1_x86_64.whl",
"has_sig": false,
"md5_digest": "00280b3441b425e002da3929f4a9f92c",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.8",
"size": 1688814,
"upload_time": "2023-03-28T02:08:36",
"upload_time_iso_8601": "2023-03-28T02:08:36.830935Z",
"url": "https://files.pythonhosted.org/packages/01/b3/a394f7e49769c3d91d2bc4ccfbb27ab9f047cee7bc731c1bb3f4448cc4b4/tiktoken_async-0.3.2-cp311-cp311-musllinux_1_1_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a7dc5fdb3d067ba06d8992dac3949425ea3cd68fae2409a4bb45472c9d8f7969",
"md5": "ab6ebb0439d18b14f923c1cfbdc1e3de",
"sha256": "51401e718c2b46a6b1899d91d4746b93cc35e8cdcac0828a5c3653fd57af7170"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp311-cp311-win_amd64.whl",
"has_sig": false,
"md5_digest": "ab6ebb0439d18b14f923c1cfbdc1e3de",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.8",
"size": 579491,
"upload_time": "2023-03-28T02:08:39",
"upload_time_iso_8601": "2023-03-28T02:08:39.239441Z",
"url": "https://files.pythonhosted.org/packages/a7/dc/5fdb3d067ba06d8992dac3949425ea3cd68fae2409a4bb45472c9d8f7969/tiktoken_async-0.3.2-cp311-cp311-win_amd64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "8ae7d6c9cf2df8e520f813e388f0355df7e105809b0598d109cbe24df75f4099",
"md5": "5fb54c6237217a4f7d54a4599eb107ca",
"sha256": "74caa0778d9ff3093613e7317514e3adac4c67d0ea61c3288b70eb38d657397d"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp38-cp38-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "5fb54c6237217a4f7d54a4599eb107ca",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.8",
"size": 736436,
"upload_time": "2023-03-28T02:08:41",
"upload_time_iso_8601": "2023-03-28T02:08:41.083475Z",
"url": "https://files.pythonhosted.org/packages/8a/e7/d6c9cf2df8e520f813e388f0355df7e105809b0598d109cbe24df75f4099/tiktoken_async-0.3.2-cp38-cp38-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "5c39d7189bebc53d5ab348f900692c1857a1f6e31cf7b33091fee88aa1d02224",
"md5": "04c604cdad9dd35b06583b0dc87d82ba",
"sha256": "3831cd07b94fccc7afed24c66dfde28c55e38fb22617bcfce6936c29ea66b411"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp38-cp38-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "04c604cdad9dd35b06583b0dc87d82ba",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.8",
"size": 705491,
"upload_time": "2023-03-28T02:08:42",
"upload_time_iso_8601": "2023-03-28T02:08:42.881223Z",
"url": "https://files.pythonhosted.org/packages/5c/39/d7189bebc53d5ab348f900692c1857a1f6e31cf7b33091fee88aa1d02224/tiktoken_async-0.3.2-cp38-cp38-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "31ece0ace43392dd259ce6616944792c25716d25655e990a0bbe96550307e0cd",
"md5": "2997a8589bc000092e6115b662de0576",
"sha256": "dfedd2ba093cfa2eed97726ca26fa6d3562009196a4eb6e541a7849073087122"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
"has_sig": false,
"md5_digest": "2997a8589bc000092e6115b662de0576",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.8",
"size": 1616963,
"upload_time": "2023-03-28T02:08:45",
"upload_time_iso_8601": "2023-03-28T02:08:45.496531Z",
"url": "https://files.pythonhosted.org/packages/31/ec/e0ace43392dd259ce6616944792c25716d25655e990a0bbe96550307e0cd/tiktoken_async-0.3.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a1f6b680fb796f37f16419e76406ff2e93016af18dacb0d5739925071ffd8ab0",
"md5": "98183fdcd672ceba0dc0b15b7421a256",
"sha256": "67c5be1908e2bc2c59855717711be35871439ab59ffca08c046746eff20b760f"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "98183fdcd672ceba0dc0b15b7421a256",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.8",
"size": 1661203,
"upload_time": "2023-03-28T02:08:48",
"upload_time_iso_8601": "2023-03-28T02:08:48.431303Z",
"url": "https://files.pythonhosted.org/packages/a1/f6/b680fb796f37f16419e76406ff2e93016af18dacb0d5739925071ffd8ab0/tiktoken_async-0.3.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1ac500cfe57acf146eb702415019e307dfe4e4c18d67cc51e7c07f99a9c53a4d",
"md5": "05466e9d224bf064f14f6539f3965a10",
"sha256": "d042f38dc00653465b7459d127278a8eab8b7f6afcd92b407d4c075403f4cdda"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp38-cp38-musllinux_1_1_aarch64.whl",
"has_sig": false,
"md5_digest": "05466e9d224bf064f14f6539f3965a10",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.8",
"size": 1647236,
"upload_time": "2023-03-28T02:08:51",
"upload_time_iso_8601": "2023-03-28T02:08:51.293593Z",
"url": "https://files.pythonhosted.org/packages/1a/c5/00cfe57acf146eb702415019e307dfe4e4c18d67cc51e7c07f99a9c53a4d/tiktoken_async-0.3.2-cp38-cp38-musllinux_1_1_aarch64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "43c24514eec44ebdedd85f9a40844075a2b2d0edfe7d81478d92ec5d3fc2cfe1",
"md5": "e0d4c9c6245606447be070849d019ac0",
"sha256": "d4df50757bfd5133c8acea8a920c16f9fb62c95c56a505fa53d4327e28d977c0"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp38-cp38-musllinux_1_1_x86_64.whl",
"has_sig": false,
"md5_digest": "e0d4c9c6245606447be070849d019ac0",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.8",
"size": 1688810,
"upload_time": "2023-03-28T02:08:54",
"upload_time_iso_8601": "2023-03-28T02:08:54.166216Z",
"url": "https://files.pythonhosted.org/packages/43/c2/4514eec44ebdedd85f9a40844075a2b2d0edfe7d81478d92ec5d3fc2cfe1/tiktoken_async-0.3.2-cp38-cp38-musllinux_1_1_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "5487088e691d859da5ae2a71ae108b71f4bab42037c2238d58eaaec506762423",
"md5": "182df807f3dc2906b063d9b4d8026926",
"sha256": "032ef349022a4668a8c309fba147d1b8e37aa4ec9873e85344067fdea1b35146"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp38-cp38-win_amd64.whl",
"has_sig": false,
"md5_digest": "182df807f3dc2906b063d9b4d8026926",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.8",
"size": 579770,
"upload_time": "2023-03-28T02:08:57",
"upload_time_iso_8601": "2023-03-28T02:08:57.072343Z",
"url": "https://files.pythonhosted.org/packages/54/87/088e691d859da5ae2a71ae108b71f4bab42037c2238d58eaaec506762423/tiktoken_async-0.3.2-cp38-cp38-win_amd64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3d91f4f18d69811e35c7c653653471b74012a416f7610f544e05d5d9786aa3e8",
"md5": "9bdc3246e388b2dc3e4d3b812c0b44e5",
"sha256": "ece21c562da8762a5eec61d1caa8e9639d76b378d30ee31d901e1babf83e30c0"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp39-cp39-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "9bdc3246e388b2dc3e4d3b812c0b44e5",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.8",
"size": 736317,
"upload_time": "2023-03-28T02:08:59",
"upload_time_iso_8601": "2023-03-28T02:08:59.243628Z",
"url": "https://files.pythonhosted.org/packages/3d/91/f4f18d69811e35c7c653653471b74012a416f7610f544e05d5d9786aa3e8/tiktoken_async-0.3.2-cp39-cp39-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "01080c3bbbddc6af141115b254ce2896444f1c0f595e6e6047b53b4f6d264e50",
"md5": "d0071589ec211bbbea07ba41efbbb668",
"sha256": "4ba3bfba7999e58782ad38241c83688cec239ab760eec52bdd081f0ad86ef7bc"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp39-cp39-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "d0071589ec211bbbea07ba41efbbb668",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.8",
"size": 705964,
"upload_time": "2023-03-28T02:09:02",
"upload_time_iso_8601": "2023-03-28T02:09:02.040334Z",
"url": "https://files.pythonhosted.org/packages/01/08/0c3bbbddc6af141115b254ce2896444f1c0f595e6e6047b53b4f6d264e50/tiktoken_async-0.3.2-cp39-cp39-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "7aa3aa6c264984d89ad376c3874722480e519a54fb660adb014ff74349f83fb8",
"md5": "7b60e76a316d6cc7fe461c0da9937f47",
"sha256": "2efd96a1977ce9a153c4070f7ee9bbdae992af94881171afefc422ec2e12f780"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
"has_sig": false,
"md5_digest": "7b60e76a316d6cc7fe461c0da9937f47",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.8",
"size": 1617032,
"upload_time": "2023-03-28T02:09:04",
"upload_time_iso_8601": "2023-03-28T02:09:04.638748Z",
"url": "https://files.pythonhosted.org/packages/7a/a3/aa6c264984d89ad376c3874722480e519a54fb660adb014ff74349f83fb8/tiktoken_async-0.3.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "7ceece99403345831a7881829f08829b4366f89618e3a5f843f431a461aefd51",
"md5": "af5f2e9b4a68eedf49329710825ffbf5",
"sha256": "aa25e57c61c2581df0141cb2c3ade793d85b1abc13d7d382a0f17ae0e17ccd6f"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "af5f2e9b4a68eedf49329710825ffbf5",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.8",
"size": 1661582,
"upload_time": "2023-03-28T02:09:07",
"upload_time_iso_8601": "2023-03-28T02:09:07.899105Z",
"url": "https://files.pythonhosted.org/packages/7c/ee/ce99403345831a7881829f08829b4366f89618e3a5f843f431a461aefd51/tiktoken_async-0.3.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "025cfba5f8d1c941369ffad48a6704d787bd48b7f3eaa7e5744a1545fd4f9496",
"md5": "8b185e7a42a087bec9a4898d63dc0e5a",
"sha256": "c15280821bf0770ccc4c2b0de5dde8f7aae2f7d3444c42bef0edf6819006092b"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp39-cp39-musllinux_1_1_aarch64.whl",
"has_sig": false,
"md5_digest": "8b185e7a42a087bec9a4898d63dc0e5a",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.8",
"size": 1647415,
"upload_time": "2023-03-28T02:09:10",
"upload_time_iso_8601": "2023-03-28T02:09:10.645836Z",
"url": "https://files.pythonhosted.org/packages/02/5c/fba5f8d1c941369ffad48a6704d787bd48b7f3eaa7e5744a1545fd4f9496/tiktoken_async-0.3.2-cp39-cp39-musllinux_1_1_aarch64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "0cd6083dfc3f0463122a082853141b054643772f194c91a383d4c0fa8dd20572",
"md5": "538eef12237c1ab543a790b70612fe29",
"sha256": "3bb88a994e013b7474b5d53688a0e305cd78bb921ba09e4bf17b6a76e7f36b9a"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp39-cp39-musllinux_1_1_x86_64.whl",
"has_sig": false,
"md5_digest": "538eef12237c1ab543a790b70612fe29",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.8",
"size": 1689137,
"upload_time": "2023-03-28T02:09:14",
"upload_time_iso_8601": "2023-03-28T02:09:14.739346Z",
"url": "https://files.pythonhosted.org/packages/0c/d6/083dfc3f0463122a082853141b054643772f194c91a383d4c0fa8dd20572/tiktoken_async-0.3.2-cp39-cp39-musllinux_1_1_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9bf15b628af0ea6999c28d19ca7b994ce260a7a27ff0d3abbd50264406471525",
"md5": "9c63f9bcf436319819c10341e4802056",
"sha256": "c3bf7e043cac0ad4f879dec9a20ad0b6b9ff7d1fa2cde505fbb9c096335cd982"
},
"downloads": -1,
"filename": "tiktoken_async-0.3.2-cp39-cp39-win_amd64.whl",
"has_sig": false,
"md5_digest": "9c63f9bcf436319819c10341e4802056",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.8",
"size": 579809,
"upload_time": "2023-03-28T02:09:17",
"upload_time_iso_8601": "2023-03-28T02:09:17.315967Z",
"url": "https://files.pythonhosted.org/packages/9b/f1/5b628af0ea6999c28d19ca7b994ce260a7a27ff0d3abbd50264406471525/tiktoken_async-0.3.2-cp39-cp39-win_amd64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "303af446013b23c59b948e11ead63fc439bf45efbe32bd271cbc5e77a92cd432",
"md5": "78684d79d730fd40ffcf29321fa200f2",
"sha256": "b14fcd2b811b99e4de49b8d4fadb78bb18775e9e58df7b33b192307741c7a72d"
},
"downloads": -1,
"filename": "tiktoken-async-0.3.2.tar.gz",
"has_sig": false,
"md5_digest": "78684d79d730fd40ffcf29321fa200f2",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 25202,
"upload_time": "2023-03-28T02:09:19",
"upload_time_iso_8601": "2023-03-28T02:09:19.001132Z",
"url": "https://files.pythonhosted.org/packages/30/3a/f446013b23c59b948e11ead63fc439bf45efbe32bd271cbc5e77a92cd432/tiktoken-async-0.3.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-03-28 02:09:19",
"github": false,
"gitlab": false,
"bitbucket": false,
"lcname": "tiktoken-async"
}
