| Name | chonkie JSON |
| Version |
0.1.2
JSON |
| download |
| home_page | None |
| Summary | 🦛 CHONK your texts with Chonkie ✨ - The no-nonsense RAG chunking library |
| upload_time | 2024-11-08 17:27:30 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.8 |
| license | MIT License Copyright (c) 2024 Bhavnick Minhas Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
chunking
rag
nlp
text-processing
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<div align='center'>
# 🦛 Chonkie ✨
[](https://pypi.org/project/chonkie/)
[](https://github.com/bhavnicksm/chonkie/blob/main/LICENSE)
[](DOCS.md)
[](https://pepy.tech/project/chonkie)
[](https://github.com/bhavnicksm/chonkie/stargazers)
_The no-nonsense RAG chunking library that's lightweight, lightning-fast, and ready to CHONK your texts_
[Installation](#installation) •
[Usage](#usage) •
[Supported Methods](#supported-methods) •
[Acknowledgements](#acknowledgements) •
[Citation](#citation)
</div>
so i found myself making another RAG bot (for the 2342148th time) and meanwhile, explaining to my juniors about why we should use chunking in our RAG bots, only to realise that i would have to write chunking all over again unless i use the bloated software library X or the extremely feature-less library Y. _WHY CAN I NOT HAVE SOMETHING JUST RIGHT, UGH?_
Can't i just install, import and run chunking and not have to worry about dependencies, bloat, speed or other factors?
Well, with chonkie you can! (chonkie boi is a gud boi)
**🚀 Feature-rich**: All the CHONKs you'd ever need </br>
**✨ Easy to use**: Install, Import, CHONK </br>
**⚡ Fast**: CHONK at the speed of light! zooooom </br>
**🌐 Wide support**: Supports all your favorite tokenizer CHONKS </br>
**🪶 Light-weight**: No bloat, just CHONK </br>
**🦛 Cute CHONK mascot**: psst it's a pygmy hippo btw </br>
**❤️ [Moto Moto](#acknowledgements)'s favorite python library** </br>
What're you waiting for, **just CHONK it**!
# Installation
To install chonkie, simply run:
```bash
pip install chonkie
```
Chonkie follows the rule to have minimal defualt installs, read the [DOCS](/DOCS.md) to know the installation for your required chunker, or simply install `all` if you don't want to think about it (not recommended).
```bash
pip install chonkie[all]
```
# Usage
Here's a basic example to get you started:
```python
# First import the chunker you want from Chonkie
from chonkie import TokenChunker
# Import your favorite tokenizer library
# Also supports AutoTokenizers, TikToken and AutoTikTokenizer
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_pretrained("gpt2")
# Initialize the chunker
chunker = TokenChunker(tokenizer)
# Chunk some text
chunks = chunker("Woah! Chonkie, the chunking library is so cool!",
"I love the tiny hippo hehe.")
# Access chunks
for chunk in chunks:
print(f"Chunk: {chunk.text}")
print(f"Tokens: {chunk.token_count}")
```
More example usages given inside the [DOCS](/DOCS.md)
# Supported Methods
Chonkie provides several chunkers to help you split your text efficiently for RAG applications. Here's a quick overview of the available chunkers:
- **TokenChunker**: Splits text into fixed-size token chunks.
- **WordChunker**: Splits text into chunks based on words.
- **SentenceChunker**: Splits text into chunks based on sentences.
- **SemanticChunker**: Splits text into chunks based on semantic similarity.
- **SDPMChunker**: Splits text using a Semantic Double-Pass Merge approach.
More on these methods and the approaches taken inside the [DOCS](/DOCS.md)
# Acknowledgements
Chonkie was developed with the support and contributions of the open-source community. We would like to thank the following projects and individuals for their invaluable help:
- **OpenAI** for their amazing [tiktoken](https://github.com/openai/tiktoken) library, which provides the backbone for our tokenization needs.
- **spaCy** for their powerful [spaCy](https://spacy.io/) library, which we use for advanced sentence segmentation.
- **Sentence Transformers** for their [sentence-transformers](https://www.sbert.net/) library, which enables semantic chunking.
- The contributors and maintainers of various open-source projects that have inspired and supported the development of Chonkie.
And to all the users and contributors who have provided feedback, reported issues, and helped improve Chonkie.
Special thanks to **[Moto Moto](https://www.youtube.com/watch?v=I0zZC4wtqDQ&t=5s)** for endorsing Chonkie with his famous quote:
> "I like them big, I like them chonkie."
> ~ Moto Moto
# Citation
If you use Chonkie in your research, please cite it as follows:
```
@misc{chonkie2024,
author = {Minhas, Bhavnick},
title = {Chonkie: A Fast Feature-full Chunking Library for RAG Bots},
year = {2024},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/bhavnick/chonkie}},
}
```
Raw data
{
"_id": null,
"home_page": null,
"name": "chonkie",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "chunking, rag, nlp, text-processing",
"author": null,
"author_email": "Bhavnick Minhas <bhavnicksm@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/4d/2c/6a2dc353c34b179acb06cf49e426072a9bab5f72d48a4d5b3863af01c904/chonkie-0.1.2.tar.gz",
"platform": null,
"description": "<div align='center'>\n\n\n\n# \ud83e\udd9b Chonkie \u2728\n\n[](https://pypi.org/project/chonkie/)\n[](https://github.com/bhavnicksm/chonkie/blob/main/LICENSE)\n[](DOCS.md)\n\n[](https://pepy.tech/project/chonkie)\n[](https://github.com/bhavnicksm/chonkie/stargazers)\n\n_The no-nonsense RAG chunking library that's lightweight, lightning-fast, and ready to CHONK your texts_\n\n[Installation](#installation) \u2022\n[Usage](#usage) \u2022\n[Supported Methods](#supported-methods) \u2022\n[Acknowledgements](#acknowledgements) \u2022\n[Citation](#citation) \n\n</div>\n\nso i found myself making another RAG bot (for the 2342148th time) and meanwhile, explaining to my juniors about why we should use chunking in our RAG bots, only to realise that i would have to write chunking all over again unless i use the bloated software library X or the extremely feature-less library Y. _WHY CAN I NOT HAVE SOMETHING JUST RIGHT, UGH?_\n\nCan't i just install, import and run chunking and not have to worry about dependencies, bloat, speed or other factors?\n\nWell, with chonkie you can! (chonkie boi is a gud boi)\n\n**\ud83d\ude80 Feature-rich**: All the CHONKs you'd ever need </br>\n**\u2728 Easy to use**: Install, Import, CHONK </br>\n**\u26a1 Fast**: CHONK at the speed of light! zooooom </br>\n**\ud83c\udf10 Wide support**: Supports all your favorite tokenizer CHONKS </br>\n**\ud83e\udeb6 Light-weight**: No bloat, just CHONK </br>\n**\ud83e\udd9b Cute CHONK mascot**: psst it's a pygmy hippo btw </br>\n**\u2764\ufe0f [Moto Moto](#acknowledgements)'s favorite python library** </br>\n\nWhat're you waiting for, **just CHONK it**!\n\n# Installation\nTo install chonkie, simply run:\n\n```bash\npip install chonkie\n```\n\nChonkie follows the rule to have minimal defualt installs, read the [DOCS](/DOCS.md) to know the installation for your required chunker, or simply install `all` if you don't want to think about it (not recommended).\n\n```bash\npip install chonkie[all]\n```\n\n# Usage\n\nHere's a basic example to get you started:\n\n```python\n# First import the chunker you want from Chonkie \nfrom chonkie import TokenChunker\n\n# Import your favorite tokenizer library\n# Also supports AutoTokenizers, TikToken and AutoTikTokenizer\nfrom tokenizers import Tokenizer \ntokenizer = Tokenizer.from_pretrained(\"gpt2\")\n\n# Initialize the chunker\nchunker = TokenChunker(tokenizer)\n\n# Chunk some text\nchunks = chunker(\"Woah! Chonkie, the chunking library is so cool!\",\n \"I love the tiny hippo hehe.\")\n\n# Access chunks\nfor chunk in chunks:\n print(f\"Chunk: {chunk.text}\")\n print(f\"Tokens: {chunk.token_count}\")\n```\n\nMore example usages given inside the [DOCS](/DOCS.md)\n\n# Supported Methods\n\nChonkie provides several chunkers to help you split your text efficiently for RAG applications. Here's a quick overview of the available chunkers:\n\n- **TokenChunker**: Splits text into fixed-size token chunks.\n- **WordChunker**: Splits text into chunks based on words.\n- **SentenceChunker**: Splits text into chunks based on sentences.\n- **SemanticChunker**: Splits text into chunks based on semantic similarity.\n- **SDPMChunker**: Splits text using a Semantic Double-Pass Merge approach.\n\nMore on these methods and the approaches taken inside the [DOCS](/DOCS.md)\n\n# Acknowledgements\n\nChonkie was developed with the support and contributions of the open-source community. We would like to thank the following projects and individuals for their invaluable help:\n\n- **OpenAI** for their amazing [tiktoken](https://github.com/openai/tiktoken) library, which provides the backbone for our tokenization needs.\n- **spaCy** for their powerful [spaCy](https://spacy.io/) library, which we use for advanced sentence segmentation.\n- **Sentence Transformers** for their [sentence-transformers](https://www.sbert.net/) library, which enables semantic chunking.\n- The contributors and maintainers of various open-source projects that have inspired and supported the development of Chonkie.\n\nAnd to all the users and contributors who have provided feedback, reported issues, and helped improve Chonkie.\n\nSpecial thanks to **[Moto Moto](https://www.youtube.com/watch?v=I0zZC4wtqDQ&t=5s)** for endorsing Chonkie with his famous quote: \n> \"I like them big, I like them chonkie.\"\n> ~ Moto Moto\n\n# Citation\n\nIf you use Chonkie in your research, please cite it as follows:\n\n```\n@misc{chonkie2024,\n author = {Minhas, Bhavnick},\n title = {Chonkie: A Fast Feature-full Chunking Library for RAG Bots},\n year = {2024},\n publisher = {GitHub},\n journal = {GitHub repository},\n howpublished = {\\url{https://github.com/bhavnick/chonkie}},\n}\n```\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) 2024 Bhavnick Minhas Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
"summary": "\ud83e\udd9b CHONK your texts with Chonkie \u2728 - The no-nonsense RAG chunking library",
"version": "0.1.2",
"project_urls": {
"Homepage": "https://github.com/bhavnicksm/chonkie"
},
"split_keywords": [
"chunking",
" rag",
" nlp",
" text-processing"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "be01e6d330910afd84080a0fc058c942886ffefaf8e4f0534bf709553399ae82",
"md5": "d580dd25f0b9e1ed60cde3177571ead1",
"sha256": "9cbe8f5f658efb2681fcbd411b74a8378d6fb9fa2054fd4741f68f8de69f159e"
},
"downloads": -1,
"filename": "chonkie-0.1.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d580dd25f0b9e1ed60cde3177571ead1",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 19545,
"upload_time": "2024-11-08T17:27:28",
"upload_time_iso_8601": "2024-11-08T17:27:28.553636Z",
"url": "https://files.pythonhosted.org/packages/be/01/e6d330910afd84080a0fc058c942886ffefaf8e4f0534bf709553399ae82/chonkie-0.1.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "4d2c6a2dc353c34b179acb06cf49e426072a9bab5f72d48a4d5b3863af01c904",
"md5": "b583a12ce7a01874627694aa1551dd2c",
"sha256": "3f00b9fede7cdc4ec1776dfb93a779058dc2f092fe6f34d0d8279caa9e84fcea"
},
"downloads": -1,
"filename": "chonkie-0.1.2.tar.gz",
"has_sig": false,
"md5_digest": "b583a12ce7a01874627694aa1551dd2c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 18168,
"upload_time": "2024-11-08T17:27:30",
"upload_time_iso_8601": "2024-11-08T17:27:30.216650Z",
"url": "https://files.pythonhosted.org/packages/4d/2c/6a2dc353c34b179acb06cf49e426072a9bab5f72d48a4d5b3863af01c904/chonkie-0.1.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-08 17:27:30",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "bhavnicksm",
"github_project": "chonkie",
"github_not_found": true,
"lcname": "chonkie"
}
