# Semantic Chunking Llama Pack
This LlamaPack implements the semantic chunking algorithm first proposed by Greg Kamradt in his [Five Levels of Text Splitting](https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/5_Levels_Of_Text_Splitting.ipynb) tutorial.
How it works:
- Split text into sentences.
- For each sentence, generate an embedding.
- Measure cosine distance between each pair of consecutive sentences.
- Get the 95% percentile cosine distance, set that as the threshold.
- Create a new chunk if the cosine distance of a sentence compared to prev. exceeds that threshold.
## CLI Usage
You can download llamapacks directly using `llamaindex-cli`, which comes installed with the `llama-index` python package:
```bash
llamaindex-cli download-llamapack LLMCompilerAgentPack --download-dir ./llm_compiler_agent_pack
```
You can then inspect the files at `./llm_compiler_agent_pack` and use them as a template for your own project!
## Code Usage
You can download the pack to a directory. **NOTE**: You must specify `skip_load=True` - the pack contains multiple files,
which makes it hard to load directly.
We will show you how to import the agent from these files!
Raw data
{
"_id": null,
"home_page": null,
"name": "llama-index-packs-node-parser-semantic-chunking",
"maintainer": "jerryjliu",
"docs_url": null,
"requires_python": "<4.0,>=3.9",
"maintainer_email": null,
"keywords": "chunk, chunking, embedding, node, parser, semantic",
"author": null,
"author_email": "Your Name <you@example.com>",
"download_url": "https://files.pythonhosted.org/packages/82/99/e2ece2051d48bcf731e6eada2da11a67698baa3be170a9675dff4b6c357d/llama_index_packs_node_parser_semantic_chunking-0.4.0.tar.gz",
"platform": null,
"description": "# Semantic Chunking Llama Pack\n\nThis LlamaPack implements the semantic chunking algorithm first proposed by Greg Kamradt in his [Five Levels of Text Splitting](https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/5_Levels_Of_Text_Splitting.ipynb) tutorial.\n\nHow it works:\n\n- Split text into sentences.\n- For each sentence, generate an embedding.\n- Measure cosine distance between each pair of consecutive sentences.\n- Get the 95% percentile cosine distance, set that as the threshold.\n- Create a new chunk if the cosine distance of a sentence compared to prev. exceeds that threshold.\n\n## CLI Usage\n\nYou can download llamapacks directly using `llamaindex-cli`, which comes installed with the `llama-index` python package:\n\n```bash\nllamaindex-cli download-llamapack LLMCompilerAgentPack --download-dir ./llm_compiler_agent_pack\n```\n\nYou can then inspect the files at `./llm_compiler_agent_pack` and use them as a template for your own project!\n\n## Code Usage\n\nYou can download the pack to a directory. **NOTE**: You must specify `skip_load=True` - the pack contains multiple files,\nwhich makes it hard to load directly.\n\nWe will show you how to import the agent from these files!\n",
"bugtrack_url": null,
"license": null,
"summary": "llama-index packs node_parser integration",
"version": "0.4.0",
"project_urls": null,
"split_keywords": [
"chunk",
" chunking",
" embedding",
" node",
" parser",
" semantic"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "65df68d58ee1616949635a1ad890ba0ad5ead497989227d9e65293b0ef5623d7",
"md5": "a47a8b18f1d8181ddb7221d44a6eb96e",
"sha256": "3464f8fa0e87ba4b048d77e88542a7c0125c5b3c7910bc7f146d6f0717816a2f"
},
"downloads": -1,
"filename": "llama_index_packs_node_parser_semantic_chunking-0.4.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a47a8b18f1d8181ddb7221d44a6eb96e",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.9",
"size": 5883,
"upload_time": "2025-07-30T21:33:15",
"upload_time_iso_8601": "2025-07-30T21:33:15.413623Z",
"url": "https://files.pythonhosted.org/packages/65/df/68d58ee1616949635a1ad890ba0ad5ead497989227d9e65293b0ef5623d7/llama_index_packs_node_parser_semantic_chunking-0.4.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "8299e2ece2051d48bcf731e6eada2da11a67698baa3be170a9675dff4b6c357d",
"md5": "22f70ab24a21e30f98f782effb6fda9d",
"sha256": "ca873f451d1aa6b3d0538f3e75ff6486a312945eb1504809a8e6dc8fe6b2ccf6"
},
"downloads": -1,
"filename": "llama_index_packs_node_parser_semantic_chunking-0.4.0.tar.gz",
"has_sig": false,
"md5_digest": "22f70ab24a21e30f98f782effb6fda9d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.9",
"size": 6049,
"upload_time": "2025-07-30T21:33:16",
"upload_time_iso_8601": "2025-07-30T21:33:16.113390Z",
"url": "https://files.pythonhosted.org/packages/82/99/e2ece2051d48bcf731e6eada2da11a67698baa3be170a9675dff4b6c357d/llama_index_packs_node_parser_semantic_chunking-0.4.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-30 21:33:16",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "llama-index-packs-node-parser-semantic-chunking"
}
JSON