```markdown
# betterhtmlchunking
A Python library for intelligently chunking HTML documents into structured, size-limited segments based on DOM tree analysis.
## Overview
This library processes HTML content to split it into semantically coherent chunks while respecting specified size constraints. It analyzes the DOM structure to identify optimal split points, preserving contextual information and document hierarchy.
## Key Features
- Custom DOM tree representation.
- Configurable chunk size limits (counting by text or HTML length).
- Intelligent region-of-interest detection.
- Dual output formats: HTML and plain text chunks.
- Preservation of structure relationships.
- Customizable tag filtering.
## Installation
```bash
pip install betterhtmlchunking
```
### Dependencies
- Python 3.12+
- attrs
- treelib
- beautifulsoup4
- parsel-text
- lxml
- attrs-strict
## Usage
### Basic Example
```python
from betterhtmlchunking import DomRepresentation
html_content = """
<html>
<body>
<div id="content">
<h1>Document Title</h1>
<p>First paragraph...</p>
<p>Second paragraph...</p>
</div>
</body>
</html>
"""
# Create document representation with 500 character chunks
doc = DomRepresentation(
MAX_NODE_REPR_LENGTH=500,
website_code=html_content,
repr_length_compared_by="html_length"
)
# Access HTML chunks
html_chunks = doc.render_system.html_render_roi
for chunk_id, chunk in html_chunks.items():
print(f"Chunk {chunk_id}:\n{chunk}\n{'='*50}")
# Access text chunks
text_chunks = doc.render_system.text_render_roi
for chunk_id, chunk in text_chunks.items():
print(f"Chunk {chunk_id}:\n{chunk}\n{'='*50}")
```
## Configuration
### Key Parameters
- `MAX_NODE_REPR_LENGTH`: Maximum allowed length for each chunk (in characters)
- `repr_length_compared_by`: Length calculation method:
- `html_length`: HTML source length
- `text_length`: Rendered text length
- `website_code`: Input HTML content
### Advanced Features
```python
# Access the DOM tree structure
tree = doc.tree_representation.tree
# Get node metadata
for node in tree.all_nodes():
print(f"XPath: {node.identifier}")
print(f"Text length: {node.data.text_length}")
print(f"HTML length: {node.data.html_length}")
# Custom tag filtering (before processing)
from betterhtmlchunking.tree_representation import DOMTreeRepresentation
from betterhtmlchunking.utils import remove_unwanted_tags
# Example usage of remove_unwanted_tags:
tree_rep = DOMTreeRepresentation(website_code=html_content)
filtered_rep = remove_unwanted_tags(tree_rep)
```
## How It Works
1. **DOM Parsing**
- Builds a tree representation of the HTML document.
- Calculates metadata (text length, HTML length) for each node.
2. **Region Detection**
- Uses **Breadth First Search (BFS)** to traverse the DOM tree in a level-order fashion, ensuring that each node is processed systematically.
- Combines nodes until the specified size limit is reached.
- Preserves parent-child relationships to maintain contextual integrity.
3. **Chunk Generation**
- Creates HTML chunks with original markup.
- Generates parallel text-only chunks.
- Maintains chunk order based on document structure.
## Comparison to popular Chunking Techniques
The actual practice (Feb. 2025) is to use **plain-text** or **token-based** chunking strategies, primarily aimed at keeping prompts within certain token limits for large language models. This approach is ideal for quick semantic retrieval or QA tasks on *unstructured* text.
By contrast, **betterhtmlchunking** preserves the **HTML DOM structure**, calculating chunk boundaries based on each node’s text or HTML length. This approach is especially useful when you want to:
- Retain or leverage the **hierarchical relationships** in the HTML (e.g., headings, nested divs)
- Filter out undesired tags or sections (like `<script>` or `<style>`)
- Pinpoint exactly where each chunk originated in the document (via positional XPaths)
You can even combine the two techniques if you need both **structured extraction** (via betterhtmlchunking) and **LLM-friendly text chunking** (via LangChain) for advanced tasks such as summarization, semantic search, or large-scale QA pipelines.
## License
MIT License
## Contributing
Feel free to open issues or submit pull requests if you have suggestions or improvements.
Raw data
{
"_id": null,
"home_page": null,
"name": "betterhtmlchunking",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "html, chunking, scraping, dom, roi, content extraction, web-scraping",
"author": null,
"author_email": "\"Carlos A. Planch\u00f3n\" <carlosandresplanchonprestes@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/6b/2e/822e28aa0f95abf2033a0ede4fc31633886a5c994651cf0536005fdad389/betterhtmlchunking-0.9.1.tar.gz",
"platform": null,
"description": "```markdown\n# betterhtmlchunking\n\nA Python library for intelligently chunking HTML documents into structured, size-limited segments based on DOM tree analysis.\n\n## Overview\n\nThis library processes HTML content to split it into semantically coherent chunks while respecting specified size constraints. It analyzes the DOM structure to identify optimal split points, preserving contextual information and document hierarchy.\n\n## Key Features\n\n- Custom DOM tree representation. \n- Configurable chunk size limits (counting by text or HTML length).\n- Intelligent region-of-interest detection.\n- Dual output formats: HTML and plain text chunks. \n- Preservation of structure relationships.\n- Customizable tag filtering.\n\n## Installation\n\n```bash\npip install betterhtmlchunking\n```\n\n### Dependencies\n- Python 3.12+\n- attrs\n- treelib\n- beautifulsoup4\n- parsel-text\n- lxml\n- attrs-strict\n\n## Usage\n\n### Basic Example\n\n```python\nfrom betterhtmlchunking import DomRepresentation\n\nhtml_content = \"\"\"\n<html>\n <body>\n <div id=\"content\">\n <h1>Document Title</h1>\n <p>First paragraph...</p>\n <p>Second paragraph...</p>\n </div>\n </body>\n</html>\n\"\"\"\n\n# Create document representation with 500 character chunks\ndoc = DomRepresentation(\n MAX_NODE_REPR_LENGTH=500,\n website_code=html_content,\n repr_length_compared_by=\"html_length\"\n)\n\n# Access HTML chunks\nhtml_chunks = doc.render_system.html_render_roi\nfor chunk_id, chunk in html_chunks.items():\n print(f\"Chunk {chunk_id}:\\n{chunk}\\n{'='*50}\")\n\n# Access text chunks\ntext_chunks = doc.render_system.text_render_roi\nfor chunk_id, chunk in text_chunks.items():\n print(f\"Chunk {chunk_id}:\\n{chunk}\\n{'='*50}\")\n```\n\n## Configuration\n\n### Key Parameters\n- `MAX_NODE_REPR_LENGTH`: Maximum allowed length for each chunk (in characters)\n- `repr_length_compared_by`: Length calculation method:\n - `html_length`: HTML source length\n - `text_length`: Rendered text length\n- `website_code`: Input HTML content\n\n### Advanced Features\n```python\n# Access the DOM tree structure\ntree = doc.tree_representation.tree\n\n# Get node metadata\nfor node in tree.all_nodes():\n print(f\"XPath: {node.identifier}\")\n print(f\"Text length: {node.data.text_length}\")\n print(f\"HTML length: {node.data.html_length}\")\n\n# Custom tag filtering (before processing)\nfrom betterhtmlchunking.tree_representation import DOMTreeRepresentation\nfrom betterhtmlchunking.utils import remove_unwanted_tags\n\n# Example usage of remove_unwanted_tags:\ntree_rep = DOMTreeRepresentation(website_code=html_content)\nfiltered_rep = remove_unwanted_tags(tree_rep)\n```\n\n## How It Works\n\n1. **DOM Parsing** \n - Builds a tree representation of the HTML document.\n - Calculates metadata (text length, HTML length) for each node.\n\n2. **Region Detection** \n - Uses **Breadth First Search (BFS)** to traverse the DOM tree in a level-order fashion, ensuring that each node is processed systematically.\n - Combines nodes until the specified size limit is reached.\n - Preserves parent-child relationships to maintain contextual integrity.\n\n3. **Chunk Generation** \n - Creates HTML chunks with original markup.\n - Generates parallel text-only chunks.\n - Maintains chunk order based on document structure.\n\n## Comparison to popular Chunking Techniques\n\nThe actual practice (Feb. 2025) is to use **plain-text** or **token-based** chunking strategies, primarily aimed at keeping prompts within certain token limits for large language models. This approach is ideal for quick semantic retrieval or QA tasks on *unstructured* text.\n\nBy contrast, **betterhtmlchunking** preserves the **HTML DOM structure**, calculating chunk boundaries based on each node\u2019s text or HTML length. This approach is especially useful when you want to:\n- Retain or leverage the **hierarchical relationships** in the HTML (e.g., headings, nested divs) \n- Filter out undesired tags or sections (like `<script>` or `<style>`) \n- Pinpoint exactly where each chunk originated in the document (via positional XPaths)\n\nYou can even combine the two techniques if you need both **structured extraction** (via betterhtmlchunking) and **LLM-friendly text chunking** (via LangChain) for advanced tasks such as summarization, semantic search, or large-scale QA pipelines.\n\n## License\n\nMIT License\n\n## Contributing\nFeel free to open issues or submit pull requests if you have suggestions or improvements.\n",
"bugtrack_url": null,
"license": "MIT License",
"summary": "A Python library for intelligent HTML segmentation and ROI extraction. It builds a DOM tree from raw HTML and extracts content-rich regions for efficient web scraping and analysis.",
"version": "0.9.1",
"project_urls": {
"repository": "https://github.com/carlosplanchon/betterhtmlchunking.git"
},
"split_keywords": [
"html",
" chunking",
" scraping",
" dom",
" roi",
" content extraction",
" web-scraping"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "367446ebed6b96588b0073d27c09342e50cd884c5a76744025f01de3caf646d7",
"md5": "309ab3534aa4ea24cd51e3a595337698",
"sha256": "dec63a03f2b31d6d567ac31ff2506a9062ddfbde02a79116be41df8a46e7c69f"
},
"downloads": -1,
"filename": "betterhtmlchunking-0.9.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "309ab3534aa4ea24cd51e3a595337698",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 11338,
"upload_time": "2025-02-14T08:21:27",
"upload_time_iso_8601": "2025-02-14T08:21:27.117527Z",
"url": "https://files.pythonhosted.org/packages/36/74/46ebed6b96588b0073d27c09342e50cd884c5a76744025f01de3caf646d7/betterhtmlchunking-0.9.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "6b2e822e28aa0f95abf2033a0ede4fc31633886a5c994651cf0536005fdad389",
"md5": "c4633c664578ab006a91ec8538020473",
"sha256": "7f152dbfde5a448fc7c0f9c2e1d91cd059c18277e3acd421e42eade97a4de4c3"
},
"downloads": -1,
"filename": "betterhtmlchunking-0.9.1.tar.gz",
"has_sig": false,
"md5_digest": "c4633c664578ab006a91ec8538020473",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 11435,
"upload_time": "2025-02-14T08:21:28",
"upload_time_iso_8601": "2025-02-14T08:21:28.950555Z",
"url": "https://files.pythonhosted.org/packages/6b/2e/822e28aa0f95abf2033a0ede4fc31633886a5c994651cf0536005fdad389/betterhtmlchunking-0.9.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-02-14 08:21:28",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "carlosplanchon",
"github_project": "betterhtmlchunking",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "betterhtmlchunking"
}
JSON
