<img src="./h-net.png" width="450px"></img>
## H-Net Dynamic Chunking
Implementation of the dynamic chunking mechanism in [H-net](https://arxiv.org/abs/2507.07955) by Hwang et al. of Carnegie Mellon
## Install
```shell
$ pip install h-net-dynamic-chunking
```
## Usage
```python
import torch
from h_net_dynamic_chunking import DynamicSequenceChunker
downsampler = DynamicSequenceChunker(512)
tokens = torch.randn(3, 1024, 512).requires_grad_()
downsampled, upsample_fn, *_ = downsampler(tokens)
assert upsample_fn(downsampled).shape == tokens.shape
```
3 layers hierarchy
```python
import torch
from h_net_dynamic_chunking import DynamicSequenceChunker
downsampler1 = DynamicSequenceChunker(512)
downsampler2 = DynamicSequenceChunker(512)
downsampler3 = DynamicSequenceChunker(512)
tokens = torch.randn(3, 1024, 512).requires_grad_()
downsampled1, upsample_fn1, aux_loss1 = downsampler1(tokens)
# hierarchical network 1 ...
downsampled2, upsample_fn2, aux_loss2 = downsampler2(downsampled1)
# hierarchical network 2 ...
downsampled3, upsample_fn3, aux_loss3 = downsampler3(downsampled2)
# inner most network
# reconstituting
assert upsample_fn1(upsample_fn2(upsample_fn3(downsampled3))).shape == tokens.shape
```
HNet wrapper
```python
import torch
from torch import nn
from h_net_dynamic_chunking.h_net import HNet
# 3 hierarchies, from 512 -> 1024 -> 2048 inner
net = HNet(
nn.Identity(),
HNet(
nn.Identity(),
HNet(
nn.Identity(),
nn.Identity(),
nn.Identity(),
dim = 2048
),
nn.Identity(),
dim = 1024,
dim_inner = 2048
),
nn.Identity(),
dim = 512,
dim_inner = 1024,
)
tokens = torch.randn(1, 1024, 512)
out, aux_loss = net(tokens) # (1, 1024, 512), (1,)
```
## Example
Enwik8 with 2 hierarchies
```shell
$ pip install '.[examples]'
```
Then
```shell
$ python train.py
```
## Citations
```bibtex
@misc{hwang2025dynamicchunkingendtoendhierarchical,
title = {Dynamic Chunking for End-to-End Hierarchical Sequence Modeling},
author = {Sukjun Hwang and Brandon Wang and Albert Gu},
year = {2025},
eprint = {2507.07955},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2507.07955},
}
```
Raw data
{
"_id": null,
"home_page": null,
"name": "h-net-dynamic-chunking",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "artificial intelligence, deep learning, learned chunking, learned tokenization",
"author": null,
"author_email": "Phil Wang <lucidrains@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/28/73/688eca1369a5fb9640ead8ed27a3448a21ab096ae1f022ea1afd05f6681f/h_net_dynamic_chunking-0.3.0.tar.gz",
"platform": null,
"description": "<img src=\"./h-net.png\" width=\"450px\"></img>\n\n## H-Net Dynamic Chunking\n\nImplementation of the dynamic chunking mechanism in [H-net](https://arxiv.org/abs/2507.07955) by Hwang et al. of Carnegie Mellon\n\n## Install\n\n```shell\n$ pip install h-net-dynamic-chunking\n```\n\n## Usage\n\n```python\nimport torch\nfrom h_net_dynamic_chunking import DynamicSequenceChunker\n\ndownsampler = DynamicSequenceChunker(512)\n\ntokens = torch.randn(3, 1024, 512).requires_grad_()\n\ndownsampled, upsample_fn, *_ = downsampler(tokens)\n\nassert upsample_fn(downsampled).shape == tokens.shape\n```\n\n3 layers hierarchy\n\n```python\nimport torch\nfrom h_net_dynamic_chunking import DynamicSequenceChunker\n\ndownsampler1 = DynamicSequenceChunker(512)\ndownsampler2 = DynamicSequenceChunker(512)\ndownsampler3 = DynamicSequenceChunker(512)\n\ntokens = torch.randn(3, 1024, 512).requires_grad_()\n\ndownsampled1, upsample_fn1, aux_loss1 = downsampler1(tokens)\n\n# hierarchical network 1 ...\n\ndownsampled2, upsample_fn2, aux_loss2 = downsampler2(downsampled1)\n\n# hierarchical network 2 ...\n\ndownsampled3, upsample_fn3, aux_loss3 = downsampler3(downsampled2)\n\n# inner most network\n\n# reconstituting\n\nassert upsample_fn1(upsample_fn2(upsample_fn3(downsampled3))).shape == tokens.shape\n```\n\nHNet wrapper\n\n```python\nimport torch\nfrom torch import nn\nfrom h_net_dynamic_chunking.h_net import HNet\n\n# 3 hierarchies, from 512 -> 1024 -> 2048 inner\n\nnet = HNet(\n nn.Identity(),\n HNet(\n nn.Identity(),\n HNet(\n nn.Identity(),\n nn.Identity(),\n nn.Identity(),\n dim = 2048\n ),\n nn.Identity(),\n dim = 1024,\n dim_inner = 2048\n ),\n nn.Identity(),\n dim = 512,\n dim_inner = 1024,\n)\n\ntokens = torch.randn(1, 1024, 512)\n\nout, aux_loss = net(tokens) # (1, 1024, 512), (1,)\n```\n\n## Example\n\nEnwik8 with 2 hierarchies\n\n```shell\n$ pip install '.[examples]'\n```\n\nThen\n\n```shell\n$ python train.py\n```\n\n## Citations\n\n```bibtex\n@misc{hwang2025dynamicchunkingendtoendhierarchical,\n title = {Dynamic Chunking for End-to-End Hierarchical Sequence Modeling},\n author = {Sukjun Hwang and Brandon Wang and Albert Gu},\n year = {2025},\n eprint = {2507.07955},\n archivePrefix = {arXiv},\n primaryClass = {cs.LG},\n url = {https://arxiv.org/abs/2507.07955},\n}\n```\n",
"bugtrack_url": null,
"license": null,
"summary": "H-Net Dynamic Chunking Modules",
"version": "0.3.0",
"project_urls": {
"Homepage": "https://pypi.org/project/h-net-dynamic-chunking/",
"Repository": "https://github.com/lucidrains/h-net-dynamic-chunking"
},
"split_keywords": [
"artificial intelligence",
" deep learning",
" learned chunking",
" learned tokenization"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "caa2906379708490a5596f22bc7ed47c1ebba19bdd19e428b954fc45653c0e85",
"md5": "2e392433ebbbadafa89a804fd30d4289",
"sha256": "e66246e13fb3251fc6d3e7922fb254c8d4d33e5fe06c9712fa33ac9d0426623f"
},
"downloads": -1,
"filename": "h_net_dynamic_chunking-0.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "2e392433ebbbadafa89a804fd30d4289",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 11307,
"upload_time": "2025-08-15T14:32:06",
"upload_time_iso_8601": "2025-08-15T14:32:06.904780Z",
"url": "https://files.pythonhosted.org/packages/ca/a2/906379708490a5596f22bc7ed47c1ebba19bdd19e428b954fc45653c0e85/h_net_dynamic_chunking-0.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "2873688eca1369a5fb9640ead8ed27a3448a21ab096ae1f022ea1afd05f6681f",
"md5": "b078ee7490b89befcb6b77a5cf44189a",
"sha256": "2b70bd8edfbd706a1a614bf35ea85c5627eaf1e914b6357fd475975010e0bde6"
},
"downloads": -1,
"filename": "h_net_dynamic_chunking-0.3.0.tar.gz",
"has_sig": false,
"md5_digest": "b078ee7490b89befcb6b77a5cf44189a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 9428,
"upload_time": "2025-08-15T14:32:08",
"upload_time_iso_8601": "2025-08-15T14:32:08.454441Z",
"url": "https://files.pythonhosted.org/packages/28/73/688eca1369a5fb9640ead8ed27a3448a21ab096ae1f022ea1afd05f6681f/h_net_dynamic_chunking-0.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-15 14:32:08",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "lucidrains",
"github_project": "h-net-dynamic-chunking",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "h-net-dynamic-chunking"
}
JSON
