| Name | hyper-connections JSON |
| Version |
0.1.9
JSON |
| download |
| home_page | None |
| Summary | Hyper-Connections |
| upload_time | 2025-01-21 18:52:35 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.9 |
| license | MIT License
Copyright (c) 2024 Phil Wang
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE. |
| keywords |
artificial intelligence
deep learning
residual
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<img src="./hyper-connections.png" width="450px"></img>
## Hyper Connections
Attempt to make multiple residual streams, proposed in [Hyper-Connections paper](https://arxiv.org/abs/2409.19606) out of Bytedance AI lab, accessible as an easy to use library, as well as for following any new research in this direction.
## Install
```bash
$ pip install hyper-connections
```
## Usage
```python
import torch
from torch import nn
# a single branch layer
branch = nn.Linear(512, 512)
# before
residual = torch.randn(2, 1024, 512)
residual = branch(residual) + residual
# after, say 4 streams in paper
from hyper_connections import get_init_and_expand_reduce_stream_functions
init_hyper_conn, expand_stream, reduce_stream = get_init_and_expand_reduce_stream_functions(4)
# 1. wrap your branch function
hyper_conn_branch = init_hyper_conn(dim = 512, branch = branch)
# 2. expand to 4 streams, this must be done before your trunk, typically a for-loop with many branch functions
residual = expand_stream(residual)
# 3. forward your residual as usual into the wrapped branch function(s)
residual = hyper_conn_branch(residual)
# 4. reduce 4 streams with a summation, this has to be done after your for-loop trunk. for transformer, unsure whether to do before or after final norm
residual = reduce_stream(residual)
```
Or doing it manually, as in the paper
```python
import torch
from torch import nn
# a single branch layer
branch = nn.Linear(512, 512)
# before
residual = torch.randn(2, 1024, 512)
residual = branch(residual) + residual
# after, say 4 streams in paper
from hyper_connections import get_init_and_expand_reduce_stream_functions
init_hyper_conn, expand_stream, reduce_stream = get_init_and_expand_reduce_stream_functions(4)
# 1. instantiate hyper connection with correct number of streams (4 in this case) - or use the init function above
hyper_conn = init_hyper_conn(dim = 512)
# 2. expand to 4 streams
residual = expand_stream(residual)
# 3. forward your residual into hyper connection for the branch input + add residual function (learned betas)
branch_input, add_residual = hyper_conn(residual)
branch_output = branch(branch_input)
residual = add_residual(branch_output)
# or you can do it in one line as so -> residual = hyper_conn.decorate_branch(branch)(residual)
# 4. reduce 4 streams with a summation, this has to be done after your for loop trunk
residual = reduce_stream(residual)
```
To compare hyper connections to plain residual without changing the code, just pass `disable = True` when fetching the functions
```python
get_init_and_expand_reduce_stream_functions(4, disable = True)
```
## Citation
```bibtex
@article{Zhu2024HyperConnections,
title = {Hyper-Connections},
author = {Defa Zhu and Hongzhi Huang and Zihao Huang and Yutao Zeng and Yunyao Mao and Banggu Wu and Qiyang Min and Xun Zhou},
journal = {ArXiv},
year = {2024},
volume = {abs/2409.19606},
url = {https://api.semanticscholar.org/CorpusID:272987528}
}
```
```bibtex
@misc{Rubin2024,
author = {Ohad Rubin},
url = {https://medium.com/@ohadrubin/exploring-weight-decay-in-layer-normalization-challenges-and-a-reparameterization-solution-ad4d12c24950}
}
```
Raw data
{
"_id": null,
"home_page": null,
"name": "hyper-connections",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "artificial intelligence, deep learning, residual",
"author": null,
"author_email": "Phil Wang <lucidrains@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/14/53/c1d5584ebbddb6d5baa258d0234c9dbaa5a331bb8d2ff5e77cc56d786fef/hyper_connections-0.1.9.tar.gz",
"platform": null,
"description": "<img src=\"./hyper-connections.png\" width=\"450px\"></img>\n\n## Hyper Connections\n\nAttempt to make multiple residual streams, proposed in [Hyper-Connections paper](https://arxiv.org/abs/2409.19606) out of Bytedance AI lab, accessible as an easy to use library, as well as for following any new research in this direction.\n\n## Install\n\n```bash\n$ pip install hyper-connections\n```\n\n## Usage\n\n```python\nimport torch\nfrom torch import nn\n\n# a single branch layer\n\nbranch = nn.Linear(512, 512)\n\n# before\n\nresidual = torch.randn(2, 1024, 512)\n\nresidual = branch(residual) + residual\n\n# after, say 4 streams in paper\n\nfrom hyper_connections import get_init_and_expand_reduce_stream_functions\n\ninit_hyper_conn, expand_stream, reduce_stream = get_init_and_expand_reduce_stream_functions(4)\n\n# 1. wrap your branch function\n\nhyper_conn_branch = init_hyper_conn(dim = 512, branch = branch)\n\n# 2. expand to 4 streams, this must be done before your trunk, typically a for-loop with many branch functions\n\nresidual = expand_stream(residual)\n\n# 3. forward your residual as usual into the wrapped branch function(s)\n\nresidual = hyper_conn_branch(residual) \n\n# 4. reduce 4 streams with a summation, this has to be done after your for-loop trunk. for transformer, unsure whether to do before or after final norm\n\nresidual = reduce_stream(residual)\n```\n\nOr doing it manually, as in the paper\n\n```python\nimport torch\nfrom torch import nn\n\n# a single branch layer\n\nbranch = nn.Linear(512, 512)\n\n# before\n\nresidual = torch.randn(2, 1024, 512)\n\nresidual = branch(residual) + residual\n\n# after, say 4 streams in paper\n\nfrom hyper_connections import get_init_and_expand_reduce_stream_functions\n\ninit_hyper_conn, expand_stream, reduce_stream = get_init_and_expand_reduce_stream_functions(4)\n\n# 1. instantiate hyper connection with correct number of streams (4 in this case) - or use the init function above\n\nhyper_conn = init_hyper_conn(dim = 512)\n\n# 2. expand to 4 streams\n\nresidual = expand_stream(residual)\n\n# 3. forward your residual into hyper connection for the branch input + add residual function (learned betas)\n\nbranch_input, add_residual = hyper_conn(residual)\n\nbranch_output = branch(branch_input)\n\nresidual = add_residual(branch_output)\n\n# or you can do it in one line as so -> residual = hyper_conn.decorate_branch(branch)(residual)\n\n# 4. reduce 4 streams with a summation, this has to be done after your for loop trunk\n\nresidual = reduce_stream(residual)\n```\n\nTo compare hyper connections to plain residual without changing the code, just pass `disable = True` when fetching the functions\n\n```python\nget_init_and_expand_reduce_stream_functions(4, disable = True)\n```\n\n## Citation\n\n```bibtex\n@article{Zhu2024HyperConnections,\n title = {Hyper-Connections},\n author = {Defa Zhu and Hongzhi Huang and Zihao Huang and Yutao Zeng and Yunyao Mao and Banggu Wu and Qiyang Min and Xun Zhou},\n journal = {ArXiv},\n year = {2024},\n volume = {abs/2409.19606},\n url = {https://api.semanticscholar.org/CorpusID:272987528}\n}\n```\n\n```bibtex\n@misc{Rubin2024,\n author = {Ohad Rubin},\n url = {https://medium.com/@ohadrubin/exploring-weight-decay-in-layer-normalization-challenges-and-a-reparameterization-solution-ad4d12c24950}\n}\n```\n",
"bugtrack_url": null,
"license": "MIT License\n \n Copyright (c) 2024 Phil Wang\n \n Permission is hereby granted, free of charge, to any person obtaining a copy\n of this software and associated documentation files (the \"Software\"), to deal\n in the Software without restriction, including without limitation the rights\n to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n copies of the Software, and to permit persons to whom the Software is\n furnished to do so, subject to the following conditions:\n \n The above copyright notice and this permission notice shall be included in all\n copies or substantial portions of the Software.\n \n THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n SOFTWARE.",
"summary": "Hyper-Connections",
"version": "0.1.9",
"project_urls": {
"Homepage": "https://pypi.org/project/hyper-connections/",
"Repository": "https://github.com/lucidrains/hyper-connections"
},
"split_keywords": [
"artificial intelligence",
" deep learning",
" residual"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "78a117871ebc100d522879b021c95472e86ff98548bfd18433c4851b599150bb",
"md5": "b236c7c2492bbf147941eb052282e94a",
"sha256": "b129c27e2c1d752c78c2ca54c0dd2d47de39808ee6de46d31c8f6ea9573913b3"
},
"downloads": -1,
"filename": "hyper_connections-0.1.9-py3-none-any.whl",
"has_sig": false,
"md5_digest": "b236c7c2492bbf147941eb052282e94a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 14404,
"upload_time": "2025-01-21T18:52:33",
"upload_time_iso_8601": "2025-01-21T18:52:33.315434Z",
"url": "https://files.pythonhosted.org/packages/78/a1/17871ebc100d522879b021c95472e86ff98548bfd18433c4851b599150bb/hyper_connections-0.1.9-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "1453c1d5584ebbddb6d5baa258d0234c9dbaa5a331bb8d2ff5e77cc56d786fef",
"md5": "5a773f9b1ca017fdab2978c1389d59bc",
"sha256": "6b8002b586e019b684c2ddc33962f0dd71fb8366cf51538dd0dc8f34610aeb1d"
},
"downloads": -1,
"filename": "hyper_connections-0.1.9.tar.gz",
"has_sig": false,
"md5_digest": "5a773f9b1ca017fdab2978c1389d59bc",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 288089,
"upload_time": "2025-01-21T18:52:35",
"upload_time_iso_8601": "2025-01-21T18:52:35.393527Z",
"url": "https://files.pythonhosted.org/packages/14/53/c1d5584ebbddb6d5baa258d0234c9dbaa5a331bb8d2ff5e77cc56d786fef/hyper_connections-0.1.9.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-21 18:52:35",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "lucidrains",
"github_project": "hyper-connections",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "hyper-connections"
}
