| Name | hyper-connections JSON |
| Version |
0.0.14
JSON |
| download |
| home_page | None |
| Summary | Hyper-Connections |
| upload_time | 2024-12-26 18:50:21 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.9 |
| license | MIT License Copyright (c) 2024 Phil Wang Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
artificial intelligence
deep learning
residual
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<img src="./hyper-connections.png" width="450px"></img>
## Hyper Connections
Attempt to make multiple residual streams, proposed in [Hyper-Connections paper](https://arxiv.org/abs/2409.19606) out of Bytedance AI lab, accessible as an easy to use library, as well as for following any new research in this direction.
## Install
```bash
$ pip install hyper-connections
```
## Usage
```python
import torch
from torch import nn
# a single branch layer
branch = nn.Linear(512, 512)
# before
residual = torch.randn(2, 1024, 512)
residual = branch(residual) + residual
# after, say 4 streams in paper
from hyper_connections import HyperConnections
init_hyper_conn, expand_stream, reduce_stream = HyperConnections.get_init_and_expand_reduce_stream_functions(4)
# 1. wrap your branch function
hyper_conn_branch = init_hyper_conn(dim = 512, branch = branch)
# 2. expand to 4 streams, this must be done before your trunk, typically a for-loop with many branch functions
residual = expand_stream(residual)
# 3. forward your residual as usual into the wrapped branch function(s)
residual = hyper_conn_branch(residual)
# 4. reduce 4 streams with a summation, this has to be done after your for-loop trunk. for transformer, unsure whether to do before or after final norm
residual = reduce_stream(residual)
```
Or doing it manually, as in the paper
```python
import torch
from torch import nn
# a single branch layer
branch = nn.Linear(512, 512)
# before
residual = torch.randn(2, 1024, 512)
residual = branch(residual) + residual
# after, say 4 streams in paper
from hyper_connections import HyperConnections
init_hyper_conn, expand_stream, reduce_stream = HyperConnections.get_init_and_expand_reduce_stream_functions(4)
# 1. instantiate hyper connection with correct number of streams (4 in this case) - or use the init function above
hyper_conn = init_hyper_conn(dim = 512)
# 2. expand to 4 streams
residual = expand_stream(residual)
# 3. forward your residual into hyper connection for the branch input + add residual function (learned betas)
branch_input, add_residual = hyper_conn(residual)
branch_output = branch(branch_input)
residual = add_residual(branch_output)
# or you can do it in one line as so -> residual = hyper_conn.decorate_branch(branch)(residual)
# 4. reduce 4 streams with a summation, this has to be done after your for loop trunk
residual = reduce_stream(residual)
```
To compare hyper connections to plain residual without changing the code, just pass `disable = True` when fetching the functions
```python
HyperConnections.get_init_and_expand_reduce_stream_functions(4, disable = True)
```
## Citation
```bibtex
@article{Zhu2024HyperConnections,
title = {Hyper-Connections},
author = {Defa Zhu and Hongzhi Huang and Zihao Huang and Yutao Zeng and Yunyao Mao and Banggu Wu and Qiyang Min and Xun Zhou},
journal = {ArXiv},
year = {2024},
volume = {abs/2409.19606},
url = {https://api.semanticscholar.org/CorpusID:272987528}
}
```
Raw data
{
"_id": null,
"home_page": null,
"name": "hyper-connections",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "artificial intelligence, deep learning, residual",
"author": null,
"author_email": "Phil Wang <lucidrains@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/b6/64/d4e4a25573e5bdc7b650284193d04f5bad28fc37c771c163a2889df2f969/hyper_connections-0.0.14.tar.gz",
"platform": null,
"description": "<img src=\"./hyper-connections.png\" width=\"450px\"></img>\n\n## Hyper Connections\n\nAttempt to make multiple residual streams, proposed in [Hyper-Connections paper](https://arxiv.org/abs/2409.19606) out of Bytedance AI lab, accessible as an easy to use library, as well as for following any new research in this direction.\n\n## Install\n\n```bash\n$ pip install hyper-connections\n```\n\n## Usage\n\n```python\nimport torch\nfrom torch import nn\n\n# a single branch layer\n\nbranch = nn.Linear(512, 512)\n\n# before\n\nresidual = torch.randn(2, 1024, 512)\n\nresidual = branch(residual) + residual\n\n# after, say 4 streams in paper\n\nfrom hyper_connections import HyperConnections\n\ninit_hyper_conn, expand_stream, reduce_stream = HyperConnections.get_init_and_expand_reduce_stream_functions(4)\n\n# 1. wrap your branch function\n\nhyper_conn_branch = init_hyper_conn(dim = 512, branch = branch)\n\n# 2. expand to 4 streams, this must be done before your trunk, typically a for-loop with many branch functions\n\nresidual = expand_stream(residual)\n\n# 3. forward your residual as usual into the wrapped branch function(s)\n\nresidual = hyper_conn_branch(residual) \n\n# 4. reduce 4 streams with a summation, this has to be done after your for-loop trunk. for transformer, unsure whether to do before or after final norm\n\nresidual = reduce_stream(residual)\n```\n\nOr doing it manually, as in the paper\n\n```python\nimport torch\nfrom torch import nn\n\n# a single branch layer\n\nbranch = nn.Linear(512, 512)\n\n# before\n\nresidual = torch.randn(2, 1024, 512)\n\nresidual = branch(residual) + residual\n\n# after, say 4 streams in paper\n\nfrom hyper_connections import HyperConnections\n\ninit_hyper_conn, expand_stream, reduce_stream = HyperConnections.get_init_and_expand_reduce_stream_functions(4)\n\n# 1. instantiate hyper connection with correct number of streams (4 in this case) - or use the init function above\n\nhyper_conn = init_hyper_conn(dim = 512)\n\n# 2. expand to 4 streams\n\nresidual = expand_stream(residual)\n\n# 3. forward your residual into hyper connection for the branch input + add residual function (learned betas)\n\nbranch_input, add_residual = hyper_conn(residual)\n\nbranch_output = branch(branch_input)\n\nresidual = add_residual(branch_output)\n\n# or you can do it in one line as so -> residual = hyper_conn.decorate_branch(branch)(residual)\n\n# 4. reduce 4 streams with a summation, this has to be done after your for loop trunk\n\nresidual = reduce_stream(residual)\n```\n\nTo compare hyper connections to plain residual without changing the code, just pass `disable = True` when fetching the functions\n\n```python\nHyperConnections.get_init_and_expand_reduce_stream_functions(4, disable = True)\n```\n\n## Citation\n\n```bibtex\n@article{Zhu2024HyperConnections,\n title = {Hyper-Connections},\n author = {Defa Zhu and Hongzhi Huang and Zihao Huang and Yutao Zeng and Yunyao Mao and Banggu Wu and Qiyang Min and Xun Zhou},\n journal = {ArXiv},\n year = {2024},\n volume = {abs/2409.19606},\n url = {https://api.semanticscholar.org/CorpusID:272987528}\n}\n```\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) 2024 Phil Wang Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.",
"summary": "Hyper-Connections",
"version": "0.0.14",
"project_urls": {
"Homepage": "https://pypi.org/project/hyper-connections/",
"Repository": "https://github.com/lucidrains/hyper-connections"
},
"split_keywords": [
"artificial intelligence",
" deep learning",
" residual"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "bd8e3879d43f117dc679d2ab691a8a20f7b9361cddc9efd355b6e3114563d926",
"md5": "d3c37cb6b14d6a918c4e16ff52dc6c50",
"sha256": "d976738165ded852496ac0de52e9ed9da371afd9fbcf260150ea459e2b9d1cfd"
},
"downloads": -1,
"filename": "hyper_connections-0.0.14-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d3c37cb6b14d6a918c4e16ff52dc6c50",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 8033,
"upload_time": "2024-12-26T18:50:19",
"upload_time_iso_8601": "2024-12-26T18:50:19.709813Z",
"url": "https://files.pythonhosted.org/packages/bd/8e/3879d43f117dc679d2ab691a8a20f7b9361cddc9efd355b6e3114563d926/hyper_connections-0.0.14-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b664d4e4a25573e5bdc7b650284193d04f5bad28fc37c771c163a2889df2f969",
"md5": "421d9dddad39e2c1316ff02a14a23313",
"sha256": "28943bb67365a2ddb584d51659833eafd4ba9c3e15b22944727cb0965471b480"
},
"downloads": -1,
"filename": "hyper_connections-0.0.14.tar.gz",
"has_sig": false,
"md5_digest": "421d9dddad39e2c1316ff02a14a23313",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 284470,
"upload_time": "2024-12-26T18:50:21",
"upload_time_iso_8601": "2024-12-26T18:50:21.219775Z",
"url": "https://files.pythonhosted.org/packages/b6/64/d4e4a25573e5bdc7b650284193d04f5bad28fc37c771c163a2889df2f969/hyper_connections-0.0.14.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-26 18:50:21",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "lucidrains",
"github_project": "hyper-connections",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "hyper-connections"
}
