aft-pt


Nameaft-pt JSON
Version 0.1.0 PyPI version JSON
download
home_pagehttps://github.com/shendezhou/aft-pytorch
SummaryAttention Free Transformer - Pytorch
upload_time2021-06-16 11:58:57
maintainer
docs_urlNone
authorShen Dezhou
requires_python
licenseMIT
keywords artificial intelligence deep learning attention free transformer self-attention transformer natural language processing
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # aft-pytorch
Unofficial PyTorch implementation of **Attention Free Transformer**'s layers by [Zhai](https://twitter.com/zhaisf?lang=en), et al. [[abs](https://openreview.net/forum?id=pW--cu2FCHY), [pdf](https://arxiv.org/pdf/2105.14103.pdf)] from Apple Inc.

<img src="https://github.com/ShenDezhou/aft-pytorch/blob/master/pic.png" width=650>

## Installation
You can install `aft_pt` via `pip`:

```bash
pip install aft_pt
```

## Usage
You can import the **AFT-Full** or **AFT-Simple** layer (as described in the paper) from the package like so:

### `AFTFull`
```python
from aft_pt import AFTFull

layer = AFTFull(
    max_seqlen=20,
    dim=512,
    hidden_dim=64
)

# a batch of sequences with 10 timesteps of length 512 each
x = torch.rand(32, 10, 512)
y = layer(x) # [32, 10, 512]
```

### `AFTSimple`
```python
from aft_pt import AFTSimple

layer = AFTSimple(
    max_seqlen=20,
    dim=512,
    hidden_dim=64
)

# a batch of sequences with 10 timesteps of length 512 each
x = torch.rand(32, 10, 512)
y = layer(x) # [32, 10, 512]
```
### `AFTLocal`
```python
from aft_pt import AFTLocal

layer = AFTLocal(
    max_seqlen=20,
    dim=512,
    hidden_dim=64
)

# a batch of sequences with 10 timesteps of length 512 each
x = torch.rand(32, 10, 512)
y = layer(x) # [32, 10, 512]
```

> This layer wrapper is a 'plug-and-play' with your existing networks / Transformers. You can swap out the Self-Attention layer with the available layers in this package with minimal changes.

## TODO
- [ ] Add full AFT architecture
- [ ] Add variants like, `AFTConv`
- [ ] Benchmark using Karpathy's [minGPT](https://github.com/karpathy/minGPT)

## Contributing
If you like this repo, please leave a star! If there are any amends or suggestions, feel free to raise a PR/issue.

## Credits
```
@misc{attention-free-transformer,
title = {An Attention Free Transformer},
author = {Shuangfei Zhai and Walter Talbott and Nitish Srivastava and Chen Huang and Hanlin Goh and Ruixiang Zhang and Josh Susskind},
year = {2021},
URL = {https://arxiv.org/pdf/2105.14103.pdf}
}
```

## License
[MIT](https://github.com/ShenDezhou/aft-pytorch/blob/main/LICENSE)



            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/shendezhou/aft-pytorch",
    "name": "aft-pt",
    "maintainer": "",
    "docs_url": null,
    "requires_python": "",
    "maintainer_email": "",
    "keywords": "artificial intelligence,deep learning,attention free transformer,self-attention,transformer,natural language processing",
    "author": "Shen Dezhou",
    "author_email": "tsinghua9boy@sina.com",
    "download_url": "https://files.pythonhosted.org/packages/31/b9/355207d29513dd1f09ed4a826591bfa80804150262965353b4661deff87d/aft_pt-0.1.0.tar.gz",
    "platform": "",
    "description": "# aft-pytorch\nUnofficial PyTorch implementation of **Attention Free Transformer**'s layers by [Zhai](https://twitter.com/zhaisf?lang=en), et al. [[abs](https://openreview.net/forum?id=pW--cu2FCHY), [pdf](https://arxiv.org/pdf/2105.14103.pdf)] from Apple Inc.\n\n<img src=\"https://github.com/ShenDezhou/aft-pytorch/blob/master/pic.png\" width=650>\n\n## Installation\nYou can install `aft_pt` via `pip`:\n\n```bash\npip install aft_pt\n```\n\n## Usage\nYou can import the **AFT-Full** or **AFT-Simple** layer (as described in the paper) from the package like so:\n\n### `AFTFull`\n```python\nfrom aft_pt import AFTFull\n\nlayer = AFTFull(\n    max_seqlen=20,\n    dim=512,\n    hidden_dim=64\n)\n\n# a batch of sequences with 10 timesteps of length 512 each\nx = torch.rand(32, 10, 512)\ny = layer(x) # [32, 10, 512]\n```\n\n### `AFTSimple`\n```python\nfrom aft_pt import AFTSimple\n\nlayer = AFTSimple(\n    max_seqlen=20,\n    dim=512,\n    hidden_dim=64\n)\n\n# a batch of sequences with 10 timesteps of length 512 each\nx = torch.rand(32, 10, 512)\ny = layer(x) # [32, 10, 512]\n```\n### `AFTLocal`\n```python\nfrom aft_pt import AFTLocal\n\nlayer = AFTLocal(\n    max_seqlen=20,\n    dim=512,\n    hidden_dim=64\n)\n\n# a batch of sequences with 10 timesteps of length 512 each\nx = torch.rand(32, 10, 512)\ny = layer(x) # [32, 10, 512]\n```\n\n> This layer wrapper is a 'plug-and-play' with your existing networks / Transformers. You can swap out the Self-Attention layer with the available layers in this package with minimal changes.\n\n## TODO\n- [ ] Add full AFT architecture\n- [ ] Add variants like, `AFTConv`\n- [ ] Benchmark using Karpathy's [minGPT](https://github.com/karpathy/minGPT)\n\n## Contributing\nIf you like this repo, please leave a star! If there are any amends or suggestions, feel free to raise a PR/issue.\n\n## Credits\n```\n@misc{attention-free-transformer,\ntitle = {An Attention Free Transformer},\nauthor = {Shuangfei Zhai and Walter Talbott and Nitish Srivastava and Chen Huang and Hanlin Goh and Ruixiang Zhang and Josh Susskind},\nyear = {2021},\nURL = {https://arxiv.org/pdf/2105.14103.pdf}\n}\n```\n\n## License\n[MIT](https://github.com/ShenDezhou/aft-pytorch/blob/main/LICENSE)\n\n\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "Attention Free Transformer - Pytorch",
    "version": "0.1.0",
    "split_keywords": [
        "artificial intelligence",
        "deep learning",
        "attention free transformer",
        "self-attention",
        "transformer",
        "natural language processing"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "0110ce8cc61d8a927fef0e240dbf2bd2",
                "sha256": "8faaefba090646127c267cc8f1a9c66a3d472e93988f0f5bb20782e8d66fd55f"
            },
            "downloads": -1,
            "filename": "aft_pt-0.1.0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "0110ce8cc61d8a927fef0e240dbf2bd2",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 4875,
            "upload_time": "2021-06-16T11:58:37",
            "upload_time_iso_8601": "2021-06-16T11:58:37.621747Z",
            "url": "https://files.pythonhosted.org/packages/cd/b1/3c563d2bf167a64adda0e03ef57c88d15e90aeff31e0f46204a1ab069ed1/aft_pt-0.1.0-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "8c7624943118f4f649e82cafbb22c6b7",
                "sha256": "2fa1980c5358824efae3af494943931ba536def99ff8bef416bcd409e07c08cb"
            },
            "downloads": -1,
            "filename": "aft_pt-0.1.0.tar.gz",
            "has_sig": false,
            "md5_digest": "8c7624943118f4f649e82cafbb22c6b7",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 4627,
            "upload_time": "2021-06-16T11:58:57",
            "upload_time_iso_8601": "2021-06-16T11:58:57.238135Z",
            "url": "https://files.pythonhosted.org/packages/31/b9/355207d29513dd1f09ed4a826591bfa80804150262965353b4661deff87d/aft_pt-0.1.0.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2021-06-16 11:58:57",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "github_user": null,
    "github_project": "shendezhou",
    "error": "Could not fetch GitHub repository",
    "lcname": "aft-pt"
}
        
Elapsed time: 0.29125s