[](https://discord.gg/qUtxnK2NMf)
# TinyGPTV
Simple Implementation of TinyGPTV in super simple Zeta lego blocks. Here all the modules from figure 2 are implemented in Zeta and Pytorch.
The flow is the following:
x -> skip connection -> layer norm -> lora -> mha + lora -> residual_rms_norm -> original_skip_connection -> mlp + rms norm
## Install
`pip3 install tiny-gptv`
## Usage
### TinyGPTVBlock, Figure3 (c):
- Layernorm
- MHA
- Lora
- QK Norm
- RMS Norm
- MLP
```python
import torch
from tiny_gptv.blocks import TinyGPTVBlock
# Random tensor, replace with your input data
x = torch.rand(2, 8, 512)
# TinyGPTVBlock
block = TinyGPTVBlock(512, 8, depth=10)
# Print the block
print(block)
# Forward pass
out = block(x)
# Print the output shape
print(out.shape)
```
### Figure3 (b) Lora Module for LLMS Block
- MHA,
- Lora,
- Normalization,
- MLP
- Skip connection
- Split then add
```python
import torch
from tiny_gptv import LoraMHA
x = torch.rand(2, 8, 512)
block = LoraMHA(512, 8)
out = block(x)
print(out.shape)
```
# Citation
```bibtex
@misc{yuan2023tinygptv,
title={TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones},
author={Zhengqing Yuan and Zhaoxu Li and Lichao Sun},
year={2023},
eprint={2312.16862},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
```
# License
MIT
Raw data
{
"_id": null,
"home_page": "https://github.com/kyegomez/TinyGPTV",
"name": "tiny-gptv",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6,<4.0",
"maintainer_email": "",
"keywords": "artificial intelligence,deep learning,optimizers,Prompt Engineering",
"author": "Kye Gomez",
"author_email": "kye@apac.ai",
"download_url": "https://files.pythonhosted.org/packages/39/d4/b88b55f804377ec5a1045df401f9aea5cae5412fd383e21593e7f27cf32e/tiny_gptv-0.0.5.tar.gz",
"platform": null,
"description": "[](https://discord.gg/qUtxnK2NMf)\n\n# TinyGPTV\nSimple Implementation of TinyGPTV in super simple Zeta lego blocks. Here all the modules from figure 2 are implemented in Zeta and Pytorch.\n\nThe flow is the following:\nx -> skip connection -> layer norm -> lora -> mha + lora -> residual_rms_norm -> original_skip_connection -> mlp + rms norm\n\n\n## Install\n`pip3 install tiny-gptv`\n\n\n## Usage\n\n### TinyGPTVBlock, Figure3 (c):\n- Layernorm\n- MHA\n- Lora\n- QK Norm\n- RMS Norm\n- MLP\n\n\n```python\nimport torch\nfrom tiny_gptv.blocks import TinyGPTVBlock\n\n# Random tensor, replace with your input data\nx = torch.rand(2, 8, 512)\n\n# TinyGPTVBlock\nblock = TinyGPTVBlock(512, 8, depth=10)\n\n# Print the block\nprint(block)\n\n# Forward pass\nout = block(x)\n\n# Print the output shape\nprint(out.shape)\n\n\n```\n\n### Figure3 (b) Lora Module for LLMS Block\n- MHA,\n- Lora,\n- Normalization,\n- MLP\n- Skip connection\n- Split then add\n\n```python\nimport torch\nfrom tiny_gptv import LoraMHA\n\nx = torch.rand(2, 8, 512)\nblock = LoraMHA(512, 8)\nout = block(x)\nprint(out.shape)\n\n```\n\n\n# Citation\n\n```bibtex\n@misc{yuan2023tinygptv,\n title={TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones}, \n author={Zhengqing Yuan and Zhaoxu Li and Lichao Sun},\n year={2023},\n eprint={2312.16862},\n archivePrefix={arXiv},\n primaryClass={cs.CV}\n}\n\n```\n\n# License\nMIT",
"bugtrack_url": null,
"license": "MIT",
"summary": "Tiny GPTV - Pytorch",
"version": "0.0.5",
"project_urls": {
"Documentation": "https://github.com/kyegomez/TinyGPTV",
"Homepage": "https://github.com/kyegomez/TinyGPTV",
"Repository": "https://github.com/kyegomez/TinyGPTV"
},
"split_keywords": [
"artificial intelligence",
"deep learning",
"optimizers",
"prompt engineering"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "5f4f6012d8709c2795670cde0d90d4b8d20aba7363e4be7979843804a02f7c5d",
"md5": "8b67b5d61014b8f4d264eb3553f406e7",
"sha256": "83b08d1f178211348d6e6c2b2e24ba38d3cad4062eddd35da2d75581faea1403"
},
"downloads": -1,
"filename": "tiny_gptv-0.0.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "8b67b5d61014b8f4d264eb3553f406e7",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6,<4.0",
"size": 4426,
"upload_time": "2023-12-31T04:19:52",
"upload_time_iso_8601": "2023-12-31T04:19:52.905967Z",
"url": "https://files.pythonhosted.org/packages/5f/4f/6012d8709c2795670cde0d90d4b8d20aba7363e4be7979843804a02f7c5d/tiny_gptv-0.0.5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "39d4b88b55f804377ec5a1045df401f9aea5cae5412fd383e21593e7f27cf32e",
"md5": "ba2b964f4a3c3bec60985054236f532b",
"sha256": "659f25aec9d149292e99d5ea812fbbf1c2f7ca6e8b94691f5287b6cf00fd7cd8"
},
"downloads": -1,
"filename": "tiny_gptv-0.0.5.tar.gz",
"has_sig": false,
"md5_digest": "ba2b964f4a3c3bec60985054236f532b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6,<4.0",
"size": 4628,
"upload_time": "2023-12-31T04:19:54",
"upload_time_iso_8601": "2023-12-31T04:19:54.508934Z",
"url": "https://files.pythonhosted.org/packages/39/d4/b88b55f804377ec5a1045df401f9aea5cae5412fd383e21593e7f27cf32e/tiny_gptv-0.0.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-12-31 04:19:54",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "kyegomez",
"github_project": "TinyGPTV",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "tiny-gptv"
}
JSON
