[](https://discord.gg/qUtxnK2NMf)
# NeVA: NVIDIA's Visual Question Answering Transformer
[NeVA is a powerful and versatile Visual Question Answering model by NVIDIA](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/playground/models/neva/bias). Neva builds upon the open-source LLaMA model, integrating it with an NVIDIA-trained GPT model to offer state-of-the-art performance.
[](https://github.com/kyegomez/neva/issues)
[](https://github.com/kyegomez/neva/network)
[](https://github.com/kyegomez/neva/stargazers) [](https://github.com/kyegomez/neva/blob/master/LICENSE)
[](https://twitter.com/intent/tweet?text=Excited%20to%20introduce%20neva,%20the%20all-new%20Multi-Modal%20model%20with%20the%20potential%20to%20revolutionize%20automation.%20Join%20us%20on%20this%20journey%20towards%20a%20smarter%20future.%20%23Neva%20%23Multi-Modal&url=https%3A%2F%2Fgithub.com%2Fkyegomez%2Fneva)
[](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fgithub.com%2Fkyegomez%2Fneva)
[](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fgithub.com%2Fkyegomez%2Fneva&title=Introducing%20neva%2C%20the%20All-New%20Multi-Modal%20Model&summary=neva%20is%20the%20next-generation%20Multi-Modal%20model%20that%20promises%20to%20transform%20industries%20with%20its%20intelligence%20and%20efficiency.%20Join%20us%20to%20be%20a%20part%20of%20this%20revolutionary%20journey%20%23Neva%20%23Multi-Modal&source=)
[](https://www.reddit.com/submit?url=https%3A%2F%2Fgithub.com%2Fkyegomez%2Fneva&title=Exciting%20Times%20Ahead%20with%20neva%2C%20the%20All-New%20Multi-Modal%20Model%20%23Neva%20%23Multi-Modal) [](https://news.ycombinator.com/submitlink?u=https%3A%2F%2Fgithub.com%2Fkyegomez%2Fneva&t=Exciting%20Times%20Ahead%20with%20neva%2C%20the%20All-New%20Multi-Modal%20Model%20%23Neva%20%23Multi-Modal)
[](https://pinterest.com/pin/create/button/?url=https%3A%2F%2Fgithub.com%2Fkyegomez%2Fneva&media=https%3A%2F%2Fexample.com%2Fimage.jpg&description=neva%2C%20the%20Revolutionary%20Multi-Modal%20Model%20that%20will%20Change%20the%20Way%20We%20Work%20%23Neva%20%23Multi-Modal)
[](https://api.whatsapp.com/send?text=I%20just%20discovered%20neva,%20the%20all-new%20Multi-Modal%20model%20that%20promises%20to%20revolutionize%20automation.%20Join%20me%20on%20this%20exciting%20journey%20towards%20a%20smarter%20future.%20%23Neva%20%23Multi-Modal%0A%0Ahttps%3A%2F%2Fgithub.com%2Fkyegomez%2Fneva)
## Appreciation
* All the creators in Agora, [Join Agora](https://discord.gg/qUtxnK2NMf) the community of AI engineers changing the world with their creations.
* LucidRains for inspiring me to devote myself to open source AI
---
## Installation
To integrate NeVA into your Python environment, you can install it via pip:
```bash
pip install nevax
```
---
## Usage
```python
import torch
from nevax import Neva
#usage
img = torch.randn(1, 3, 256, 256)
caption_tokens = torch.randint(0, 4)
model = Neva()
output = model(img, caption_tokens)
```
## Description
At a high level, NeVA utilizes a frozen Hugging Face CLIP model to encode images. These encoded images are projected to text embedding dimensions, concatenated with the embeddings of the given prompt, and subsequently passed through the language model. The training process comprises two main stages:
1. **Pretraining**: Only the projection layer is trained with the language model kept frozen. This stage uses image-caption pairs for training.
2. **Finetuning**: Both the language model and the projection layer are trained. This stage utilizes synthetic instruction data generated with GPT4.
## Model Specifications
- **Architecture Type**: Transformer
- **Network Architecture**: GPT + CLIP
- **Model versions**: 8B, 22B, 43B
## Input & Output
- **Input Format**: RGB Image + Text
- **Input Parameters**: temperature, max output tokens, quality, toxicity, humor, creativity, violence, helpfulness, not_appropriate
- **Output Format**: Text
## Integration and Compatibility
- **Supported Hardware Platforms**: Hopper, Ampere/Turing
- **Supported Operating Systems**: Linux
- **Runtime**: N/A
## Training & Fine-tuning Data
**Pretraining Dataset**:
- **Link**: [CC-3M](#)
- **Description**: The dataset comprises CC3M images and captions, refined to 595,000 samples.
- **License**: [COCO](#), [CC-3M](#), [BLIP](#)
**Finetuning Dataset**:
- **Link**: Synthetic data produced by GPT4
- **Description**: The dataset, with 158,000 samples, was synthetically generated by GPT4. It encompasses a blend of short question answers, detailed image descriptions, and higher-level reasoning questions.
- **License**: [CC-BY-NC 4.0 License](https://creativecommons.org/licenses/by-nc/4.0/)
## Inference
- **Engine**: Triton
- **Test Hardware**: Other
## References
- [Visual Instruction Tuning paper](#)
- [Blog](#)
- [Codebase](#)
- [Demo](#)
## Licensing
This project is licensed under the [MIT License](/LICENSE) license.
# Citation
Raw data
{
"_id": null,
"home_page": "https://github.com/kyegomez/neva",
"name": "nevax",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6,<4.0",
"maintainer_email": "",
"keywords": "",
"author": "Kye Gomez",
"author_email": "kye@apac.ai",
"download_url": "https://files.pythonhosted.org/packages/77/dc/9646c4a0b5821a1f2fa0318222764d025d6325f4583487bb8eb5a762be2a/nevax-0.1.4.tar.gz",
"platform": null,
"description": "[](https://discord.gg/qUtxnK2NMf)\n\n\n# NeVA: NVIDIA's Visual Question Answering Transformer\n\n[NeVA is a powerful and versatile Visual Question Answering model by NVIDIA](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/playground/models/neva/bias). Neva builds upon the open-source LLaMA model, integrating it with an NVIDIA-trained GPT model to offer state-of-the-art performance. \n\n\n[](https://github.com/kyegomez/neva/issues) \n[](https://github.com/kyegomez/neva/network) \n[](https://github.com/kyegomez/neva/stargazers) [](https://github.com/kyegomez/neva/blob/master/LICENSE)\n[](https://twitter.com/intent/tweet?text=Excited%20to%20introduce%20neva,%20the%20all-new%20Multi-Modal%20model%20with%20the%20potential%20to%20revolutionize%20automation.%20Join%20us%20on%20this%20journey%20towards%20a%20smarter%20future.%20%23Neva%20%23Multi-Modal&url=https%3A%2F%2Fgithub.com%2Fkyegomez%2Fneva)\n[](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fgithub.com%2Fkyegomez%2Fneva)\n[](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fgithub.com%2Fkyegomez%2Fneva&title=Introducing%20neva%2C%20the%20All-New%20Multi-Modal%20Model&summary=neva%20is%20the%20next-generation%20Multi-Modal%20model%20that%20promises%20to%20transform%20industries%20with%20its%20intelligence%20and%20efficiency.%20Join%20us%20to%20be%20a%20part%20of%20this%20revolutionary%20journey%20%23Neva%20%23Multi-Modal&source=)\n\n[](https://www.reddit.com/submit?url=https%3A%2F%2Fgithub.com%2Fkyegomez%2Fneva&title=Exciting%20Times%20Ahead%20with%20neva%2C%20the%20All-New%20Multi-Modal%20Model%20%23Neva%20%23Multi-Modal) [](https://news.ycombinator.com/submitlink?u=https%3A%2F%2Fgithub.com%2Fkyegomez%2Fneva&t=Exciting%20Times%20Ahead%20with%20neva%2C%20the%20All-New%20Multi-Modal%20Model%20%23Neva%20%23Multi-Modal)\n[](https://pinterest.com/pin/create/button/?url=https%3A%2F%2Fgithub.com%2Fkyegomez%2Fneva&media=https%3A%2F%2Fexample.com%2Fimage.jpg&description=neva%2C%20the%20Revolutionary%20Multi-Modal%20Model%20that%20will%20Change%20the%20Way%20We%20Work%20%23Neva%20%23Multi-Modal)\n[](https://api.whatsapp.com/send?text=I%20just%20discovered%20neva,%20the%20all-new%20Multi-Modal%20model%20that%20promises%20to%20revolutionize%20automation.%20Join%20me%20on%20this%20exciting%20journey%20towards%20a%20smarter%20future.%20%23Neva%20%23Multi-Modal%0A%0Ahttps%3A%2F%2Fgithub.com%2Fkyegomez%2Fneva)\n\n\n## Appreciation\n* All the creators in Agora, [Join Agora](https://discord.gg/qUtxnK2NMf) the community of AI engineers changing the world with their creations.\n* LucidRains for inspiring me to devote myself to open source AI\n\n\n\n---\n\n## Installation\n\nTo integrate NeVA into your Python environment, you can install it via pip:\n\n```bash\npip install nevax\n```\n---\n\n## Usage\n\n```python\nimport torch\nfrom nevax import Neva\n\n#usage\nimg = torch.randn(1, 3, 256, 256)\ncaption_tokens = torch.randint(0, 4)\n\nmodel = Neva()\noutput = model(img, caption_tokens)\n```\n\n## Description\n\nAt a high level, NeVA utilizes a frozen Hugging Face CLIP model to encode images. These encoded images are projected to text embedding dimensions, concatenated with the embeddings of the given prompt, and subsequently passed through the language model. The training process comprises two main stages:\n\n1. **Pretraining**: Only the projection layer is trained with the language model kept frozen. This stage uses image-caption pairs for training.\n2. **Finetuning**: Both the language model and the projection layer are trained. This stage utilizes synthetic instruction data generated with GPT4.\n\n## Model Specifications\n\n- **Architecture Type**: Transformer\n- **Network Architecture**: GPT + CLIP\n- **Model versions**: 8B, 22B, 43B\n\n## Input & Output\n\n- **Input Format**: RGB Image + Text\n- **Input Parameters**: temperature, max output tokens, quality, toxicity, humor, creativity, violence, helpfulness, not_appropriate\n- **Output Format**: Text\n\n## Integration and Compatibility\n\n- **Supported Hardware Platforms**: Hopper, Ampere/Turing\n- **Supported Operating Systems**: Linux\n- **Runtime**: N/A\n\n## Training & Fine-tuning Data\n\n**Pretraining Dataset**:\n- **Link**: [CC-3M](#)\n- **Description**: The dataset comprises CC3M images and captions, refined to 595,000 samples.\n- **License**: [COCO](#), [CC-3M](#), [BLIP](#)\n\n**Finetuning Dataset**:\n- **Link**: Synthetic data produced by GPT4\n- **Description**: The dataset, with 158,000 samples, was synthetically generated by GPT4. It encompasses a blend of short question answers, detailed image descriptions, and higher-level reasoning questions.\n- **License**: [CC-BY-NC 4.0 License](https://creativecommons.org/licenses/by-nc/4.0/)\n\n## Inference\n\n- **Engine**: Triton\n- **Test Hardware**: Other\n\n## References\n\n- [Visual Instruction Tuning paper](#)\n- [Blog](#)\n- [Codebase](#)\n- [Demo](#)\n\n## Licensing\n\nThis project is licensed under the [MIT License](/LICENSE) license.\n\n\n# Citation",
"bugtrack_url": null,
"license": "Attribution-NonCommercial 4.0 International",
"summary": "NeVA: NVIDIA's Visual Question Answering Transformer",
"version": "0.1.4",
"project_urls": {
"Homepage": "https://github.com/kyegomez/neva",
"Repository": "https://github.com/kyegomez/neva"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c93d00c9249789914397a46363d40294d757e04eda17a6dcec70013c32172c05",
"md5": "c77b6d0d28a3062704c19fbe7144b819",
"sha256": "49557984824dc81246e7c10f0a4a50acd3d19528ae5b7d42379ba8e3175b6c07"
},
"downloads": -1,
"filename": "nevax-0.1.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c77b6d0d28a3062704c19fbe7144b819",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6,<4.0",
"size": 22687,
"upload_time": "2023-08-26T03:02:54",
"upload_time_iso_8601": "2023-08-26T03:02:54.162000Z",
"url": "https://files.pythonhosted.org/packages/c9/3d/00c9249789914397a46363d40294d757e04eda17a6dcec70013c32172c05/nevax-0.1.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "77dc9646c4a0b5821a1f2fa0318222764d025d6325f4583487bb8eb5a762be2a",
"md5": "df25ef918142a8b48b407b2c6d022ca4",
"sha256": "ab606c9cc8a8c1204e198fb0f41010e014330679e96120551efcb9e974f5a4f8"
},
"downloads": -1,
"filename": "nevax-0.1.4.tar.gz",
"has_sig": false,
"md5_digest": "df25ef918142a8b48b407b2c6d022ca4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6,<4.0",
"size": 24288,
"upload_time": "2023-08-26T03:02:55",
"upload_time_iso_8601": "2023-08-26T03:02:55.789966Z",
"url": "https://files.pythonhosted.org/packages/77/dc/9646c4a0b5821a1f2fa0318222764d025d6325f4583487bb8eb5a762be2a/nevax-0.1.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-26 03:02:55",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "kyegomez",
"github_project": "neva",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "nevax"
}
JSON
