[](https://discord.gg/qUtxnK2NMf)
# NeoCortex: Multi-Modality Foundation Model for Humanoid Robots
NeoCortex is a cutting-edge, multi-modality foundation model engineered for humanoid robots. With the capability to process an array of modalities including images, text, videos, depth, heatmaps, LIDAR, 3D scenes, point clouds, and more, NeoCortex represents the zenith of robotic perception and cognition.
## Appreciation
* All the creators in Agora, [Join Agora](https://discord.gg/qUtxnK2NMf) the community of AI engineers changing the world with their creations.
* LucidRains for inspiring me to devote myself to open source AI
## Installation
To integrate NeoCortex into your development environment:
```bash
pip install neox
```
## Usage
```python
import torch
from neox.model import NeoCortex
#usage
img = torch.randn(1, 3, 256, 256)
caption_tokens = torch.randint(0, 4)
model = NeoCortex()
output = model(img, caption_tokens)
```
## Model Architecture
Building on the paradigms established by models like NeVA, NeoCortex expands its horizons to embrace a multi-faceted input system:
- **Images**: Processed through a frozen variant of the Hugging Face CLIP model, producing image embeddings.
- **Text**: Integrated seamlessly with an NVIDIA-trained GPT variant.
- **LIDAR**: Point clouds processed through PointNet or KPConv, extracting essential spatial features. These features, once captured, are projected to text embedding dimensions.
- **Videos**: Processed using 3D-CNNs to capture temporal information, then integrated with the main architecture.
- **Heatmaps, 3D Scenes, and Depth**: These are funneled through specialized modules tailored for each modality before being integrated.
Training comprises three stages:
1. **Pretraining**: Specific modules (like PointNet for LIDAR) are pretrained, with the main model frozen.
2. **Inter-modality Training**: The embeddings from different modalities are concatenated, projected, and trained to ensure inter-modality coherence.
3. **Finetuning**: Utilizing synthetic data generated with advanced GPT versions to ensure the model understands the intricate relationship between various modalities.
## Specifications
- **Architecture Type**: Multi-Modality Transformer
- **Sub-Architectures**: GPT, CLIP, PointNet, 3D-CNN
- **Model versions**: 10B, 30B, 50B
## Input & Output
- **Input Formats**:
- Images: RGB
- Text: Tokens
- LIDAR: Point Clouds
- Videos: Frames
- And more...
- **Output Format**: Text or Action Tokens
## Integration and Compatibility
- **Supported Hardware Platforms**: Hopper, Ampere/Turing, specialized robotic chipsets
- **Supported Operating Systems**: Linux, RoboOS
## Training & Fine-tuning Data
- **Image Dataset**: [Link to Dataset]
- **LIDAR Dataset**: [Link to Dataset]
- **Textual Data**: Synthetically produced by GPT versions
- **Licenses**: Various, mainly MIT and CC-BY-NC 4.0
## Inference
- **Engine**: Triton and specialized robotic inference engines
- **Test Hardware**: Humanoid robotic platforms
## References and More
- [Visual Instruction Tuning paper](#)
- [Blog](#)
- [Codebase](https://github.com/kyegomez/NeoCortex)
- [Demo](#)
## Licensing
This project is licensed under the MIT License.
For more details, contributions, and support, please refer to the [official repository](https://github.com/kyegomez/NeoCortex).
# Todo
[] -
Raw data
{
"_id": null,
"home_page": "https://github.com/kyegomez/NeoCortex",
"name": "neox",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6,<4.0",
"maintainer_email": "",
"keywords": "",
"author": "Kye Gomez",
"author_email": "kye@apac.ai",
"download_url": "https://files.pythonhosted.org/packages/5e/f2/a1e8aaa28c73dcbd85481c9d5fa0c4747e5f691e2d6932fd7b8143fcedad/neox-0.1.3.tar.gz",
"platform": null,
"description": "[](https://discord.gg/qUtxnK2NMf)\n\n\n# NeoCortex: Multi-Modality Foundation Model for Humanoid Robots\n\nNeoCortex is a cutting-edge, multi-modality foundation model engineered for humanoid robots. With the capability to process an array of modalities including images, text, videos, depth, heatmaps, LIDAR, 3D scenes, point clouds, and more, NeoCortex represents the zenith of robotic perception and cognition.\n\n## Appreciation\n* All the creators in Agora, [Join Agora](https://discord.gg/qUtxnK2NMf) the community of AI engineers changing the world with their creations.\n* LucidRains for inspiring me to devote myself to open source AI\n\n\n## Installation\nTo integrate NeoCortex into your development environment:\n\n```bash\npip install neox\n```\n\n## Usage\n```python\nimport torch\nfrom neox.model import NeoCortex\n\n#usage\nimg = torch.randn(1, 3, 256, 256)\ncaption_tokens = torch.randint(0, 4)\n\nmodel = NeoCortex()\noutput = model(img, caption_tokens)\n```\n\n## Model Architecture\nBuilding on the paradigms established by models like NeVA, NeoCortex expands its horizons to embrace a multi-faceted input system:\n\n- **Images**: Processed through a frozen variant of the Hugging Face CLIP model, producing image embeddings.\n- **Text**: Integrated seamlessly with an NVIDIA-trained GPT variant.\n- **LIDAR**: Point clouds processed through PointNet or KPConv, extracting essential spatial features. These features, once captured, are projected to text embedding dimensions.\n- **Videos**: Processed using 3D-CNNs to capture temporal information, then integrated with the main architecture.\n- **Heatmaps, 3D Scenes, and Depth**: These are funneled through specialized modules tailored for each modality before being integrated.\n\nTraining comprises three stages:\n1. **Pretraining**: Specific modules (like PointNet for LIDAR) are pretrained, with the main model frozen.\n2. **Inter-modality Training**: The embeddings from different modalities are concatenated, projected, and trained to ensure inter-modality coherence.\n3. **Finetuning**: Utilizing synthetic data generated with advanced GPT versions to ensure the model understands the intricate relationship between various modalities.\n\n## Specifications\n\n- **Architecture Type**: Multi-Modality Transformer\n- **Sub-Architectures**: GPT, CLIP, PointNet, 3D-CNN\n- **Model versions**: 10B, 30B, 50B\n\n## Input & Output\n\n- **Input Formats**: \n - Images: RGB\n - Text: Tokens\n - LIDAR: Point Clouds\n - Videos: Frames\n - And more...\n- **Output Format**: Text or Action Tokens\n\n## Integration and Compatibility\n\n- **Supported Hardware Platforms**: Hopper, Ampere/Turing, specialized robotic chipsets\n- **Supported Operating Systems**: Linux, RoboOS\n\n## Training & Fine-tuning Data\n\n- **Image Dataset**: [Link to Dataset]\n- **LIDAR Dataset**: [Link to Dataset]\n- **Textual Data**: Synthetically produced by GPT versions\n- **Licenses**: Various, mainly MIT and CC-BY-NC 4.0\n\n## Inference\n\n- **Engine**: Triton and specialized robotic inference engines\n- **Test Hardware**: Humanoid robotic platforms\n\n## References and More\n\n- [Visual Instruction Tuning paper](#)\n- [Blog](#)\n- [Codebase](https://github.com/kyegomez/NeoCortex)\n- [Demo](#)\n\n## Licensing\nThis project is licensed under the MIT License.\n\nFor more details, contributions, and support, please refer to the [official repository](https://github.com/kyegomez/NeoCortex).\n\n\n# Todo\n\n[] - ",
"bugtrack_url": null,
"license": "Attribution-NonCommercial 4.0 International",
"summary": "An Multi-Modality Foundation Model for Humanoid robots",
"version": "0.1.3",
"project_urls": {
"Homepage": "https://github.com/kyegomez/NeoCortex",
"Repository": "https://github.com/kyegomez/NeoCortex"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "59b8ebcbdc5bf166b145456a840eee1371fdc5be48b2532abadc6b1f6c43fd99",
"md5": "27e06567f8af64edc1cb7683ae0a2359",
"sha256": "6611add5b936c77f1ea2536a9305d47d0f408618d5ecd7ca974d60a22b21c6f1"
},
"downloads": -1,
"filename": "neox-0.1.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "27e06567f8af64edc1cb7683ae0a2359",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6,<4.0",
"size": 22003,
"upload_time": "2023-08-26T03:15:04",
"upload_time_iso_8601": "2023-08-26T03:15:04.018076Z",
"url": "https://files.pythonhosted.org/packages/59/b8/ebcbdc5bf166b145456a840eee1371fdc5be48b2532abadc6b1f6c43fd99/neox-0.1.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "5ef2a1e8aaa28c73dcbd85481c9d5fa0c4747e5f691e2d6932fd7b8143fcedad",
"md5": "2f8f64e84f5f8dcaec19e42608bcbc4a",
"sha256": "2aa8937c2d1cbd0002079fb6f8bb2a87f18f1a3efbb8bfbeb5340e4c3959bb48"
},
"downloads": -1,
"filename": "neox-0.1.3.tar.gz",
"has_sig": false,
"md5_digest": "2f8f64e84f5f8dcaec19e42608bcbc4a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6,<4.0",
"size": 22897,
"upload_time": "2023-08-26T03:15:05",
"upload_time_iso_8601": "2023-08-26T03:15:05.587639Z",
"url": "https://files.pythonhosted.org/packages/5e/f2/a1e8aaa28c73dcbd85481c9d5fa0c4747e5f691e2d6932fd7b8143fcedad/neox-0.1.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-26 03:15:05",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "kyegomez",
"github_project": "NeoCortex",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "neox"
}
JSON
