<div align="center">
<h1>multimodal-maestro</h1>
<br>
[](https://badge.fury.io/py/maestro)
[](https://github.com/roboflow/multimodal-maestro/blob/main/LICENSE)
[](https://badge.fury.io/py/maestro)
[](https://huggingface.co/spaces/Roboflow/SoM)
[](https://colab.research.google.com/github/roboflow/multimodal-maestro/blob/main/cookbooks/multimodal_maestro_gpt_4_vision.ipynb
)
</div>
## 👋 hello
Multimodal-Maestro gives you more control over large multimodal models to get the
outputs you want. With more effective prompting tactics, you can get multimodal models
to do tasks you didn't know (or think!) were possible. Curious how it works? Try our
HF [space](https://huggingface.co/spaces/Roboflow/SoM)!
🚧 The project is still under construction and the API is prone to change.
## 💻 install
⚠️ Our package has been renamed to `maestro`. Install package in a
[**3.11>=Python>=3.8**](https://www.python.org/) environment.
```bash
pip install maestro
```
## 🚀 examples
### GPT-4 Vision
```
Find dog.
>>> The dog is prominently featured in the center of the image with the label [9].
```
<details close>
<summary>👉 read more</summary>
<br>
- **load image**
```python
import cv2
image = cv2.imread("...")
```
- **create and refine marks**
```python
import maestro as mm
generator = mm.SegmentAnythingMarkGenerator(device='cuda')
marks = generator.generate(image=image)
marks = mm.refine_marks(marks=marks)
```
- **visualize marks**
```python
mark_visualizer = mm.MarkVisualizer()
marked_image = mark_visualizer.visualize(image=image, marks=marks)
```
- **prompt**
```python
prompt = "Find dog."
response = mm.prompt_image(api_key=api_key, image=marked_image, prompt=prompt)
```
```
>>> "The dog is prominently featured in the center of the image with the label [9]."
```
- **extract related marks**
```python
masks = mm.extract_relevant_masks(text=response, detections=refined_marks)
```
```
>>> {'6': array([
... [False, False, False, ..., False, False, False],
... [False, False, False, ..., False, False, False],
... [False, False, False, ..., False, False, False],
... ...,
... [ True, True, True, ..., False, False, False],
... [ True, True, True, ..., False, False, False],
... [ True, True, True, ..., False, False, False]])
... }
```
</details>
## 🚧 roadmap
- [ ] Documentation page.
- [ ] Segment Anything guided marks generation.
- [ ] Non-Max Suppression marks refinement.
- [ ] LLaVA demo.
## 💜 acknowledgement
- [Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding
in GPT-4V](https://arxiv.org/abs/2310.11441) by Jianwei Yang, Hao Zhang, Feng Li, Xueyan
Zou, Chunyuan Li, Jianfeng Gao.
## 🦸 contribution
We would love your help in making this repository even better! If you noticed any bug,
or if you have any suggestions for improvement, feel free to open an
[issue](https://github.com/roboflow/set-of-mark/issues) or submit a
[pull request](https://github.com/roboflow/set-of-mark/pulls).
Raw data
{
"_id": null,
"home_page": "https://github.com/roboflow/multimodal-maestro",
"name": "maestro",
"maintainer": "Piotr Skalski",
"docs_url": null,
"requires_python": ">=3.8,<3.12.0",
"maintainer_email": "piotr.skalski92@gmail.com",
"keywords": "",
"author": "Piotr Skalski",
"author_email": "piotr.skalski92@gmail.com",
"download_url": "",
"platform": null,
"description": "\n<div align=\"center\">\n\n <h1>multimodal-maestro</h1>\n\n <br>\n\n [](https://badge.fury.io/py/maestro)\n [](https://github.com/roboflow/multimodal-maestro/blob/main/LICENSE)\n [](https://badge.fury.io/py/maestro)\n [](https://huggingface.co/spaces/Roboflow/SoM)\n [](https://colab.research.google.com/github/roboflow/multimodal-maestro/blob/main/cookbooks/multimodal_maestro_gpt_4_vision.ipynb\n)\n\n</div>\n\n## \ud83d\udc4b hello\n\nMultimodal-Maestro gives you more control over large multimodal models to get the \noutputs you want. With more effective prompting tactics, you can get multimodal models \nto do tasks you didn't know (or think!) were possible. Curious how it works? Try our \nHF [space](https://huggingface.co/spaces/Roboflow/SoM)!\n\n\ud83d\udea7 The project is still under construction and the API is prone to change.\n\n## \ud83d\udcbb install\n\n\u26a0\ufe0f Our package has been renamed to `maestro`. Install package in a\n[**3.11>=Python>=3.8**](https://www.python.org/) environment.\n\n```bash\npip install maestro\n```\n\n## \ud83d\ude80 examples\n\n### GPT-4 Vision\n\n```\nFind dog.\n\n>>> The dog is prominently featured in the center of the image with the label [9].\n```\n\n<details close>\n<summary>\ud83d\udc49 read more</summary>\n\n<br>\n\n- **load image**\n\n ```python\n import cv2\n \n image = cv2.imread(\"...\")\n ```\n\n- **create and refine marks**\n\n ```python\n import maestro as mm\n \n generator = mm.SegmentAnythingMarkGenerator(device='cuda')\n marks = generator.generate(image=image)\n marks = mm.refine_marks(marks=marks)\n ```\n\n- **visualize marks**\n\n ```python\n mark_visualizer = mm.MarkVisualizer()\n marked_image = mark_visualizer.visualize(image=image, marks=marks)\n ```\n \n\n- **prompt**\n\n ```python\n prompt = \"Find dog.\"\n \n response = mm.prompt_image(api_key=api_key, image=marked_image, prompt=prompt)\n ```\n \n ```\n >>> \"The dog is prominently featured in the center of the image with the label [9].\"\n ```\n\n- **extract related marks**\n\n ```python\n masks = mm.extract_relevant_masks(text=response, detections=refined_marks)\n ```\n \n ```\n >>> {'6': array([\n ... [False, False, False, ..., False, False, False],\n ... [False, False, False, ..., False, False, False],\n ... [False, False, False, ..., False, False, False],\n ... ...,\n ... [ True, True, True, ..., False, False, False],\n ... [ True, True, True, ..., False, False, False],\n ... [ True, True, True, ..., False, False, False]])\n ... }\n ```\n\n</details>\n\n\n\n## \ud83d\udea7 roadmap\n\n- [ ] Documentation page.\n- [ ] Segment Anything guided marks generation.\n- [ ] Non-Max Suppression marks refinement.\n- [ ] LLaVA demo.\n\n## \ud83d\udc9c acknowledgement\n\n- [Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding\nin GPT-4V](https://arxiv.org/abs/2310.11441) by Jianwei Yang, Hao Zhang, Feng Li, Xueyan\nZou, Chunyuan Li, Jianfeng Gao.\n\n## \ud83e\uddb8 contribution\n\nWe would love your help in making this repository even better! If you noticed any bug, \nor if you have any suggestions for improvement, feel free to open an \n[issue](https://github.com/roboflow/set-of-mark/issues) or submit a \n[pull request](https://github.com/roboflow/set-of-mark/pulls).\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Visual Prompting for Large Multimodal Models (LMMs)",
"version": "0.1.0",
"project_urls": {
"Documentation": "https://roboflow.github.io/multimodal-maestro",
"Homepage": "https://github.com/roboflow/multimodal-maestro",
"Repository": "https://github.com/roboflow/multimodal-maestro"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "6453f518cd396659d3a3b564d4653adaa6c0bf6285cda54d6920e5c08195139a",
"md5": "8b4094f305c72c17bdead11be56658e5",
"sha256": "2d75ea40d74480dcc39b26f91134792ea1103b84cd72ab2ec629d533a70484c0"
},
"downloads": -1,
"filename": "maestro-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "8b4094f305c72c17bdead11be56658e5",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8,<3.12.0",
"size": 11631,
"upload_time": "2023-11-30T17:09:48",
"upload_time_iso_8601": "2023-11-30T17:09:48.928394Z",
"url": "https://files.pythonhosted.org/packages/64/53/f518cd396659d3a3b564d4653adaa6c0bf6285cda54d6920e5c08195139a/maestro-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-30 17:09:48",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "roboflow",
"github_project": "multimodal-maestro",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "maestro"
}
JSON
