| Name | lmm-tools JSON |
| Version |
0.0.8
JSON |
| download |
| home_page | |
| Summary | Toolset for Large Multi-Modal Models |
| upload_time | 2024-02-23 21:42:46 |
| maintainer | |
| docs_url | None |
| author | Landing AI |
| requires_python | >=3.10,<4.0 |
| license | |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Large Multimodal Model Tools
LMM-Tools (Large Multmodal Model Tools) is a minimal library that helps you utilize multimodal models to organize and structure your image data. One of the problems of dealing with image data is it can be difficult to organize and quickly search. For example, you might have a bunch of pictures of houses and want to count how many yellow houses you have, or how many houses with adobe roofs. This library utilizes LMMs to help create these tags or descriptions and allow you to search over them, or use them in a database to do other operations.
## Getting Started
### LMMs
To get started you can create an LMM and start generating text from images. The following code will grab the LLaVA-1.6 34B model and generate a description of the image you pass it.
```python
import lmm_tools as lmt
model = lmt.lmm.get_model("llava")
model.generate("Describe this image", "image.png")
>>> "A yellow house with a green lawn."
```
**WARNING** We are hosting the LLaVA-1.6 34B model, if it times out please wait ~5-10 min for the server to warm up as it shuts down when usage is low.
### DataStore
You can use the `DataStore` class to store your images, add new metadata to them such as descriptions, and search over different columns.
```python
import lmm_tools as lmt
import pandas as pd
df = pd.DataFrame({"image_paths": ["image1.png", "image2.png", "image3.png"]})
ds = lmt.data.DataStore(df)
ds = ds.add_lmm(lmt.lmm.get_model("llava"))
ds = ds.add_embedder(lmt.emb.get_embedder("sentence-transformer"))
ds = ds.add_column("descriptions", "Describe this image.")
```
This will use the prompt you passed, "Describe this image.", and the LMM to create a new column of descriptions for your image. Your data will now contain a new column with the descriptions of each image:
| image\_paths | image\_id | descriptions |
| --- | --- | --- |
| image1.png | 1 | "A yellow house with a green lawn." |
| image2.png | 2 | "A white house with a two door garage." |
| image3.png | 3 | "A wooden house in the middle of the forest." |
You can now create an index on the descriptions column and search over it to find images that match your query.
```python
ds = ds.build_index("descriptions")
ds.search("A yellow house.", top_k=1)
>>> [{'image_paths': 'image1.png', 'image_id': 1, 'descriptions': 'A yellow house with a green lawn.'}]
```
You can also create other columns for you data such as `is_yellow`:
```python
ds = ds.add_column("is_yellow", "Is the house in this image yellow? Please answer yes or no.")
```
which would give you a dataset similar to this:
| image\_paths | image\_id | descriptions | is\_yellow |
| --- | --- | --- | --- |
| image1.png | 1 | "A yellow house with a green lawn." | "yes" |
| image2.png | 2 | "A white house with a two door garage." | "no" |
| image3.png | 3 | "A wooden house in the middle of the forest." | "no" |
Raw data
{
"_id": null,
"home_page": "",
"name": "lmm-tools",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.10,<4.0",
"maintainer_email": "",
"keywords": "",
"author": "Landing AI",
"author_email": "dev@landing.ai",
"download_url": "https://files.pythonhosted.org/packages/98/2d/d88e02e47a1219eae3d402f0dd7560b0f092428e67f56d3696f3b309be32/lmm_tools-0.0.8.tar.gz",
"platform": null,
"description": "# Large Multimodal Model Tools\nLMM-Tools (Large Multmodal Model Tools) is a minimal library that helps you utilize multimodal models to organize and structure your image data. One of the problems of dealing with image data is it can be difficult to organize and quickly search. For example, you might have a bunch of pictures of houses and want to count how many yellow houses you have, or how many houses with adobe roofs. This library utilizes LMMs to help create these tags or descriptions and allow you to search over them, or use them in a database to do other operations.\n\n## Getting Started\n### LMMs\nTo get started you can create an LMM and start generating text from images. The following code will grab the LLaVA-1.6 34B model and generate a description of the image you pass it.\n\n```python\nimport lmm_tools as lmt\n\nmodel = lmt.lmm.get_model(\"llava\")\nmodel.generate(\"Describe this image\", \"image.png\")\n>>> \"A yellow house with a green lawn.\"\n```\n\n**WARNING** We are hosting the LLaVA-1.6 34B model, if it times out please wait ~5-10 min for the server to warm up as it shuts down when usage is low.\n\n### DataStore\nYou can use the `DataStore` class to store your images, add new metadata to them such as descriptions, and search over different columns.\n\n```python\nimport lmm_tools as lmt\nimport pandas as pd\n\ndf = pd.DataFrame({\"image_paths\": [\"image1.png\", \"image2.png\", \"image3.png\"]})\nds = lmt.data.DataStore(df)\nds = ds.add_lmm(lmt.lmm.get_model(\"llava\"))\nds = ds.add_embedder(lmt.emb.get_embedder(\"sentence-transformer\"))\n\nds = ds.add_column(\"descriptions\", \"Describe this image.\")\n```\n\nThis will use the prompt you passed, \"Describe this image.\", and the LMM to create a new column of descriptions for your image. Your data will now contain a new column with the descriptions of each image:\n\n| image\\_paths | image\\_id | descriptions |\n| --- | --- | --- |\n| image1.png | 1 | \"A yellow house with a green lawn.\" |\n| image2.png | 2 | \"A white house with a two door garage.\" |\n| image3.png | 3 | \"A wooden house in the middle of the forest.\" |\n\nYou can now create an index on the descriptions column and search over it to find images that match your query.\n\n```python\nds = ds.build_index(\"descriptions\")\nds.search(\"A yellow house.\", top_k=1)\n>>> [{'image_paths': 'image1.png', 'image_id': 1, 'descriptions': 'A yellow house with a green lawn.'}]\n```\n\nYou can also create other columns for you data such as `is_yellow`:\n\n```python\nds = ds.add_column(\"is_yellow\", \"Is the house in this image yellow? Please answer yes or no.\")\n```\n\nwhich would give you a dataset similar to this:\n\n| image\\_paths | image\\_id | descriptions | is\\_yellow |\n| --- | --- | --- | --- |\n| image1.png | 1 | \"A yellow house with a green lawn.\" | \"yes\" |\n| image2.png | 2 | \"A white house with a two door garage.\" | \"no\" |\n| image3.png | 3 | \"A wooden house in the middle of the forest.\" | \"no\" |\n",
"bugtrack_url": null,
"license": "",
"summary": "Toolset for Large Multi-Modal Models",
"version": "0.0.8",
"project_urls": {
"Homepage": "https://landing.ai",
"documentation": "https://github.com/landing-ai/lmm-tools",
"repository": "https://github.com/landing-ai/lmm-tools"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "3698f9bd6523e93068aed82014339b615be6b4252eec47c6693665bf9f2d9e7c",
"md5": "27aeb2959462a33a18c834a13139e864",
"sha256": "921ea469a4da16815c38580f23ff466586b37369014e64993bc46011fb0b3366"
},
"downloads": -1,
"filename": "lmm_tools-0.0.8-py3-none-any.whl",
"has_sig": false,
"md5_digest": "27aeb2959462a33a18c834a13139e864",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10,<4.0",
"size": 11052,
"upload_time": "2024-02-23T21:42:45",
"upload_time_iso_8601": "2024-02-23T21:42:45.368925Z",
"url": "https://files.pythonhosted.org/packages/36/98/f9bd6523e93068aed82014339b615be6b4252eec47c6693665bf9f2d9e7c/lmm_tools-0.0.8-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "982dd88e02e47a1219eae3d402f0dd7560b0f092428e67f56d3696f3b309be32",
"md5": "6305a846d35aa9bdadb045ba59148808",
"sha256": "b0897855df3e456d696d7b8a56ab7978fed300e2b18c85e2b97544bd7ce41c89"
},
"downloads": -1,
"filename": "lmm_tools-0.0.8.tar.gz",
"has_sig": false,
"md5_digest": "6305a846d35aa9bdadb045ba59148808",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10,<4.0",
"size": 9551,
"upload_time": "2024-02-23T21:42:46",
"upload_time_iso_8601": "2024-02-23T21:42:46.531144Z",
"url": "https://files.pythonhosted.org/packages/98/2d/d88e02e47a1219eae3d402f0dd7560b0f092428e67f56d3696f3b309be32/lmm_tools-0.0.8.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-23 21:42:46",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "landing-ai",
"github_project": "lmm-tools",
"github_not_found": true,
"lcname": "lmm-tools"
}
