# Coin-CLIP πͺ : Enhancing Coin Image Retrieval with CLIP
<div align="center">
<strong>
[[δΈζ]](./README_cn.md) | [English]
</strong>
</div>
**[Coin-CLIP](https://huggingface.co/breezedeus/coin-clip-vit-base-patch32)** `breezedeus/coin-clip-vit-base-patch32` is built upon
OpenAI's **[CLIP](https://huggingface.co/openai/clip-vit-base-patch32) (ViT-B/32)** model and fine-tuned on
a dataset of more than `340,000` coin images using contrastive learning techniques. This specialized model is designed to significantly improve feature extraction for coin images, leading to more accurate image-based search capabilities. Coin-CLIP combines the power of Visual Transformer (ViT) with CLIP's multimodal learning capabilities, specifically tailored for the numismatic domain.
**Key Features:**
- State-of-the-art coin image retrieval;
- Enhanced feature extraction for numismatic images;
- Seamless integration with CLIP's multimodal learning.
To further simplify the use of the **Coin-CLIP** model, this project provides tools for quickly building a coin image retrieval engine.
# Comparison: Coin-CLIP vs. CLIP
#### Example 1 (Left: Coin-CLIP; Right: CLIP)
#### Example 2 (Left: Coin-CLIP; Right: CLIP)
#### More Examples
<details>
<summary>more</summary>
Example 3 (Left: Coin-CLIP; Right: CLIP)
Example 4 (Left: Coin-CLIP; Right: CLIP)
Example 5 (Left: Coin-CLIP; Right: CLIP)
Example 6 (Left: Coin-CLIP; Right: CLIP)
</details>
# Install
```bash
pip install coin_clip
```
# Usage
## Code Examples
### Extract Feature Vectors from Coin Images
```python
from coin_clip import CoinClip
# Automatically download the model from Huggingface
model = CoinClip(model_name='breezedeus/coin-clip-vit-base-patch32')
images = ['examples/10_back.jpg', 'examples/16_back.jpg']
img_feats, success_ids = model.get_image_features(images)
print(img_feats.shape) # --> (2, 512)
```
> β οΈ **Note**:
>
> The above code automatically downloads the [`breezedeus/coin-clip-vit-base-patch32`](https://huggingface.co/breezedeus/coin-clip-vit-base-patch32) model from Huggingface.
If you cannot download automatically, please manually download the model locally, and then initialize `CoinClip` by specifying the local directory of the model through the `model_name` parameter, like `model_name='path/to/coin-clip-vit-base-patch32'`.
## Command line tools
### Building a Vector Retrieval Engine
`coin-clip build-db` can be used to build a vector search engine. It extracts features from all coin images πͺ in a specified directory and builds a ChromaDB vector search engine.
```bash
$ coin-clip build-db -h
Usage: coin-clip build-db [OPTIONS]
Extract vectors from a candidate image set and build a search engine based
on it.
Options:
-m, --model-name TEXT Model Name; either local path or huggingface
model name [default: breezedeus/coin-clip-vit-
base-patch32]
-d, --device TEXT ['cpu', 'cuda']; Either 'cpu' or 'gpu', or
specify a specific GPU like 'cuda:0'. Default is
'cpu'. [default: cpu]
-i, --input-image-dir TEXT Folder with Coin Images to be indexed. [required]
-o, --output-db-dir TEXT Folder where the built search engine is stored.
[default: ./coin_clip_chroma.db]
-h, --help Show this message and exit.
```
For instance,
```bash
$ coin-clip build-db -i examples -o coin_clip_chroma.db
```
### Querying
After building the vector search engine with the above command, you can use the `coin-clip retrieve` command to retrieve the coin images πͺ most similar to a specified coin image.
```bash
$ coin-clip retrieve -h
Usage: coin-clip retrieve [OPTIONS]
Retrieve images from the search engine, based on the query image.
Options:
-m, --model-name TEXT Model Name; either local path or huggingface model
name [default: breezedeus/coin-clip-vit-base-
patch32]
-d, --device TEXT ['cpu', 'cuda']; Either 'cpu' or 'gpu', or specify a
specific GPU like 'cuda:0'. Default is 'cpu'.
[default: cpu]
--db-dir TEXT Folder where the built search engine is stored.
[default: ./coin_clip_chroma.db]
-i, --image-fp TEXT Image Path to retrieve [required]
-h, --help Show this message and exit.
```
For instance,
```bash
$ coin-clip retrieve --db-dir coin_clip_chroma.db -i examples/10_back.jpg
```
Raw data
{
"_id": null,
"home_page": "https://github.com/breezedeus/coin-clip",
"name": "coin-clip",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "",
"author": "breezedeus",
"author_email": "breezedeus@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/e9/b4/7fcd054569570b25a0b684a199b03460306f95110e55efb1bc630afa9d40/coin_clip-0.1.tar.gz",
"platform": "Mac",
"description": "# Coin-CLIP \ud83e\ude99 : Enhancing Coin Image Retrieval with CLIP\n\n <div align=\"center\">\n <strong>\n\n[[\u4e2d\u6587]](./README_cn.md) | [English]\n\n </strong>\n</div>\n\n**[Coin-CLIP](https://huggingface.co/breezedeus/coin-clip-vit-base-patch32)** `breezedeus/coin-clip-vit-base-patch32` is built upon \nOpenAI's **[CLIP](https://huggingface.co/openai/clip-vit-base-patch32) (ViT-B/32)** model and fine-tuned on \na dataset of more than `340,000` coin images using contrastive learning techniques. This specialized model is designed to significantly improve feature extraction for coin images, leading to more accurate image-based search capabilities. Coin-CLIP combines the power of Visual Transformer (ViT) with CLIP's multimodal learning capabilities, specifically tailored for the numismatic domain.\n\n**Key Features:**\n- State-of-the-art coin image retrieval;\n- Enhanced feature extraction for numismatic images;\n- Seamless integration with CLIP's multimodal learning.\n\n\nTo further simplify the use of the **Coin-CLIP** model, this project provides tools for quickly building a coin image retrieval engine.\n\n# Comparison: Coin-CLIP vs. CLIP\n\n#### Example 1 (Left: Coin-CLIP; Right: CLIP)\n\n\n\n#### Example 2 (Left: Coin-CLIP; Right: CLIP)\n\n\n\n#### More Examples\n\n<details>\n\n<summary>more</summary>\n\nExample 3 (Left: Coin-CLIP; Right: CLIP)\n\n\nExample 4 (Left: Coin-CLIP; Right: CLIP)\n\n\nExample 5 (Left: Coin-CLIP; Right: CLIP)\n\n\nExample 6 (Left: Coin-CLIP; Right: CLIP)\n\n\n</details>\n\n# Install\n\n```bash\npip install coin_clip\n```\n\n# Usage\n## Code Examples\n\n### Extract Feature Vectors from Coin Images\n\n```python\nfrom coin_clip import CoinClip\n\n# Automatically download the model from Huggingface\nmodel = CoinClip(model_name='breezedeus/coin-clip-vit-base-patch32')\nimages = ['examples/10_back.jpg', 'examples/16_back.jpg']\nimg_feats, success_ids = model.get_image_features(images)\nprint(img_feats.shape) # --> (2, 512)\n```\n\n> \u26a0\ufe0f **Note**:\n> \n> The above code automatically downloads the [`breezedeus/coin-clip-vit-base-patch32`](https://huggingface.co/breezedeus/coin-clip-vit-base-patch32) model from Huggingface.\nIf you cannot download automatically, please manually download the model locally, and then initialize `CoinClip` by specifying the local directory of the model through the `model_name` parameter, like `model_name='path/to/coin-clip-vit-base-patch32'`.\n\n## Command line tools\n\n### Building a Vector Retrieval Engine\n\n`coin-clip build-db` can be used to build a vector search engine. It extracts features from all coin images \ud83e\ude99 in a specified directory and builds a ChromaDB vector search engine.\n\n```bash\n$ coin-clip build-db -h\nUsage: coin-clip build-db [OPTIONS]\n\n Extract vectors from a candidate image set and build a search engine based\n on it.\n\nOptions:\n -m, --model-name TEXT Model Name; either local path or huggingface\n model name [default: breezedeus/coin-clip-vit-\n base-patch32]\n -d, --device TEXT ['cpu', 'cuda']; Either 'cpu' or 'gpu', or\n specify a specific GPU like 'cuda:0'. Default is\n 'cpu'. [default: cpu]\n -i, --input-image-dir TEXT Folder with Coin Images to be indexed. [required]\n -o, --output-db-dir TEXT Folder where the built search engine is stored.\n [default: ./coin_clip_chroma.db]\n -h, --help Show this message and exit.\n```\n\nFor instance, \n\n```bash\n$ coin-clip build-db -i examples -o coin_clip_chroma.db\n```\n\n### Querying\nAfter building the vector search engine with the above command, you can use the `coin-clip retrieve` command to retrieve the coin images \ud83e\ude99 most similar to a specified coin image.\n\n```bash\n$ coin-clip retrieve -h\nUsage: coin-clip retrieve [OPTIONS]\n\n Retrieve images from the search engine, based on the query image.\n\nOptions:\n -m, --model-name TEXT Model Name; either local path or huggingface model\n name [default: breezedeus/coin-clip-vit-base-\n patch32]\n -d, --device TEXT ['cpu', 'cuda']; Either 'cpu' or 'gpu', or specify a\n specific GPU like 'cuda:0'. Default is 'cpu'.\n [default: cpu]\n --db-dir TEXT Folder where the built search engine is stored.\n [default: ./coin_clip_chroma.db]\n -i, --image-fp TEXT Image Path to retrieve [required]\n -h, --help Show this message and exit.\n```\n\nFor instance, \n\n```bash\n$ coin-clip retrieve --db-dir coin_clip_chroma.db -i examples/10_back.jpg\n```\n\n\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "Enhancing Coin Image Retrieval with CLIP",
"version": "0.1",
"project_urls": {
"Homepage": "https://github.com/breezedeus/coin-clip"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8c27db2140dde6d01ec48f7444aa094b1d9db3a4e763fcdb604a2557414c316c",
"md5": "376c45c674b093988d071269ba23fb22",
"sha256": "760bd66cc291cb2aa100333b4392659a4b2c37902d3258d0e7a9ca2782591a46"
},
"downloads": -1,
"filename": "coin_clip-0.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "376c45c674b093988d071269ba23fb22",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 15927,
"upload_time": "2023-12-06T13:38:05",
"upload_time_iso_8601": "2023-12-06T13:38:05.553951Z",
"url": "https://files.pythonhosted.org/packages/8c/27/db2140dde6d01ec48f7444aa094b1d9db3a4e763fcdb604a2557414c316c/coin_clip-0.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "e9b47fcd054569570b25a0b684a199b03460306f95110e55efb1bc630afa9d40",
"md5": "386499dda76e14fb5e4fdfeee2d43ad8",
"sha256": "b23e7830875a88693dd2d826712184b5150b6306cfd1a9e0021082b2a561071e"
},
"downloads": -1,
"filename": "coin_clip-0.1.tar.gz",
"has_sig": false,
"md5_digest": "386499dda76e14fb5e4fdfeee2d43ad8",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 12610,
"upload_time": "2023-12-06T13:38:07",
"upload_time_iso_8601": "2023-12-06T13:38:07.847349Z",
"url": "https://files.pythonhosted.org/packages/e9/b4/7fcd054569570b25a0b684a199b03460306f95110e55efb1bc630afa9d40/coin_clip-0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-12-06 13:38:07",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "breezedeus",
"github_project": "coin-clip",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "annotated-types",
"specs": [
[
"==",
"0.6.0"
]
]
},
{
"name": "anyio",
"specs": [
[
"==",
"3.7.1"
]
]
},
{
"name": "asgiref",
"specs": [
[
"==",
"3.7.2"
]
]
},
{
"name": "backoff",
"specs": [
[
"==",
"2.2.1"
]
]
},
{
"name": "bcrypt",
"specs": [
[
"==",
"4.1.1"
]
]
},
{
"name": "cachetools",
"specs": [
[
"==",
"5.3.2"
]
]
},
{
"name": "certifi",
"specs": [
[
"==",
"2023.11.17"
]
]
},
{
"name": "charset-normalizer",
"specs": [
[
"==",
"3.3.2"
]
]
},
{
"name": "chroma-hnswlib",
"specs": [
[
"==",
"0.7.3"
]
]
},
{
"name": "chromadb",
"specs": [
[
"==",
"0.4.18"
]
]
},
{
"name": "click",
"specs": [
[
"==",
"8.1.7"
]
]
},
{
"name": "coloredlogs",
"specs": [
[
"==",
"15.0.1"
]
]
},
{
"name": "colorlog",
"specs": [
[
"==",
"6.8.0"
]
]
},
{
"name": "deprecated",
"specs": [
[
"==",
"1.2.14"
]
]
},
{
"name": "exceptiongroup",
"specs": [
[
"==",
"1.2.0"
]
]
},
{
"name": "fastapi",
"specs": [
[
"==",
"0.104.1"
]
]
},
{
"name": "filelock",
"specs": [
[
"==",
"3.13.1"
]
]
},
{
"name": "flatbuffers",
"specs": [
[
"==",
"23.5.26"
]
]
},
{
"name": "fsspec",
"specs": [
[
"==",
"2023.12.1"
]
]
},
{
"name": "google-auth",
"specs": [
[
"==",
"2.25.1"
]
]
},
{
"name": "googleapis-common-protos",
"specs": [
[
"==",
"1.61.0"
]
]
},
{
"name": "grpcio",
"specs": [
[
"==",
"1.59.3"
]
]
},
{
"name": "h11",
"specs": [
[
"==",
"0.14.0"
]
]
},
{
"name": "httptools",
"specs": [
[
"==",
"0.6.1"
]
]
},
{
"name": "huggingface-hub",
"specs": [
[
"==",
"0.19.4"
]
]
},
{
"name": "humanfriendly",
"specs": [
[
"==",
"10.0"
]
]
},
{
"name": "idna",
"specs": [
[
"==",
"3.6"
]
]
},
{
"name": "importlib-metadata",
"specs": [
[
"==",
"6.11.0"
]
]
},
{
"name": "importlib-resources",
"specs": [
[
"==",
"6.1.1"
]
]
},
{
"name": "jinja2",
"specs": [
[
"==",
"3.1.2"
]
]
},
{
"name": "kubernetes",
"specs": [
[
"==",
"28.1.0"
]
]
},
{
"name": "markupsafe",
"specs": [
[
"==",
"2.1.3"
]
]
},
{
"name": "mmh3",
"specs": [
[
"==",
"4.0.1"
]
]
},
{
"name": "monotonic",
"specs": [
[
"==",
"1.6"
]
]
},
{
"name": "mpmath",
"specs": [
[
"==",
"1.3.0"
]
]
},
{
"name": "networkx",
"specs": [
[
"==",
"3.2.1"
]
]
},
{
"name": "numpy",
"specs": [
[
"==",
"1.26.2"
]
]
},
{
"name": "oauthlib",
"specs": [
[
"==",
"3.2.2"
]
]
},
{
"name": "onnxruntime",
"specs": [
[
"==",
"1.16.3"
]
]
},
{
"name": "opentelemetry-api",
"specs": [
[
"==",
"1.21.0"
]
]
},
{
"name": "opentelemetry-exporter-otlp-proto-common",
"specs": [
[
"==",
"1.21.0"
]
]
},
{
"name": "opentelemetry-exporter-otlp-proto-grpc",
"specs": [
[
"==",
"1.21.0"
]
]
},
{
"name": "opentelemetry-instrumentation",
"specs": [
[
"==",
"0.42b0"
]
]
},
{
"name": "opentelemetry-instrumentation-asgi",
"specs": [
[
"==",
"0.42b0"
]
]
},
{
"name": "opentelemetry-instrumentation-fastapi",
"specs": [
[
"==",
"0.42b0"
]
]
},
{
"name": "opentelemetry-proto",
"specs": [
[
"==",
"1.21.0"
]
]
},
{
"name": "opentelemetry-sdk",
"specs": [
[
"==",
"1.21.0"
]
]
},
{
"name": "opentelemetry-semantic-conventions",
"specs": [
[
"==",
"0.42b0"
]
]
},
{
"name": "opentelemetry-util-http",
"specs": [
[
"==",
"0.42b0"
]
]
},
{
"name": "overrides",
"specs": [
[
"==",
"7.4.0"
]
]
},
{
"name": "packaging",
"specs": [
[
"==",
"23.2"
]
]
},
{
"name": "pillow",
"specs": [
[
"==",
"10.1.0"
]
]
},
{
"name": "posthog",
"specs": [
[
"==",
"3.1.0"
]
]
},
{
"name": "protobuf",
"specs": [
[
"==",
"4.25.1"
]
]
},
{
"name": "pulsar-client",
"specs": [
[
"==",
"3.3.0"
]
]
},
{
"name": "pyasn1",
"specs": [
[
"==",
"0.5.1"
]
]
},
{
"name": "pyasn1-modules",
"specs": [
[
"==",
"0.3.0"
]
]
},
{
"name": "pydantic",
"specs": [
[
"==",
"2.5.2"
]
]
},
{
"name": "pydantic-core",
"specs": [
[
"==",
"2.14.5"
]
]
},
{
"name": "pypika",
"specs": [
[
"==",
"0.48.9"
]
]
},
{
"name": "python-dateutil",
"specs": [
[
"==",
"2.8.2"
]
]
},
{
"name": "python-dotenv",
"specs": [
[
"==",
"1.0.0"
]
]
},
{
"name": "pyyaml",
"specs": [
[
"==",
"6.0.1"
]
]
},
{
"name": "regex",
"specs": [
[
"==",
"2023.10.3"
]
]
},
{
"name": "requests",
"specs": [
[
"==",
"2.31.0"
]
]
},
{
"name": "requests-oauthlib",
"specs": [
[
"==",
"1.3.1"
]
]
},
{
"name": "rsa",
"specs": [
[
"==",
"4.9"
]
]
},
{
"name": "safetensors",
"specs": [
[
"==",
"0.4.1"
]
]
},
{
"name": "six",
"specs": [
[
"==",
"1.16.0"
]
]
},
{
"name": "sniffio",
"specs": [
[
"==",
"1.3.0"
]
]
},
{
"name": "starlette",
"specs": [
[
"==",
"0.27.0"
]
]
},
{
"name": "sympy",
"specs": [
[
"==",
"1.12"
]
]
},
{
"name": "tenacity",
"specs": [
[
"==",
"8.2.3"
]
]
},
{
"name": "tokenizers",
"specs": [
[
"==",
"0.15.0"
]
]
},
{
"name": "torch",
"specs": [
[
"==",
"2.1.1"
]
]
},
{
"name": "torchvision",
"specs": [
[
"==",
"0.16.1"
]
]
},
{
"name": "tqdm",
"specs": [
[
"==",
"4.66.1"
]
]
},
{
"name": "transformers",
"specs": [
[
"==",
"4.35.2"
]
]
},
{
"name": "typer",
"specs": [
[
"==",
"0.9.0"
]
]
},
{
"name": "typing-extensions",
"specs": [
[
"==",
"4.8.0"
]
]
},
{
"name": "urllib3",
"specs": [
[
"==",
"1.26.18"
]
]
},
{
"name": "uvicorn",
"specs": [
[
"==",
"0.24.0.post1"
]
]
},
{
"name": "uvloop",
"specs": [
[
"==",
"0.19.0"
]
]
},
{
"name": "watchfiles",
"specs": [
[
"==",
"0.21.0"
]
]
},
{
"name": "websocket-client",
"specs": [
[
"==",
"1.7.0"
]
]
},
{
"name": "websockets",
"specs": [
[
"==",
"12.0"
]
]
},
{
"name": "wrapt",
"specs": [
[
"==",
"1.16.0"
]
]
},
{
"name": "zipp",
"specs": [
[
"==",
"3.17.0"
]
]
}
],
"lcname": "coin-clip"
}
