# all_clip
[](https://pypi.python.org/pypi/all_clip)
[](https://colab.research.google.com/github/rom1504/all_clip/blob/master/notebook/all_clip_getting_started.ipynb)
[](https://gitpod.io/#https://github.com/rom1504/all_clip)
Load any clip model with a standardized interface
## Install
pip install all_clip
## Python examples
```python
from all_clip import load_clip
import torch
from PIL import Image
import pathlib
model, preprocess, tokenizer = load_clip("open_clip:ViT-B-32/laion2b_s34b_b79k", device="cpu", use_jit=False)
image = preprocess(Image.open(str(pathlib.Path(__file__).parent.resolve()) + "/CLIP.png")).unsqueeze(0)
text = tokenizer(["a diagram", "a dog", "a cat"])
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs) # prints: [[1., 0., 0.]]
```
Checkout these examples to call this as a lib:
* [example.py](examples/example.py)
## API
This module exposes a single function `load_clip`:
* **clip_model** CLIP model to load (default *ViT-B/32*). See below supported models section.
* **use_jit** uses jit for the clip model (default *True*)
* **warmup_batch_size** warmup batch size (default *1*)
* **clip_cache_path** cache path for clip (default *None*)
* **device** device (default *None*)
## Related projects
* [clip-retrieval](https://github.com/rom1504/clip-retrieval) to use clip for inference, and retrieval
* [open_clip](https://github.com/mlfoundations/open_clip) to train clip models
* [CLIP_benchmark](https://github.com/LAION-AI/CLIP_benchmark) to evaluate clip models
## Supported models
### OpenAI
Specify the model as "ViT-B-32"
### Openclip
`"open_clip:ViT-B-32/laion2b_s34b_b79k"` to use the [open_clip](https://github.com/mlfoundations/open_clip)
### HF CLIP
`"hf_clip:patrickjohncyh/fashion-clip"` to use the [hugging face](https://huggingface.co/docs/transformers/model_doc/clip)
### Deepsparse backend
[DeepSparse](https://github.com/neuralmagic/deepsparse) is an inference runtime for fast sparse model inference on CPUs. There is a backend available within clip-retrieval by installing it with `pip install deepsparse-nightly[clip]`, and specifying a `clip_model` with a prepended `"nm:"`, such as [`"nm:neuralmagic/CLIP-ViT-B-32-256x256-DataComp-s34B-b86K-quant-ds"`](https://huggingface.co/neuralmagic/CLIP-ViT-B-32-256x256-DataComp-s34B-b86K-quant-ds) or [`"nm:mgoin/CLIP-ViT-B-32-laion2b_s34b_b79k-ds"`](https://huggingface.co/mgoin/CLIP-ViT-B-32-laion2b_s34b_b79k-ds).
### Japanese clip
[japanese-clip](https://github.com/rinnakk/japanese-clip) provides some models for japanese.
For example one is `ja_clip:rinna/japanese-clip-vit-b-16`
## For development
Either locally, or in [gitpod](https://gitpod.io/#https://github.com/rom1504/all_clip) (do `export PIP_USER=false` there)
Setup a virtualenv:
```
python3 -m venv .env
source .env/bin/activate
pip install -e .
```
to run tests:
```
pip install -r requirements-test.txt
```
then
```
make lint
make test
```
You can use `make black` to reformat the code
`python -m pytest -x -s -v tests -k "ja_clip"` to run a specific test
Raw data
{
"_id": null,
"home_page": "https://github.com/rom1504/all_clip",
"name": "all-clip",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "machine learning",
"author": "Romain Beaumont",
"author_email": "romain.rom1@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/fc/18/ea63cebb0313a8f0120b2e65603cc034edda4b51852ce2eb1fd604b230a9/all_clip-1.2.0.tar.gz",
"platform": null,
"description": "# all_clip\n[](https://pypi.python.org/pypi/all_clip)\n[](https://colab.research.google.com/github/rom1504/all_clip/blob/master/notebook/all_clip_getting_started.ipynb)\n[](https://gitpod.io/#https://github.com/rom1504/all_clip)\n\nLoad any clip model with a standardized interface\n\n## Install\n\npip install all_clip\n\n## Python examples\n\n```python\nfrom all_clip import load_clip\nimport torch\nfrom PIL import Image\nimport pathlib\n\n\nmodel, preprocess, tokenizer = load_clip(\"open_clip:ViT-B-32/laion2b_s34b_b79k\", device=\"cpu\", use_jit=False)\n\n\nimage = preprocess(Image.open(str(pathlib.Path(__file__).parent.resolve()) + \"/CLIP.png\")).unsqueeze(0)\ntext = tokenizer([\"a diagram\", \"a dog\", \"a cat\"])\n\nwith torch.no_grad(), torch.cuda.amp.autocast():\n image_features = model.encode_image(image)\n text_features = model.encode_text(text)\n image_features /= image_features.norm(dim=-1, keepdim=True)\n text_features /= text_features.norm(dim=-1, keepdim=True)\n\n text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)\n\nprint(\"Label probs:\", text_probs) # prints: [[1., 0., 0.]]\n```\n\nCheckout these examples to call this as a lib:\n* [example.py](examples/example.py)\n\n## API\n\nThis module exposes a single function `load_clip`:\n\n* **clip_model** CLIP model to load (default *ViT-B/32*). See below supported models section.\n* **use_jit** uses jit for the clip model (default *True*)\n* **warmup_batch_size** warmup batch size (default *1*)\n* **clip_cache_path** cache path for clip (default *None*)\n* **device** device (default *None*)\n\n## Related projects\n\n* [clip-retrieval](https://github.com/rom1504/clip-retrieval) to use clip for inference, and retrieval\n* [open_clip](https://github.com/mlfoundations/open_clip) to train clip models\n* [CLIP_benchmark](https://github.com/LAION-AI/CLIP_benchmark) to evaluate clip models\n\n## Supported models\n\n### OpenAI\n\nSpecify the model as \"ViT-B-32\"\n\n### Openclip\n\n`\"open_clip:ViT-B-32/laion2b_s34b_b79k\"` to use the [open_clip](https://github.com/mlfoundations/open_clip)\n\n### HF CLIP\n\n`\"hf_clip:patrickjohncyh/fashion-clip\"` to use the [hugging face](https://huggingface.co/docs/transformers/model_doc/clip)\n\n### Deepsparse backend\n\n[DeepSparse](https://github.com/neuralmagic/deepsparse) is an inference runtime for fast sparse model inference on CPUs. There is a backend available within clip-retrieval by installing it with `pip install deepsparse-nightly[clip]`, and specifying a `clip_model` with a prepended `\"nm:\"`, such as [`\"nm:neuralmagic/CLIP-ViT-B-32-256x256-DataComp-s34B-b86K-quant-ds\"`](https://huggingface.co/neuralmagic/CLIP-ViT-B-32-256x256-DataComp-s34B-b86K-quant-ds) or [`\"nm:mgoin/CLIP-ViT-B-32-laion2b_s34b_b79k-ds\"`](https://huggingface.co/mgoin/CLIP-ViT-B-32-laion2b_s34b_b79k-ds).\n\n### Japanese clip\n\n[japanese-clip](https://github.com/rinnakk/japanese-clip) provides some models for japanese.\nFor example one is `ja_clip:rinna/japanese-clip-vit-b-16`\n\n## For development\n\nEither locally, or in [gitpod](https://gitpod.io/#https://github.com/rom1504/all_clip) (do `export PIP_USER=false` there)\n\nSetup a virtualenv:\n\n```\npython3 -m venv .env\nsource .env/bin/activate\npip install -e .\n```\n\nto run tests:\n```\npip install -r requirements-test.txt\n```\nthen \n```\nmake lint\nmake test\n```\n\nYou can use `make black` to reformat the code\n\n`python -m pytest -x -s -v tests -k \"ja_clip\"` to run a specific test\n\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Load any clip model with a standardized interface",
"version": "1.2.0",
"project_urls": {
"Homepage": "https://github.com/rom1504/all_clip"
},
"split_keywords": [
"machine",
"learning"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a88fa6a77fc6b320e1a9b6e0f76f4a1780a16e9e0ee14b03412e34f742e889f1",
"md5": "2591d6e53581490454e564f58e878041",
"sha256": "cbbff134f8e8fed2cf5066dda62e209e535f7ce63c9e201876cc7683dee85540"
},
"downloads": -1,
"filename": "all_clip-1.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "2591d6e53581490454e564f58e878041",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 10998,
"upload_time": "2024-01-21T21:40:26",
"upload_time_iso_8601": "2024-01-21T21:40:26.947941Z",
"url": "https://files.pythonhosted.org/packages/a8/8f/a6a77fc6b320e1a9b6e0f76f4a1780a16e9e0ee14b03412e34f742e889f1/all_clip-1.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "fc18ea63cebb0313a8f0120b2e65603cc034edda4b51852ce2eb1fd604b230a9",
"md5": "43ed1f0b4943cc38e641818d6dd14b9b",
"sha256": "ca38e50567b511c0eb2b82159bea11a4fac3097254cb1456a7b2118da4cc2390"
},
"downloads": -1,
"filename": "all_clip-1.2.0.tar.gz",
"has_sig": false,
"md5_digest": "43ed1f0b4943cc38e641818d6dd14b9b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 8997,
"upload_time": "2024-01-21T21:40:28",
"upload_time_iso_8601": "2024-01-21T21:40:28.069874Z",
"url": "https://files.pythonhosted.org/packages/fc/18/ea63cebb0313a8f0120b2e65603cc034edda4b51852ce2eb1fd604b230a9/all_clip-1.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-21 21:40:28",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "rom1504",
"github_project": "all_clip",
"github_not_found": true,
"lcname": "all-clip"
}
JSON
