all-clip


Nameall-clip JSON
Version 1.2.0 PyPI version JSON
download
home_pagehttps://github.com/rom1504/all_clip
SummaryLoad any clip model with a standardized interface
upload_time2024-01-21 21:40:28
maintainer
docs_urlNone
authorRomain Beaumont
requires_python
licenseMIT
keywords machine learning
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # all_clip
[![pypi](https://img.shields.io/pypi/v/all_clip.svg)](https://pypi.python.org/pypi/all_clip)
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/rom1504/all_clip/blob/master/notebook/all_clip_getting_started.ipynb)
[![Try it on gitpod](https://img.shields.io/badge/try-on%20gitpod-brightgreen.svg)](https://gitpod.io/#https://github.com/rom1504/all_clip)

Load any clip model with a standardized interface

## Install

pip install all_clip

## Python examples

```python
from all_clip import load_clip
import torch
from PIL import Image
import pathlib


model, preprocess, tokenizer = load_clip("open_clip:ViT-B-32/laion2b_s34b_b79k", device="cpu", use_jit=False)


image = preprocess(Image.open(str(pathlib.Path(__file__).parent.resolve()) + "/CLIP.png")).unsqueeze(0)
text = tokenizer(["a diagram", "a dog", "a cat"])

with torch.no_grad(), torch.cuda.amp.autocast():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)

    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print("Label probs:", text_probs)  # prints: [[1., 0., 0.]]
```

Checkout these examples to call this as a lib:
* [example.py](examples/example.py)

## API

This module exposes a single function `load_clip`:

* **clip_model** CLIP model to load (default *ViT-B/32*). See below supported models section.
* **use_jit** uses jit for the clip model (default *True*)
* **warmup_batch_size** warmup batch size (default *1*)
* **clip_cache_path** cache path for clip (default *None*)
* **device** device (default *None*)

## Related projects

* [clip-retrieval](https://github.com/rom1504/clip-retrieval) to use clip for inference, and retrieval
* [open_clip](https://github.com/mlfoundations/open_clip) to train clip models
* [CLIP_benchmark](https://github.com/LAION-AI/CLIP_benchmark) to evaluate clip models

## Supported models

### OpenAI

Specify the model as "ViT-B-32"

### Openclip

`"open_clip:ViT-B-32/laion2b_s34b_b79k"` to use the [open_clip](https://github.com/mlfoundations/open_clip)

### HF CLIP

`"hf_clip:patrickjohncyh/fashion-clip"` to use the [hugging face](https://huggingface.co/docs/transformers/model_doc/clip)

### Deepsparse backend

[DeepSparse](https://github.com/neuralmagic/deepsparse) is an inference runtime for fast sparse model inference on CPUs. There is a backend available within clip-retrieval by installing it with `pip install deepsparse-nightly[clip]`, and specifying a `clip_model` with a prepended `"nm:"`, such as [`"nm:neuralmagic/CLIP-ViT-B-32-256x256-DataComp-s34B-b86K-quant-ds"`](https://huggingface.co/neuralmagic/CLIP-ViT-B-32-256x256-DataComp-s34B-b86K-quant-ds) or [`"nm:mgoin/CLIP-ViT-B-32-laion2b_s34b_b79k-ds"`](https://huggingface.co/mgoin/CLIP-ViT-B-32-laion2b_s34b_b79k-ds).

### Japanese clip

[japanese-clip](https://github.com/rinnakk/japanese-clip) provides some models for japanese.
For example one is `ja_clip:rinna/japanese-clip-vit-b-16`

## For development

Either locally, or in [gitpod](https://gitpod.io/#https://github.com/rom1504/all_clip) (do `export PIP_USER=false` there)

Setup a virtualenv:

```
python3 -m venv .env
source .env/bin/activate
pip install -e .
```

to run tests:
```
pip install -r requirements-test.txt
```
then 
```
make lint
make test
```

You can use `make black` to reformat the code

`python -m pytest -x -s -v tests -k "ja_clip"` to run a specific test



            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/rom1504/all_clip",
    "name": "all-clip",
    "maintainer": "",
    "docs_url": null,
    "requires_python": "",
    "maintainer_email": "",
    "keywords": "machine learning",
    "author": "Romain Beaumont",
    "author_email": "romain.rom1@gmail.com",
    "download_url": "https://files.pythonhosted.org/packages/fc/18/ea63cebb0313a8f0120b2e65603cc034edda4b51852ce2eb1fd604b230a9/all_clip-1.2.0.tar.gz",
    "platform": null,
    "description": "# all_clip\n[![pypi](https://img.shields.io/pypi/v/all_clip.svg)](https://pypi.python.org/pypi/all_clip)\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/rom1504/all_clip/blob/master/notebook/all_clip_getting_started.ipynb)\n[![Try it on gitpod](https://img.shields.io/badge/try-on%20gitpod-brightgreen.svg)](https://gitpod.io/#https://github.com/rom1504/all_clip)\n\nLoad any clip model with a standardized interface\n\n## Install\n\npip install all_clip\n\n## Python examples\n\n```python\nfrom all_clip import load_clip\nimport torch\nfrom PIL import Image\nimport pathlib\n\n\nmodel, preprocess, tokenizer = load_clip(\"open_clip:ViT-B-32/laion2b_s34b_b79k\", device=\"cpu\", use_jit=False)\n\n\nimage = preprocess(Image.open(str(pathlib.Path(__file__).parent.resolve()) + \"/CLIP.png\")).unsqueeze(0)\ntext = tokenizer([\"a diagram\", \"a dog\", \"a cat\"])\n\nwith torch.no_grad(), torch.cuda.amp.autocast():\n    image_features = model.encode_image(image)\n    text_features = model.encode_text(text)\n    image_features /= image_features.norm(dim=-1, keepdim=True)\n    text_features /= text_features.norm(dim=-1, keepdim=True)\n\n    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)\n\nprint(\"Label probs:\", text_probs)  # prints: [[1., 0., 0.]]\n```\n\nCheckout these examples to call this as a lib:\n* [example.py](examples/example.py)\n\n## API\n\nThis module exposes a single function `load_clip`:\n\n* **clip_model** CLIP model to load (default *ViT-B/32*). See below supported models section.\n* **use_jit** uses jit for the clip model (default *True*)\n* **warmup_batch_size** warmup batch size (default *1*)\n* **clip_cache_path** cache path for clip (default *None*)\n* **device** device (default *None*)\n\n## Related projects\n\n* [clip-retrieval](https://github.com/rom1504/clip-retrieval) to use clip for inference, and retrieval\n* [open_clip](https://github.com/mlfoundations/open_clip) to train clip models\n* [CLIP_benchmark](https://github.com/LAION-AI/CLIP_benchmark) to evaluate clip models\n\n## Supported models\n\n### OpenAI\n\nSpecify the model as \"ViT-B-32\"\n\n### Openclip\n\n`\"open_clip:ViT-B-32/laion2b_s34b_b79k\"` to use the [open_clip](https://github.com/mlfoundations/open_clip)\n\n### HF CLIP\n\n`\"hf_clip:patrickjohncyh/fashion-clip\"` to use the [hugging face](https://huggingface.co/docs/transformers/model_doc/clip)\n\n### Deepsparse backend\n\n[DeepSparse](https://github.com/neuralmagic/deepsparse) is an inference runtime for fast sparse model inference on CPUs. There is a backend available within clip-retrieval by installing it with `pip install deepsparse-nightly[clip]`, and specifying a `clip_model` with a prepended `\"nm:\"`, such as [`\"nm:neuralmagic/CLIP-ViT-B-32-256x256-DataComp-s34B-b86K-quant-ds\"`](https://huggingface.co/neuralmagic/CLIP-ViT-B-32-256x256-DataComp-s34B-b86K-quant-ds) or [`\"nm:mgoin/CLIP-ViT-B-32-laion2b_s34b_b79k-ds\"`](https://huggingface.co/mgoin/CLIP-ViT-B-32-laion2b_s34b_b79k-ds).\n\n### Japanese clip\n\n[japanese-clip](https://github.com/rinnakk/japanese-clip) provides some models for japanese.\nFor example one is `ja_clip:rinna/japanese-clip-vit-b-16`\n\n## For development\n\nEither locally, or in [gitpod](https://gitpod.io/#https://github.com/rom1504/all_clip) (do `export PIP_USER=false` there)\n\nSetup a virtualenv:\n\n```\npython3 -m venv .env\nsource .env/bin/activate\npip install -e .\n```\n\nto run tests:\n```\npip install -r requirements-test.txt\n```\nthen \n```\nmake lint\nmake test\n```\n\nYou can use `make black` to reformat the code\n\n`python -m pytest -x -s -v tests -k \"ja_clip\"` to run a specific test\n\n\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "Load any clip model with a standardized interface",
    "version": "1.2.0",
    "project_urls": {
        "Homepage": "https://github.com/rom1504/all_clip"
    },
    "split_keywords": [
        "machine",
        "learning"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "a88fa6a77fc6b320e1a9b6e0f76f4a1780a16e9e0ee14b03412e34f742e889f1",
                "md5": "2591d6e53581490454e564f58e878041",
                "sha256": "cbbff134f8e8fed2cf5066dda62e209e535f7ce63c9e201876cc7683dee85540"
            },
            "downloads": -1,
            "filename": "all_clip-1.2.0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "2591d6e53581490454e564f58e878041",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 10998,
            "upload_time": "2024-01-21T21:40:26",
            "upload_time_iso_8601": "2024-01-21T21:40:26.947941Z",
            "url": "https://files.pythonhosted.org/packages/a8/8f/a6a77fc6b320e1a9b6e0f76f4a1780a16e9e0ee14b03412e34f742e889f1/all_clip-1.2.0-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "fc18ea63cebb0313a8f0120b2e65603cc034edda4b51852ce2eb1fd604b230a9",
                "md5": "43ed1f0b4943cc38e641818d6dd14b9b",
                "sha256": "ca38e50567b511c0eb2b82159bea11a4fac3097254cb1456a7b2118da4cc2390"
            },
            "downloads": -1,
            "filename": "all_clip-1.2.0.tar.gz",
            "has_sig": false,
            "md5_digest": "43ed1f0b4943cc38e641818d6dd14b9b",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 8997,
            "upload_time": "2024-01-21T21:40:28",
            "upload_time_iso_8601": "2024-01-21T21:40:28.069874Z",
            "url": "https://files.pythonhosted.org/packages/fc/18/ea63cebb0313a8f0120b2e65603cc034edda4b51852ce2eb1fd604b230a9/all_clip-1.2.0.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-01-21 21:40:28",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "rom1504",
    "github_project": "all_clip",
    "github_not_found": true,
    "lcname": "all-clip"
}
        
Elapsed time: 0.33175s