# CLIP Benchmark
[](https://pypi.python.org/pypi/clip_benchmark)
The goal of this repo is to evaluate CLIP-like models on a standard set
of datasets on different tasks such as zero-shot classification and zero-shot
retrieval, and captioning.
Below we show the average rank (1 is the best, lower is better) of different CLIP models, evaluated
on different datasets.
The current detailed results of the benchmark can be seen [here](benchmark/README.md)
or directly in the [notebook](benchmark/results.ipynb).
## Features
* Support for zero-shot classification and zero-shot retrieval, linear probing, and captioning.
* Support for [OpenCLIP](https://github.com/mlfoundations/open_clip) pre-trained models, [Japanese CLIP](https://github.com/rinnakk/japanese-clip), and [NLLB CLIP](https://arxiv.org/abs/2309.01859) for general multilingual abilities.
* Support various datasets from [torchvision](https://pytorch.org/vision/stable/datasets.html), [tensorflow datasets](https://www.tensorflow.org/datasets), and [VTAB](https://github.com/google-research/task_adaptation).
* Support for various multilingual datasets for classification and retrieval
* Support for compositionality tasks
## How to install?
`pip install clip-benchmark`
## How to use?
To evaluate we recommend to create a models.txt like
```
ViT-B-32,openai
```
to get the list of datasets
```
wget https://raw.githubusercontent.com/LAION-AI/CLIP_benchmark/main/benchmark/webdatasets.txt
```
Then to run
```
clip_benchmark eval --pretrained_model models.txt \
--dataset "webdatasets.txt" \
--dataset_root "https://huggingface.co/datasets/clip-benchmark/wds_{dataset_cleaned}/tree/main" \
--output "benchmark_{dataset}_{pretrained}_{model}_{language}_{task}.json"
```
Then to get the full table
```
clip_benchmark build benchmark_*.json --output benchmark.csv
```
### Command line interface (CLI)
The easiest way to benchmark the models is using the CLI, `clip_benchmark`.
You can specify the model to use, the dataset and the task to evaluate on. Once it is done, evaluation is performed and
the results are written into a JSON file.
### Using other models than openclip
It is possible to use other models than openclip ones. For example japanese-clip is supported
Here is an example of use
```
>>> python3 clip_benchmark/cli.py eval \
--model_type "ja_clip" \ # flag to use japanese-clip
--pretrained "rinna/japanese-cloob-vit-b-16" \ # now, we have `rinna/japanese-cloob-vit-b-16` or `rinna/japanese-clip-vit-b-16`.
--language "jp" \
--task "zeroshot_classification" \
--dataset "imagenet1k" \
--dataset_root {ROOT_PATH}
>>> cat result.json
{"dataset": "imagenet1k", "model": "ViT-B-32-quickgelu", "pretrained": "rinna/japanese-cloob-vit-b-16", "task": "zeroshot_classification", "metrics": {"acc1": 0.54636, "acc5": 0.72856, "mean_per_class_recall": 0.54522}, "language": "jp"}
```
### How to add other CLIP models
Please follow these steps:
1. Add a identity file to load model in `clip_benchmark/models`
2. Define a loading function, that returns a tuple (model, transform, tokenizer). Please see `clip_benchmark/models/open_clip.py` as an example.
3. Add the function into `TYPE2FUNC` in `clip_benchmark/models/__init__.py`
Remarks:
- The new tokenizer/model must enable to do the following things as https://github.com/openai/CLIP#usage
- `tokenizer(texts).to(device)` ... `texts` is a list of string
- `model.encode_text(tokenized_texts)` ... `tokenized_texts` is a output from `tokenizer(texts).to(device)`
- `model.encode_image(images)` ... `images` is a image tensor by the `transform`
### CIFAR-10 example
Here is an example for CIFAR-10 zero-shot classification using OpenCLIP's pre-trained model on LAION-400m:
`clip_benchmark eval --dataset=cifar10 --task=zeroshot_classification --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64`
By default, the dataset is downloaded into `--dataset_root`, which by default is `root`.
Here is the content of `result.json` after the evaluation is done:
```json
{
"dataset": "cifar10", "model": "ViT-B-32-quickgelu",
"pretrained": "laion400m_e32", "task": "zeroshot_classification",
"metrics": {"acc1": 0.9074, "acc5": 0.998}
}
```
### VOC2007 example
Here is another example with VOC2007, which is a multi-label classification dataset.
`clip_benchmark eval --dataset=voc2007_multilabel --task=zeroshot_classification --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64`
Here is the content of `result.json` after the evaluation is done:
```json
{"dataset": "voc2007_multilabel", "model": "ViT-B-32-quickgelu", "pretrained": "laion400m_e32", "task": "zeroshot_classification", "metrics": {"mean_average_precision": 0.7627869844436646}}
```
Here, we compute the mean average precision or mAP, more details about that metric [here](https://fangdahan.medium.com/calculate-mean-average-precision-map-for-multi-label-classification-b082679d31be) in the context of multi-label classification.
### VTAB example
Here is an example on how to run it on [VTAB](https://github.com/google-research/task_adaptation) classification tasks.
First, you need to install VTAB's dedicated package.
`pip install task_adaptation==0.1`
Then, you can run it by providing the full dataset name.
Example with `eurosat`:
`clip_benchmark eval --dataset=vtab/eurosat --task=zeroshot_classification --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64`
See [clip_benchmark/datasets/builder.py#L634](clip_benchmark/datasets/builder.py#L634) for the full list of
VTAB dataset collection.
### TensorFlow dataset example
Here is an example on how to run it on [Tensorflow datasets](https://www.tensorflow.org/datasets).
First, you need to install `tfds-nightly` and `timm`.
`pip install timm tfds-nightly`
The name of the dataset follows the template `tfds/<DATASET_NAME>`.
Example with `cifar10`:
`clip_benchmark eval --dataset=tfds/cifar10 --task=zeroshot_classification --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64`
### COCO captions retrieval example
Here is an example for COCO captions zero-shot retrieval:
`clip_benchmark eval --dataset=mscoco_captions --task=zeroshot_retrieval --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64`
Note that for using COCO, you also need to install `pycocotools` (e.g., using `pip install pycocotools`).
### COCO captions captioning example
Here is an example for COCO captions captioning task:
`clip_benchmark eval --dataset=mscoco_captions --task=captioning --model=coca_ViT-L-14 --output=result.json --pretrained mscoco_finetuned_laion2b_s13b_b90k`
Note that for using COCO, you also need to install `pycocotools` (e.g., using `pip install pycocotools`) and `pycocoevalcap`.
### Linear probing example
Full linear probing on train split, evaluate on test split:
`clip_benchmark eval --dataset=cifar10 --task=linear_probe --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64 --fewshot_lr 0.1 --fewshot_epochs 20 --batch_size 512 --train_split train --test_split test`
few-shot (k=5) linear probing on train split, evaluate on test split:
`clip_benchmark eval --dataset=cifar10 --task=linear_probe --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64 --fewshot_k 5 --fewshot_lr 0.1 --fewshot_epochs 20 --batch_size 512 --train_split train --test_split test`
Split train into train (90%), val (10%), do linear probing on train split, tune on val split, evaluate on test split:
`clip_benchmark eval --dataset=cifar10 --task=linear_probe --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64 --fewshot_lr 0.1 --fewshot_epochs 20 --batch_size 512 --train_split train --val_proportion 0.1 --test_split test`
For other datasets that have an official val split, one can also specify the val split:
`clip_benchmark eval --dataset=fgvc_aircraft --task=linear_probe --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64 --fewshot_lr 0.1 --fewshot_epochs 20 --batch_size 512 --train_split train --val_split val --test_split test`
### Multilingual evaluation
We also provide datasets for evaluating multilingual models (see e.g. https://github.com/mlfoundations/open_clip#vit-b32-xlm-roberta-base, and https://github.com/mlfoundations/open_clip/blob/main/docs/openclip_multilingual_retrieval_results.csv) by specifying `--language`.
For ImageNet-1k (zero-shot classification):
- `clip_benchmark eval --model xlm-roberta-base-ViT-B-32 --pretrained laion5b_s13b_b90k --dataset=imagenet1k --output=result.json --batch_size=64 --language=<LANG>`, where `<LANG>` can be among `zh` (chinese), `it` (italian), `jp` (japanese), `en` (english), `ar` (arabic).
- We also support Babel ImageNet classnames and prompts (https://github.com/gregor-ge/Babel-ImageNet), which can be used as the following: `clip_benchmark eval --model xlm-roberta-base-ViT-B-32 --pretrained laion5b_s13b_b90k --dataset=babel_imagenet --output=result.json --batch_size=64 --language=<LANG>`,
where `<LANG>` is a two letter string from the [ISO language code list](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes). Supported values for language are: `'ne', 'id', 'de', 'nl', 'af', 'he', 'sq', 'uz', 'kn', 'ku', 'ta', 'lv', 'ko', 'ug', 'br', 'el', 'su', 'kk', 'sk', 'gl', 'om', 'fa', 'jv', 'cs', 'lo', 'hy', 'xh', 'hr', 'so', 'gu', 'am', 'ar', 'sa', 'ca', 'is', 'it', 'sv', 'ga', 'bg', 'vi', 'sd', 'ur', 'km', 'pl', 'hu', 'sr', 'fr', 'hi', 'fy', 'et', 'bs', 'sw', 'az', 'mk', 'es', 'mn', 'ja', 'tl', 'tr', 'gd', 'ro', 'mg', 'mr', 'sl', 'pt', 'lt', 'no', 'yi', 'uk', 'ky', 'ka', 'bn', 'or', 'my', 'en', 'ps', 'fi', 'zh', 'da', 'ml', 'be', 'eo', 'ha', 'eu', 'as', 'te', 'th', 'cy', 'si', 'ru', 'la', 'pa', 'ms'`
for COCO (zero-shot retrieval):
- `clip_benchmark eval --model xlm-roberta-base-ViT-B-32 --pretrained laion5b_s13b_b90k --dataset=multilingual_mscoco_captions --output=result.json --batch_size=64 --language=<LANG>`, where `<LANG>` can be among `es` (spanish), `it` (italian), `jp` (japanese), `ko` (korean), `pl` (polish), `ru` (russian), `tr` (Turkish), `zh` (chinese), `en` (english), `fr` (french), `de` (german).
For Flickr-30k (zero-shot retrieval)
- `clip_benchmark eval --model xlm-roberta-base-ViT-B-32 --pretrained laion5b_s13b_b90k --dataset=flickr30k --output=result.json --batch_size=64 --language=<LANG>`, where `<LANG>` can be among `en` (english), `zh` (chinese).
For Flickr-8k (zero-shot retrieval)
- `clip_benchmark eval --model xlm-roberta-base-ViT-B-32 --pretrained laion5b_s13b_b90k --dataset=flickr8k --output=result.json --batch_size=64 --language=<LANG>`, where `<LANG>` can be among `en` (english), `zh` (chinese).
For [Crossmodal-3600](https://google.github.io/crossmodal-3600/) (zero-shot retrieval)
- `clip_benchmark eval --model xlm-roberta-base-ViT-B-32 --pretrained laion5b_s13b_b90k --dataset=crossmodal3600 --output=result.json --batch_size=64 --language=<LANG>`, see supported languages [here](https://github.com/LAION-AI/CLIP_benchmark/blob/main/clip_benchmark/datasets/crossmodal3600.py#L9).
For Flickr30k-200 dataset, which has 1000 captions from Flickr30k dataset translated to 200 languages using [NLLB-3.3B model](https://huggingface.co/facebook/nllb-200-3.3B) (zero-shot retrieval)
- `clip_benchmark eval --model xlm-roberta-base-ViT-B-32 --pretrained laion5b_s13b_b90k --dataset=flickr30k-200 --output=result.json --batch_size=64 --language=<LANG>`, see supported languages [here](https://github.com/LAION-AI/CLIP_benchmark/blob/main/clip_benchmark/datasets/flickr30k_200.py#L15).
For XTD200 dataset, which has captions from [XTD10](https://github.com/adobe-research/Cross-lingual-Test-Dataset-XTD10) dataset translated to 200 languages using [NLLB-3.3B model](https://huggingface.co/facebook/nllb-200-3.3B) (zero-shot retrieval)
- `clip_benchmark eval --model xlm-roberta-base-ViT-B-32 --pretrained laion5b_s13b_b90k --dataset=xtd200 --output=result.json --batch_size=64 --language=<LANG>`, see supported languages [here](https://github.com/LAION-AI/CLIP_benchmark/blob/main/clip_benchmark/datasets/xtd200.py#L15).
### Compositionality evaluation
For [Sugar Crepe](https://github.com/RAIVNLab/sugar-crepe):
`clip_benchmark eval --model ViT-B-32 --pretrained laion400m_e32 --dataset=sugar_crepe/<TASK> --output=result.json`
where `<TASK>` can be among `add_att`, `add_obj`, `replace_att`, `replace_obj`, `replace_rel`, `swap_att`, `swap_obj`.
To evaluate on all the tasks together, you can do:
`clip_benchmark eval --model ViT-B-32 --pretrained laion400m_e32 --dataset=sugar_crepe --output=result.json`
For [winoground](https://huggingface.co/datasets/facebook/winoground/):
`clip_benchmark eval --model ViT-B-32 --pretrained laion400m_e32 --dataset=winoground --output=result.json`
NB: `pip install datasets` is required for winoground.
### Webdataset example
Here is an example on how to run it on [webdatasets](https://github.com/webdataset/webdataset).
First, you need to install `webdataset`.
`pip install webdataset`
#### Creating a webdataset
You can either convert an already supported CLIP_benchmark dataset to webdataset format, or manually create your own with the same file structure. For already supported datasets use the CLI command `clip_benchmark_export_wds` as in this example:
```
$ clip_benchmark_export_wds --dataset cifar10 --plit train --dataset_root DATA_DIR/ --output wds_cifar10/
$ clip_benchmark_export_wds --dataset cifar10 --split test --dataset_root DATA_DIR/ --output wds_cifar10/
```
which will convert the train and test splits for CIFAR-10 (downloaded to `DATA_DIR/`) and save the webdataset to `wds_cifar10/` (upload to Huggingface Hub must be done manually for now). Retrieval datasets are also supported with the `--retrieval` flag.
For other datasets, data must be stored with the following file structure:
```
root_dir/
train/
nshards.txt
0.tar
1.tar
...
test/
nshards.txt
0.tar
...
classnames.txt
zeroshot_classification_templates.txt
dataset_type.txt
```
Each split should be contained in its own folder and `nshards.txt` should contain a single integer corresponding to the number of TAR files. The TAR files should follow webdataset format, with an image file (.webp, .png, or .jpg) and a label (.cls) for each example. Classnames and templates are required for zeroshot classification evaluation, with each classname or template on its own line. Dataset type is required for distinguishing zeroshot retrieval evaluation: the file should just contain the text `retrieval`.
#### Evaluating on a webdataset
The name of the dataset follows the template `wds/<DATASET_NAME>`. Note that the dataset name currently only affects the name in the results output - classnames and templates are loaded directly from the included files. The dataset root directory can be either a local path to the `root_dir` as specified above, or an HTTP URL pointing to a Huggingface Hub dataset file tree.
Example with `vtab/cifar10` (zero-shot classification):
- local: `clip_benchmark eval --dataset wds/vtab/cifar10 --dataset_root ROOT_DIR/wds_vtab-cifar10/`
- remote: `clip_benchmark eval --dataset wds/vtab/cifar10 --dataset_root https://huggingface.co/datasets/clip-benchmark/wds_{dataset_cleaned}/tree/main`
Example with `mscoco_captions` (retrieval):
- local: `clip_benchmark eval --dataset=wds/mscoco_captions --dataset_root ROOT_DIR/wds_vtab-mscoco_captions/ --task=zeroshot_retrieval`
- remote: `clip_benchmark eval --dataset=wds/mscoco_captions --dataset_root="https://huggingface.co/datasets/clip-benchmark/wds_{dataset_cleaned}/tree/main" --task=zeroshot_retrieval`
Example with `mscoco_captions` (captioning):
- local: `clip_benchmark eval --dataset=wds/mscoco_captions --dataset_root ROOT_DIR/wds_vtab-mscoco_captions/ --task=captioning`
- remote: `clip_benchmark eval --dataset=wds/mscoco_captions --dataset_root="https://huggingface.co/datasets/clip-benchmark/wds_{dataset_cleaned}/tree/main" --task=captioning`
All other arguments remain the same as in the other examples. See `https://huggingface.co/clip-benchmark` for a full list of datasets that have already been uploaded to Huggingface.
## Evaluate mulitple models on multiple datasets
For the purpose of benchmarking, it is possible to run the CLI with multiple
pre-trained models on multiple datasets.
### Pretrained models and datasets list as arguments
For models, we can provide list of pretrained model names in the form of 'model,pretrained' (so `model` and `pretrained` are comma separated). For datasets, we can provide a list of datasets. For languages, we can provide a list of languages.
Example:
```bash
clip_benchmark eval --pretrained_model ViT-B-32-quickgelu,laion400m_e32 ViT-L-14,laion400m_e32 \
--dataset cifar10 cifar100 --dataset_root "clip_benchmark_datasets/{dataset}" --language en jp \
--output "{dataset}_{pretrained}_{model}_{language}_{task}.json"
```
Note that `--dataset_root` and `--output` can be now in the form of a template that depends on the dataset/model/language/task (for `--output`) and dataset name (for `--dataset_root`).
Note that If the benchmark fails at some point, it is possible to resume it by skipping already evaluated models using `--skip_existing`.
### Pretrained models and datasets list as files
We can also provide a path to files with models (each line is in the form of 'model,pretrained' where `model` and `pretrained` are comma separated) and datasets list (one dataset per line):
```bash
clip_benchmark eval --pretrained_model benchmark/models.txt \
--dataset benchmark/datasets.txt --dataset_root "clip_benchmark_datasets/{dataset}" \
--output "{dataset}_{pretrained}_{model}_{language}_{task}.json"
```
Examples are available in [benchmark/datasets.txt](benchmark/datasets.txt) and [benchmark/models.txt](benchmark/models.txt)
### Multiple checkpoints from the same model
It is also common to evaluate multiple checkpoints from the same model:
```bash
clip_benchmark eval --model ViT-B-32 --pretrained *.pt \
--dataset benchmark/datasets.txt --dataset_root "clip_benchmark_datasets/{dataset}" \
--output "{dataset}_{pretrained}_{model}_{language}_{task}.json"
```
### Model and dataset collections
We can also provide model collection names (`openai`, `openclip_base`, `openclip_multilingual`, `openclip_full` are supported) or dataset collection names (`vtab`, `vtab+`, `retrieval`, `imagenet_robustness` are supported):
```bash
clip_benchmark eval --pretrained_model openai openclip_base --dataset vtab+ retrieval \
--dataset_root "clip_benchmark_datasets/{dataset}" --not quiet \
--output "{dataset}_{pretrained}_{model}_{language}_{task}.json"
```
See [clip_benchmark/models.py#L6](clip_benchmark/models.py#L6) and [clip_benchmark/datasets/builder.py#L634](clip_benchmark/datasets/builder.py#L634) for more information
about the collections.
### Custom templates / prompts / classnames
It is also possible to use custom prompts by providing a custom template file and/or a custom classname file.
For instance:
`clip_benchmark eval --dataset "imagenet1k" --model ViT-B-32 --pretrained laion400m_e32 --custom_template_file <path> --custom_classname_file <path>`
The template file can be either in the usual format https://github.com/LAION-AI/CLIP_benchmark/blob/main/clip_benchmark/datasets/en_zeroshot_classification_templates.json or in the CuPL format https://github.com/LAION-AI/CLIP_benchmark/blob/main/clip_benchmark/datasets/cupl_prompts.json to have class-specific prompts. In the case of the CuPL format, the classnames file will not be used, thus one only needs to provide the template file `--custom_template_file`.
For instance, the prompts from the CuPL paper https://arxiv.org/abs/2209.03320 for ImagetNet-1k can be used this way :
`clip_benchmark eval --dataset "imagenet1k" --model ViT-B-32 --pretrained laion400m_e32 --custom_template_file cupl_prompts.json`
### Development
For development, you can also do this:
```bash
git clone https://github.com/LAION-AI/CLIP_benchmark
cd CLIP_benchmark
python setup.py install
```
## Credits
- Thanks to [OpenCLIP](https://github.com/mlfoundations/open_clip) authors, zero-shot accuracy code is adapted from there and pre-trained models are used in the command line interface.
- Thanks to [SLIP](https://github.com/facebookresearch/SLIP) authors, some zero-shot templates and classnames are from there.
- Thanks to [Wise-ft](https://github.com/mlfoundations/wise-ft) authors, Imagenet robustness datasets code is adapted from there
- Thanks to [LiT](https://arxiv.org/abs/2111.07991.pdf) authors, some zero-shot templates and classnames of VTAB datasets are from there.
- Thanks to [Sugar Crepe](https://github.com/RAIVNLab/sugar-crepe) authors for compositionality tasks evaluation on COCO
- Thanks to [Babel ImageNet](https://github.com/gregor-ge/Babel-ImageNet) authors for multilingual evaluation of ImageNet-1k zero-shot classification.
- Thanks to [ImageNet-W](https://github.com/facebookresearch/Whac-A-Mole) authors for ImageNet-W evaluation
- Thanks to [CuPL](https://github.com/sarahpratt/CuPL) for CuPL prompts.
- Thanks to [PyCOCOevalcap](https://github.com/salaniz/pycocoevalcap) and [@gpucce](https://github.com/gpucce) for COCO captions image captioning evaluation.
- Thanks to [@li-xirong](https://github.com/li-xirong/cross-lingual-cap) for chinese Flickr-30k/FLickr-8k.
- Thanks to [Chinese CLIP](https://github.com/OFA-Sys/Chinese-CLIP) authors for chinese ImageNet-1k classnames/prompts (zero-shot classification).
- Thanks to [@rinnakk](https://github.com/rinnakk/japanese-clip) and [@mkshing](https://github.com/mkshing) for japanese ImageNet-1k classnames/prompts (zero-shot classification) and japanese CLIP support.
- Thanks to [@KhalidAlt](https://github.com/KhalidAlt) for arabic ImageNet-1k classnames/prompts (zero-shot classification).
- Thanks to [@djghosh13](https://github.com/djghosh13) for WebDataset support.
- Thanks to [@FreddeFrallan](https://github.com/FreddeFrallan) for for multilingual COCO.
- Thanks to [@mitchellnw](https://github.com/mitchellnw) for linear probing support.
- Thanks to [@teasgen](https://github.com/teasgen) for support of validation set and tuning linear probing similar to OpenAI's CLIP.
- Thanks to [@visheratin](https://github.com/visheratin) for multilingual retrieval datasets support from <https://arxiv.org/abs/2309.01859>.
- This package was created with [Cookiecutter](https://github.com/audreyr/cookiecutter) and the [audreyr/cookiecutter-pypackage](https://github.com/audreyr/cookiecutter-pypackage) project template. Thanks to the author.
## History
### 1.6.1
* Fix missing sugar crepe example #119 thanks to @samarth4149
### 1.6.0
* Fix overwritten zeroshot templates issue (https://github.com/LAION-AI/CLIP_benchmark/issues/109)
* Support new multilingual retrieval datasets: Crossmodal-3600, XTD10, Flickr30k-200, and XTD200
* Support tuning linear probing on validation set
### 1.5.0
* Custom classnames and templates
* support wds for captioning evaluation
* support imagenet-w
* support babel imagenet
* support chinese flickr30k/8k
* support sugar crepe (compositionality)
* support (optional) sharding evaluation based on rank, for parallel runs
* fix many issues
### 1.4.0
* Fix silent webdataset error-handling
* Added support for wds/voc2007_multilabel
* default to float32
* add mscoco generative benchmark
### 1.3.0
* update flickr8k results, solve issue #48, thanks to @orchidmajumder
* Evaluate multiple models/datasets/languages using the CLI directly
* Support Japanese CLIP by rinna
* Add arabic imagenet
* updating CuPL prompts with more generated sentences + ensembled with openAI prompts
* put model in eval mode before evaluation
* Webdataset updates
* Make verbose the default
### 1.2.0
* Added support for loading webdatasets
### 1.1.0
* Added better support for multilingual eval
* Added better support for linear probing
* Added support for CuPL prompts
### 1.0.1
* pypi description as markdown
### 1.0.0
* Actual first release on PyPI.
### 0.1.0
* First release on PyPI.
Raw data
{
"_id": null,
"home_page": "https://github.com/mehdidc/clip_benchmark",
"name": "clip-benchmark",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": "",
"keywords": "clip_benchmark",
"author": "Mehdi Cherti",
"author_email": "mehdicherti@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/c4/af/733be824fa6e9b366036388f12893870af84aef8fb06064a5ebc067ac86c/clip_benchmark-1.6.1.tar.gz",
"platform": null,
"description": "# CLIP Benchmark\n[](https://pypi.python.org/pypi/clip_benchmark)\n\nThe goal of this repo is to evaluate CLIP-like models on a standard set\nof datasets on different tasks such as zero-shot classification and zero-shot\nretrieval, and captioning.\n\nBelow we show the average rank (1 is the best, lower is better) of different CLIP models, evaluated\non different datasets.\n\n\n\nThe current detailed results of the benchmark can be seen [here](benchmark/README.md)\nor directly in the [notebook](benchmark/results.ipynb).\n\n## Features\n\n* Support for zero-shot classification and zero-shot retrieval, linear probing, and captioning.\n* Support for [OpenCLIP](https://github.com/mlfoundations/open_clip) pre-trained models, [Japanese CLIP](https://github.com/rinnakk/japanese-clip), and [NLLB CLIP](https://arxiv.org/abs/2309.01859) for general multilingual abilities.\n* Support various datasets from [torchvision](https://pytorch.org/vision/stable/datasets.html), [tensorflow datasets](https://www.tensorflow.org/datasets), and [VTAB](https://github.com/google-research/task_adaptation).\n* Support for various multilingual datasets for classification and retrieval\n* Support for compositionality tasks\n\n## How to install?\n\n`pip install clip-benchmark`\n\n## How to use?\n\nTo evaluate we recommend to create a models.txt like\n```\nViT-B-32,openai\n```\n\nto get the list of datasets \n```\nwget https://raw.githubusercontent.com/LAION-AI/CLIP_benchmark/main/benchmark/webdatasets.txt\n```\n\nThen to run\n\n```\nclip_benchmark eval --pretrained_model models.txt \\\n --dataset \"webdatasets.txt\" \\\n --dataset_root \"https://huggingface.co/datasets/clip-benchmark/wds_{dataset_cleaned}/tree/main\" \\\n --output \"benchmark_{dataset}_{pretrained}_{model}_{language}_{task}.json\"\n```\n\nThen to get the full table\n\n```\nclip_benchmark build benchmark_*.json --output benchmark.csv\n```\n\n\n### Command line interface (CLI)\n\nThe easiest way to benchmark the models is using the CLI, `clip_benchmark`.\nYou can specify the model to use, the dataset and the task to evaluate on. Once it is done, evaluation is performed and\nthe results are written into a JSON file.\n\n### Using other models than openclip\n\nIt is possible to use other models than openclip ones. For example japanese-clip is supported\n\nHere is an example of use\n\n```\n>>> python3 clip_benchmark/cli.py eval \\\n --model_type \"ja_clip\" \\ # flag to use japanese-clip\n --pretrained \"rinna/japanese-cloob-vit-b-16\" \\ # now, we have `rinna/japanese-cloob-vit-b-16` or `rinna/japanese-clip-vit-b-16`. \n --language \"jp\" \\\n --task \"zeroshot_classification\" \\\n --dataset \"imagenet1k\" \\\n --dataset_root {ROOT_PATH} \n\n>>> cat result.json\n{\"dataset\": \"imagenet1k\", \"model\": \"ViT-B-32-quickgelu\", \"pretrained\": \"rinna/japanese-cloob-vit-b-16\", \"task\": \"zeroshot_classification\", \"metrics\": {\"acc1\": 0.54636, \"acc5\": 0.72856, \"mean_per_class_recall\": 0.54522}, \"language\": \"jp\"}\n```\n\n### How to add other CLIP models\n\nPlease follow these steps:\n1. Add a identity file to load model in `clip_benchmark/models`\n2. Define a loading function, that returns a tuple (model, transform, tokenizer). Please see `clip_benchmark/models/open_clip.py` as an example. \n3. Add the function into `TYPE2FUNC` in `clip_benchmark/models/__init__.py`\n\nRemarks:\n- The new tokenizer/model must enable to do the following things as https://github.com/openai/CLIP#usage\n - `tokenizer(texts).to(device)` ... `texts` is a list of string\n - `model.encode_text(tokenized_texts)` ... `tokenized_texts` is a output from `tokenizer(texts).to(device)`\n - `model.encode_image(images)` ... `images` is a image tensor by the `transform`\n\n\n### CIFAR-10 example\n\n Here is an example for CIFAR-10 zero-shot classification using OpenCLIP's pre-trained model on LAION-400m:\n\n `clip_benchmark eval --dataset=cifar10 --task=zeroshot_classification --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64`\n\n By default, the dataset is downloaded into `--dataset_root`, which by default is `root`.\n\nHere is the content of `result.json` after the evaluation is done:\n\n```json\n{\n \"dataset\": \"cifar10\", \"model\": \"ViT-B-32-quickgelu\", \n \"pretrained\": \"laion400m_e32\", \"task\": \"zeroshot_classification\",\n \"metrics\": {\"acc1\": 0.9074, \"acc5\": 0.998}\n}\n```\n\n\n### VOC2007 example\n\nHere is another example with VOC2007, which is a multi-label classification dataset.\n\n `clip_benchmark eval --dataset=voc2007_multilabel --task=zeroshot_classification --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64`\n\nHere is the content of `result.json` after the evaluation is done:\n\n```json\n{\"dataset\": \"voc2007_multilabel\", \"model\": \"ViT-B-32-quickgelu\", \"pretrained\": \"laion400m_e32\", \"task\": \"zeroshot_classification\", \"metrics\": {\"mean_average_precision\": 0.7627869844436646}}\n```\n\nHere, we compute the mean average precision or mAP, more details about that metric [here](https://fangdahan.medium.com/calculate-mean-average-precision-map-for-multi-label-classification-b082679d31be) in the context of multi-label classification.\n\n### VTAB example\n\nHere is an example on how to run it on [VTAB](https://github.com/google-research/task_adaptation) classification tasks.\nFirst, you need to install VTAB's dedicated package.\n\n`pip install task_adaptation==0.1`\n\nThen, you can run it by providing the full dataset name.\nExample with `eurosat`:\n\n `clip_benchmark eval --dataset=vtab/eurosat --task=zeroshot_classification --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64`\n\nSee [clip_benchmark/datasets/builder.py#L634](clip_benchmark/datasets/builder.py#L634) for the full list of \nVTAB dataset collection.\n\n\n### TensorFlow dataset example\n\nHere is an example on how to run it on [Tensorflow datasets](https://www.tensorflow.org/datasets).\nFirst, you need to install `tfds-nightly` and `timm`.\n\n`pip install timm tfds-nightly`\n\n\nThe name of the dataset follows the template `tfds/<DATASET_NAME>`.\n\nExample with `cifar10`:\n\n `clip_benchmark eval --dataset=tfds/cifar10 --task=zeroshot_classification --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64`\n\n\n### COCO captions retrieval example\n\n Here is an example for COCO captions zero-shot retrieval:\n\n `clip_benchmark eval --dataset=mscoco_captions --task=zeroshot_retrieval --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64` \n\n Note that for using COCO, you also need to install `pycocotools` (e.g., using `pip install pycocotools`).\n\n\n### COCO captions captioning example\n\n Here is an example for COCO captions captioning task:\n\n `clip_benchmark eval --dataset=mscoco_captions --task=captioning --model=coca_ViT-L-14 --output=result.json --pretrained mscoco_finetuned_laion2b_s13b_b90k`\n\n Note that for using COCO, you also need to install `pycocotools` (e.g., using `pip install pycocotools`) and `pycocoevalcap`.\n\n### Linear probing example\n\nFull linear probing on train split, evaluate on test split:\n\n`clip_benchmark eval --dataset=cifar10 --task=linear_probe --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64 --fewshot_lr 0.1 --fewshot_epochs 20 --batch_size 512 --train_split train --test_split test`\n\n\nfew-shot (k=5) linear probing on train split, evaluate on test split:\n\n`clip_benchmark eval --dataset=cifar10 --task=linear_probe --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64 --fewshot_k 5 --fewshot_lr 0.1 --fewshot_epochs 20 --batch_size 512 --train_split train --test_split test`\n\nSplit train into train (90%), val (10%), do linear probing on train split, tune on val split, evaluate on test split:\n\n`clip_benchmark eval --dataset=cifar10 --task=linear_probe --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64 --fewshot_lr 0.1 --fewshot_epochs 20 --batch_size 512 --train_split train --val_proportion 0.1 --test_split test`\n\nFor other datasets that have an official val split, one can also specify the val split:\n\n`clip_benchmark eval --dataset=fgvc_aircraft --task=linear_probe --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64 --fewshot_lr 0.1 --fewshot_epochs 20 --batch_size 512 --train_split train --val_split val --test_split test`\n\n### Multilingual evaluation\n\nWe also provide datasets for evaluating multilingual models (see e.g. https://github.com/mlfoundations/open_clip#vit-b32-xlm-roberta-base, and https://github.com/mlfoundations/open_clip/blob/main/docs/openclip_multilingual_retrieval_results.csv) by specifying `--language`.\n\nFor ImageNet-1k (zero-shot classification):\n\n- `clip_benchmark eval --model xlm-roberta-base-ViT-B-32 --pretrained laion5b_s13b_b90k --dataset=imagenet1k --output=result.json --batch_size=64 --language=<LANG>`, where `<LANG>` can be among `zh` (chinese), `it` (italian), `jp` (japanese), `en` (english), `ar` (arabic).\n- We also support Babel ImageNet classnames and prompts (https://github.com/gregor-ge/Babel-ImageNet), which can be used as the following: `clip_benchmark eval --model xlm-roberta-base-ViT-B-32 --pretrained laion5b_s13b_b90k --dataset=babel_imagenet --output=result.json --batch_size=64 --language=<LANG>`, \nwhere `<LANG>` is a two letter string from the [ISO language code list](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes). Supported values for language are: `'ne', 'id', 'de', 'nl', 'af', 'he', 'sq', 'uz', 'kn', 'ku', 'ta', 'lv', 'ko', 'ug', 'br', 'el', 'su', 'kk', 'sk', 'gl', 'om', 'fa', 'jv', 'cs', 'lo', 'hy', 'xh', 'hr', 'so', 'gu', 'am', 'ar', 'sa', 'ca', 'is', 'it', 'sv', 'ga', 'bg', 'vi', 'sd', 'ur', 'km', 'pl', 'hu', 'sr', 'fr', 'hi', 'fy', 'et', 'bs', 'sw', 'az', 'mk', 'es', 'mn', 'ja', 'tl', 'tr', 'gd', 'ro', 'mg', 'mr', 'sl', 'pt', 'lt', 'no', 'yi', 'uk', 'ky', 'ka', 'bn', 'or', 'my', 'en', 'ps', 'fi', 'zh', 'da', 'ml', 'be', 'eo', 'ha', 'eu', 'as', 'te', 'th', 'cy', 'si', 'ru', 'la', 'pa', 'ms'` \n\nfor COCO (zero-shot retrieval):\n\n- `clip_benchmark eval --model xlm-roberta-base-ViT-B-32 --pretrained laion5b_s13b_b90k --dataset=multilingual_mscoco_captions --output=result.json --batch_size=64 --language=<LANG>`, where `<LANG>` can be among `es` (spanish), `it` (italian), `jp` (japanese), `ko` (korean), `pl` (polish), `ru` (russian), `tr` (Turkish), `zh` (chinese), `en` (english), `fr` (french), `de` (german).\n\nFor Flickr-30k (zero-shot retrieval)\n\n- `clip_benchmark eval --model xlm-roberta-base-ViT-B-32 --pretrained laion5b_s13b_b90k --dataset=flickr30k --output=result.json --batch_size=64 --language=<LANG>`, where `<LANG>` can be among `en` (english), `zh` (chinese).\n\nFor Flickr-8k (zero-shot retrieval)\n\n- `clip_benchmark eval --model xlm-roberta-base-ViT-B-32 --pretrained laion5b_s13b_b90k --dataset=flickr8k --output=result.json --batch_size=64 --language=<LANG>`, where `<LANG>` can be among `en` (english), `zh` (chinese).\n\nFor [Crossmodal-3600](https://google.github.io/crossmodal-3600/) (zero-shot retrieval)\n\n- `clip_benchmark eval --model xlm-roberta-base-ViT-B-32 --pretrained laion5b_s13b_b90k --dataset=crossmodal3600 --output=result.json --batch_size=64 --language=<LANG>`, see supported languages [here](https://github.com/LAION-AI/CLIP_benchmark/blob/main/clip_benchmark/datasets/crossmodal3600.py#L9).\n\nFor Flickr30k-200 dataset, which has 1000 captions from Flickr30k dataset translated to 200 languages using [NLLB-3.3B model](https://huggingface.co/facebook/nllb-200-3.3B) (zero-shot retrieval)\n\n- `clip_benchmark eval --model xlm-roberta-base-ViT-B-32 --pretrained laion5b_s13b_b90k --dataset=flickr30k-200 --output=result.json --batch_size=64 --language=<LANG>`, see supported languages [here](https://github.com/LAION-AI/CLIP_benchmark/blob/main/clip_benchmark/datasets/flickr30k_200.py#L15).\n\nFor XTD200 dataset, which has captions from [XTD10](https://github.com/adobe-research/Cross-lingual-Test-Dataset-XTD10) dataset translated to 200 languages using [NLLB-3.3B model](https://huggingface.co/facebook/nllb-200-3.3B) (zero-shot retrieval)\n\n- `clip_benchmark eval --model xlm-roberta-base-ViT-B-32 --pretrained laion5b_s13b_b90k --dataset=xtd200 --output=result.json --batch_size=64 --language=<LANG>`, see supported languages [here](https://github.com/LAION-AI/CLIP_benchmark/blob/main/clip_benchmark/datasets/xtd200.py#L15).\n\n\n### Compositionality evaluation\n\n\nFor [Sugar Crepe](https://github.com/RAIVNLab/sugar-crepe):\n\n\n`clip_benchmark eval --model ViT-B-32 --pretrained laion400m_e32 --dataset=sugar_crepe/<TASK> --output=result.json`\n\nwhere `<TASK>` can be among `add_att`, `add_obj`, `replace_att`, `replace_obj`, `replace_rel`, `swap_att`, `swap_obj`.\nTo evaluate on all the tasks together, you can do:\n\n`clip_benchmark eval --model ViT-B-32 --pretrained laion400m_e32 --dataset=sugar_crepe --output=result.json`\n\nFor [winoground](https://huggingface.co/datasets/facebook/winoground/):\n\n`clip_benchmark eval --model ViT-B-32 --pretrained laion400m_e32 --dataset=winoground --output=result.json`\n\nNB: `pip install datasets` is required for winoground.\n\n### Webdataset example\n\nHere is an example on how to run it on [webdatasets](https://github.com/webdataset/webdataset).\nFirst, you need to install `webdataset`.\n\n`pip install webdataset`\n\n#### Creating a webdataset\n\nYou can either convert an already supported CLIP_benchmark dataset to webdataset format, or manually create your own with the same file structure. For already supported datasets use the CLI command `clip_benchmark_export_wds` as in this example:\n\n```\n$ clip_benchmark_export_wds --dataset cifar10 --plit train --dataset_root DATA_DIR/ --output wds_cifar10/\n$ clip_benchmark_export_wds --dataset cifar10 --split test --dataset_root DATA_DIR/ --output wds_cifar10/\n```\n\nwhich will convert the train and test splits for CIFAR-10 (downloaded to `DATA_DIR/`) and save the webdataset to `wds_cifar10/` (upload to Huggingface Hub must be done manually for now). Retrieval datasets are also supported with the `--retrieval` flag.\n\nFor other datasets, data must be stored with the following file structure:\n\n```\nroot_dir/\n train/\n nshards.txt\n 0.tar\n 1.tar\n ...\n test/\n nshards.txt\n 0.tar\n ...\n classnames.txt\n zeroshot_classification_templates.txt\n dataset_type.txt\n```\n\nEach split should be contained in its own folder and `nshards.txt` should contain a single integer corresponding to the number of TAR files. The TAR files should follow webdataset format, with an image file (.webp, .png, or .jpg) and a label (.cls) for each example. Classnames and templates are required for zeroshot classification evaluation, with each classname or template on its own line. Dataset type is required for distinguishing zeroshot retrieval evaluation: the file should just contain the text `retrieval`.\n\n#### Evaluating on a webdataset\n\nThe name of the dataset follows the template `wds/<DATASET_NAME>`. Note that the dataset name currently only affects the name in the results output - classnames and templates are loaded directly from the included files. The dataset root directory can be either a local path to the `root_dir` as specified above, or an HTTP URL pointing to a Huggingface Hub dataset file tree.\n\nExample with `vtab/cifar10` (zero-shot classification):\n\n- local: `clip_benchmark eval --dataset wds/vtab/cifar10 --dataset_root ROOT_DIR/wds_vtab-cifar10/`\n- remote: `clip_benchmark eval --dataset wds/vtab/cifar10 --dataset_root https://huggingface.co/datasets/clip-benchmark/wds_{dataset_cleaned}/tree/main`\n\nExample with `mscoco_captions` (retrieval):\n\n- local: `clip_benchmark eval --dataset=wds/mscoco_captions --dataset_root ROOT_DIR/wds_vtab-mscoco_captions/ --task=zeroshot_retrieval`\n- remote: `clip_benchmark eval --dataset=wds/mscoco_captions --dataset_root=\"https://huggingface.co/datasets/clip-benchmark/wds_{dataset_cleaned}/tree/main\" --task=zeroshot_retrieval`\n\n\nExample with `mscoco_captions` (captioning):\n\n- local: `clip_benchmark eval --dataset=wds/mscoco_captions --dataset_root ROOT_DIR/wds_vtab-mscoco_captions/ --task=captioning`\n- remote: `clip_benchmark eval --dataset=wds/mscoco_captions --dataset_root=\"https://huggingface.co/datasets/clip-benchmark/wds_{dataset_cleaned}/tree/main\" --task=captioning`\n\n\nAll other arguments remain the same as in the other examples. See `https://huggingface.co/clip-benchmark` for a full list of datasets that have already been uploaded to Huggingface.\n\n## Evaluate mulitple models on multiple datasets\n\nFor the purpose of benchmarking, it is possible to run the CLI with multiple\npre-trained models on multiple datasets.\n\n\n### Pretrained models and datasets list as arguments\n\nFor models, we can provide list of pretrained model names in the form of 'model,pretrained' (so `model` and `pretrained` are comma separated). For datasets, we can provide a list of datasets. For languages, we can provide a list of languages.\nExample:\n\n```bash\nclip_benchmark eval --pretrained_model ViT-B-32-quickgelu,laion400m_e32 ViT-L-14,laion400m_e32 \\\n--dataset cifar10 cifar100 --dataset_root \"clip_benchmark_datasets/{dataset}\" --language en jp \\\n --output \"{dataset}_{pretrained}_{model}_{language}_{task}.json\"\n```\n\nNote that `--dataset_root` and `--output` can be now in the form of a template that depends on the dataset/model/language/task (for `--output`) and dataset name (for `--dataset_root`).\n\nNote that If the benchmark fails at some point, it is possible to resume it by skipping already evaluated models using `--skip_existing`.\n\n### Pretrained models and datasets list as files\n\nWe can also provide a path to files with models (each line is in the form of 'model,pretrained' where `model` and `pretrained` are comma separated) and datasets list (one dataset per line):\n\n```bash\nclip_benchmark eval --pretrained_model benchmark/models.txt \\\n--dataset benchmark/datasets.txt --dataset_root \"clip_benchmark_datasets/{dataset}\" \\\n --output \"{dataset}_{pretrained}_{model}_{language}_{task}.json\"\n```\n\nExamples are available in [benchmark/datasets.txt](benchmark/datasets.txt) and [benchmark/models.txt](benchmark/models.txt)\n\n### Multiple checkpoints from the same model\n\nIt is also common to evaluate multiple checkpoints from the same model:\n\n```bash\nclip_benchmark eval --model ViT-B-32 --pretrained *.pt \\\n--dataset benchmark/datasets.txt --dataset_root \"clip_benchmark_datasets/{dataset}\" \\\n --output \"{dataset}_{pretrained}_{model}_{language}_{task}.json\"\n```\n\n### Model and dataset collections\n\nWe can also provide model collection names (`openai`, `openclip_base`, `openclip_multilingual`, `openclip_full` are supported) or dataset collection names (`vtab`, `vtab+`, `retrieval`, `imagenet_robustness` are supported):\n\n```bash\nclip_benchmark eval --pretrained_model openai openclip_base --dataset vtab+ retrieval \\\n--dataset_root \"clip_benchmark_datasets/{dataset}\" --not quiet \\\n--output \"{dataset}_{pretrained}_{model}_{language}_{task}.json\"\n```\n\nSee [clip_benchmark/models.py#L6](clip_benchmark/models.py#L6) and [clip_benchmark/datasets/builder.py#L634](clip_benchmark/datasets/builder.py#L634) for more information\nabout the collections.\n\n### Custom templates / prompts / classnames\n\nIt is also possible to use custom prompts by providing a custom template file and/or a custom classname file.\nFor instance:\n\n`clip_benchmark eval --dataset \"imagenet1k\" --model ViT-B-32 --pretrained laion400m_e32 --custom_template_file <path> --custom_classname_file <path>`\n\nThe template file can be either in the usual format https://github.com/LAION-AI/CLIP_benchmark/blob/main/clip_benchmark/datasets/en_zeroshot_classification_templates.json or in the CuPL format https://github.com/LAION-AI/CLIP_benchmark/blob/main/clip_benchmark/datasets/cupl_prompts.json to have class-specific prompts. In the case of the CuPL format, the classnames file will not be used, thus one only needs to provide the template file `--custom_template_file`.\n\nFor instance, the prompts from the CuPL paper https://arxiv.org/abs/2209.03320 for ImagetNet-1k can be used this way :\n\n`clip_benchmark eval --dataset \"imagenet1k\" --model ViT-B-32 --pretrained laion400m_e32 --custom_template_file cupl_prompts.json`\n\n### Development \n\nFor development, you can also do this:\n\n```bash\ngit clone https://github.com/LAION-AI/CLIP_benchmark\ncd CLIP_benchmark\npython setup.py install\n```\n\n## Credits\n\n- Thanks to [OpenCLIP](https://github.com/mlfoundations/open_clip) authors, zero-shot accuracy code is adapted from there and pre-trained models are used in the command line interface.\n- Thanks to [SLIP](https://github.com/facebookresearch/SLIP) authors, some zero-shot templates and classnames are from there.\n- Thanks to [Wise-ft](https://github.com/mlfoundations/wise-ft) authors, Imagenet robustness datasets code is adapted from there\n- Thanks to [LiT](https://arxiv.org/abs/2111.07991.pdf) authors, some zero-shot templates and classnames of VTAB datasets are from there.\n- Thanks to [Sugar Crepe](https://github.com/RAIVNLab/sugar-crepe) authors for compositionality tasks evaluation on COCO\n- Thanks to [Babel ImageNet](https://github.com/gregor-ge/Babel-ImageNet) authors for multilingual evaluation of ImageNet-1k zero-shot classification.\n- Thanks to [ImageNet-W](https://github.com/facebookresearch/Whac-A-Mole) authors for ImageNet-W evaluation\n- Thanks to [CuPL](https://github.com/sarahpratt/CuPL) for CuPL prompts.\n- Thanks to [PyCOCOevalcap](https://github.com/salaniz/pycocoevalcap) and [@gpucce](https://github.com/gpucce) for COCO captions image captioning evaluation.\n- Thanks to [@li-xirong](https://github.com/li-xirong/cross-lingual-cap) for chinese Flickr-30k/FLickr-8k.\n- Thanks to [Chinese CLIP](https://github.com/OFA-Sys/Chinese-CLIP) authors for chinese ImageNet-1k classnames/prompts (zero-shot classification).\n- Thanks to [@rinnakk](https://github.com/rinnakk/japanese-clip) and [@mkshing](https://github.com/mkshing) for japanese ImageNet-1k classnames/prompts (zero-shot classification) and japanese CLIP support.\n- Thanks to [@KhalidAlt](https://github.com/KhalidAlt) for arabic ImageNet-1k classnames/prompts (zero-shot classification).\n- Thanks to [@djghosh13](https://github.com/djghosh13) for WebDataset support.\n- Thanks to [@FreddeFrallan](https://github.com/FreddeFrallan) for for multilingual COCO.\n- Thanks to [@mitchellnw](https://github.com/mitchellnw) for linear probing support.\n- Thanks to [@teasgen](https://github.com/teasgen) for support of validation set and tuning linear probing similar to OpenAI's CLIP.\n- Thanks to [@visheratin](https://github.com/visheratin) for multilingual retrieval datasets support from <https://arxiv.org/abs/2309.01859>.\n- This package was created with [Cookiecutter](https://github.com/audreyr/cookiecutter) and the [audreyr/cookiecutter-pypackage](https://github.com/audreyr/cookiecutter-pypackage) project template. Thanks to the author.\n\n\n## History\n\n### 1.6.1\n\n* Fix missing sugar crepe example #119 thanks to @samarth4149\n\n### 1.6.0\n\n\n* Fix overwritten zeroshot templates issue (https://github.com/LAION-AI/CLIP_benchmark/issues/109)\n* Support new multilingual retrieval datasets: Crossmodal-3600, XTD10, Flickr30k-200, and XTD200\n* Support tuning linear probing on validation set\n\n### 1.5.0\n\n* Custom classnames and templates\n* support wds for captioning evaluation\n* support imagenet-w\n* support babel imagenet\n* support chinese flickr30k/8k\n* support sugar crepe (compositionality)\n* support (optional) sharding evaluation based on rank, for parallel runs\n* fix many issues\n\n### 1.4.0\n\n* Fix silent webdataset error-handling\n* Added support for wds/voc2007_multilabel \n* default to float32 \n* add mscoco generative benchmark\n\n### 1.3.0\n\n* update flickr8k results, solve issue #48, thanks to @orchidmajumder\n* Evaluate multiple models/datasets/languages using the CLI directly\n* Support Japanese CLIP by rinna\n* Add arabic imagenet\n* updating CuPL prompts with more generated sentences + ensembled with openAI prompts\n* put model in eval mode before evaluation\n* Webdataset updates\n* Make verbose the default\n\n### 1.2.0\n\n* Added support for loading webdatasets\n\n### 1.1.0\n\n* Added better support for multilingual eval\n* Added better support for linear probing\n* Added support for CuPL prompts\n\n### 1.0.1\n\n* pypi description as markdown\n\n### 1.0.0\n\n* Actual first release on PyPI.\n\n\n### 0.1.0\n\n* First release on PyPI.\n\n\n",
"bugtrack_url": null,
"license": "MIT license",
"summary": "CLIP-like models benchmarks on various datasets",
"version": "1.6.1",
"project_urls": {
"Homepage": "https://github.com/mehdidc/clip_benchmark"
},
"split_keywords": [
"clip_benchmark"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4ac69bb7554f39f9b306b3ce704ad86bd9afe1555a584445a89565b52ba64ddd",
"md5": "6f1518947ffa30cede9e1ccce42ae98a",
"sha256": "d83f524d18aaad1cb72bc96d39a2c6f572f34c54e82791a16492d43f77036796"
},
"downloads": -1,
"filename": "clip_benchmark-1.6.1-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "6f1518947ffa30cede9e1ccce42ae98a",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": ">=3.6",
"size": 1944703,
"upload_time": "2024-02-19T14:29:05",
"upload_time_iso_8601": "2024-02-19T14:29:05.381191Z",
"url": "https://files.pythonhosted.org/packages/4a/c6/9bb7554f39f9b306b3ce704ad86bd9afe1555a584445a89565b52ba64ddd/clip_benchmark-1.6.1-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c4af733be824fa6e9b366036388f12893870af84aef8fb06064a5ebc067ac86c",
"md5": "3dc0a6b05d95be6db93d3082c4b0c13c",
"sha256": "eb222cec1759bf80458cc9f509c5d01557e63c5f5d8dce4018bf5c0ef3a427e8"
},
"downloads": -1,
"filename": "clip_benchmark-1.6.1.tar.gz",
"has_sig": false,
"md5_digest": "3dc0a6b05d95be6db93d3082c4b0c13c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 2056113,
"upload_time": "2024-02-19T14:29:07",
"upload_time_iso_8601": "2024-02-19T14:29:07.277513Z",
"url": "https://files.pythonhosted.org/packages/c4/af/733be824fa6e9b366036388f12893870af84aef8fb06064a5ebc067ac86c/clip_benchmark-1.6.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-19 14:29:07",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mehdidc",
"github_project": "clip_benchmark",
"github_not_found": true,
"lcname": "clip-benchmark"
}
JSON
