| Name | saliency-maps-metrics JSON |
| Version |
0.0.24
JSON |
| download |
| home_page | |
| Summary | This repository implements the faithfulness metrics mentionned in the paper --Computing and evaluating saliency maps for image classification: a tutorial-- in Pytorch. |
| upload_time | 2023-04-26 07:21:35 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.7 |
| license | |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Faithfulness metrics for saliency maps
TODO: explain the role of each script
This repository implements the faithfulness metrics mentionned in the paper "Computing and evaluating saliency maps for image classification: a tutorial" in Pytorch.
This can be used to compute the following metrics:
- Deletion Area Under Curve (DAUC)/Insertion Area Under Curve (IAUC) [(Petsiuk et al. 2019)](https://arxiv.org/abs/1806.07421)
- Deletion Correlation (DC)/Insertion Correlation (IC) [(Gomez et al. 2022)](https://link.springer.com/chapter/10.1007/978-3-031-09037-0_8)
- Increase In Confidence (IIC)/Average Drop (AD) [(Chattopadhyay et al. 2017)](https://arxiv.org/abs/1710.11063)
- Avereage Drop in Deletion (ADD) [(Jung et al.)](https://arxiv.org/pdf/2102.05228.pdf)
## Single step metrics
This section covers the use of the Increase In Confidence (IIC), Average Drop (AD) and Average Drop in Deletion (ADD) metrics.
First, generate the saliency map of the image however you want. The only constraint is that the map should be a tensor of size (Nx1xH'xW'):
```
saliency_map = gradcam.attribute(img,class_ind)
```
Then, compute the metric. In this example, we use the class IIC_AD of this library to compute both the AD and IIC metric, as they require similar computations:
```
iic_ad = IIC_AD()
metric_dict = iic_ad(model,data,explanations,class_to_explain)
mean_iic = metric_dict["iic"]
mean_ad = metric_dict["ad"]
```
The resulting dictionary has two entries ```ic``` and ```ad``` which correspond to the mean value of the two metrics.
The ```__call__()``` method for all the metrics requires the following arguments :
- ```model```: a ```torch.nn.Module``` that outputs a score tensor of shape (NxC), on which a softmax activation has been applied.
- ```data```: the input image tensor of shape (Nx3xHxW).
- ```explanations```: the saliency maps tensor of shape (Nx1xH'xW')
- ```class_to_explain```: The index of the class to explain for each input image. The shape shoud be (N).
The ADD class is used similarly:
```
add = ADD()
metric_dict = add(model,data,explanations,class_to_explain)
mean = metric_dict["add"]
```
This resulting dictionary has only one entry ```add``` which correspond to the mean value of the ADD metric.
## Multi-step metrics
This section covers the use of the DAUC, IAUC, DC and IC metrics.
These metrics work similarly but some argument have to be passed to the constructor:
```
deletion = Deletion(data_shape,explanation_shape,bound_max_step=True)
```
Where ```data_shape``` and ```explanation_shape``` are the shape of the image and saliency map tensor.
A high resolution saliency map of size 56x56 would require approximately 3k inferences.
To prevent too much computation, you can set the ```bound_max_step``` argument to True to limits the amount of computation that can be computed.
More precisly, this argument forces to mask/reveal several pixels before computing a new inference if the resolution of the saliency map is superior to 14x14.
The call method returns a dictionary with two entries: ```dauc``` (which correspond to the original Deletion metric by Petsiuk et al.) and ```dc``` (the variant proposed by Gomez et al.).
```
result_dic = deletion(model,data,explanations,class_to_explain)
dauc = result_dic["dauc"]
dc = result_dic["dc"]
```
The Insertion metric is computed similarly:
```
insertion = Insertion(data_shape,explanation_shape,bound_max_step=True)
result_dic = insertion(model,data,explanations,class_to_explain)
iauc = result_dic["iauc"]
ic = result_dic["ic"]
```
## Changing the filling method
These metrics work by removing parts of the image and replacing it with something else, for e.g. black pixels (Deletion, AD, IIC, ADD) or a blurred version of the input image (Insertion).
The default replacing method can be changed with the ```data_replace_method``` argument passed to the constructor:
```
insertion = Insertion(data_shape,explanation_shape,bound_max_step=True,data_replace_method="black")
add = ADD(data_replace_method="blur")
```
Currently, the autorized values are:
- "black": replaces with black pixels
- "blur": replaces with a blurred version of the input image
## Demonstration
Look at the ```demo.ipynb``` script for a demonstration.
If you want to re-run the demo, you should download the [model's weights](https://drive.google.com/file/d/1JdHJjvCb9IAtcwKizo_KGLDR73UmtFYY/view?usp=sharing) pretrained on the CUB dataset and put it on the project's root.
You should also download the [CUB test dataset](http://www.vision.caltech.edu/datasets/cub_200_2011/) and put it in a "data" folder located at the project's root.
The dataset should be formated as expected by the ```torchvision.datasets.ImageFolder``` class from ```torchvision```.
Raw data
{
"_id": null,
"home_page": "",
"name": "saliency-maps-metrics",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "",
"author": "",
"author_email": "Tristan Gomez <tristan.gomez@univ-nantes.fr>",
"download_url": "https://files.pythonhosted.org/packages/bc/54/4c4c216e911c2aa7e96695aecccb3767fd5efdb6e038dc67c57ebbb4b9e8/saliency_maps_metrics-0.0.24.tar.gz",
"platform": null,
"description": "# Faithfulness metrics for saliency maps \n\n\nTODO: explain the role of each script\n\n\nThis repository implements the faithfulness metrics mentionned in the paper \"Computing and evaluating saliency maps for image classification: a tutorial\" in Pytorch.\nThis can be used to compute the following metrics: \n- Deletion Area Under Curve (DAUC)/Insertion Area Under Curve (IAUC) [(Petsiuk et al. 2019)](https://arxiv.org/abs/1806.07421)\n- Deletion Correlation (DC)/Insertion Correlation (IC) [(Gomez et al. 2022)](https://link.springer.com/chapter/10.1007/978-3-031-09037-0_8)\n- Increase In Confidence (IIC)/Average Drop (AD) [(Chattopadhyay et al. 2017)](https://arxiv.org/abs/1710.11063)\n- Avereage Drop in Deletion (ADD) [(Jung et al.)](https://arxiv.org/pdf/2102.05228.pdf)\n\n\n\n## Single step metrics\n\nThis section covers the use of the Increase In Confidence (IIC), Average Drop (AD) and Average Drop in Deletion (ADD) metrics.\nFirst, generate the saliency map of the image however you want. The only constraint is that the map should be a tensor of size (Nx1xH'xW'):\n```\nsaliency_map = gradcam.attribute(img,class_ind)\n```\nThen, compute the metric. In this example, we use the class IIC_AD of this library to compute both the AD and IIC metric, as they require similar computations:\n\n```\niic_ad = IIC_AD()\nmetric_dict = iic_ad(model,data,explanations,class_to_explain)\nmean_iic = metric_dict[\"iic\"]\nmean_ad = metric_dict[\"ad\"]\n```\n\nThe resulting dictionary has two entries ```ic``` and ```ad``` which correspond to the mean value of the two metrics.\n\nThe ```__call__()``` method for all the metrics requires the following arguments :\n- ```model```: a ```torch.nn.Module``` that outputs a score tensor of shape (NxC), on which a softmax activation has been applied.\n- ```data```: the input image tensor of shape (Nx3xHxW).\n- ```explanations```: the saliency maps tensor of shape (Nx1xH'xW')\n- ```class_to_explain```: The index of the class to explain for each input image. The shape shoud be (N).\n\nThe ADD class is used similarly:\n```\nadd = ADD()\nmetric_dict = add(model,data,explanations,class_to_explain)\nmean = metric_dict[\"add\"]\n```\nThis resulting dictionary has only one entry ```add``` which correspond to the mean value of the ADD metric.\n\n## Multi-step metrics \n\nThis section covers the use of the DAUC, IAUC, DC and IC metrics.\nThese metrics work similarly but some argument have to be passed to the constructor:\n\n```\ndeletion = Deletion(data_shape,explanation_shape,bound_max_step=True)\n```\nWhere ```data_shape``` and ```explanation_shape``` are the shape of the image and saliency map tensor.\nA high resolution saliency map of size 56x56 would require approximately 3k inferences.\nTo prevent too much computation, you can set the ```bound_max_step``` argument to True to limits the amount of computation that can be computed.\nMore precisly, this argument forces to mask/reveal several pixels before computing a new inference if the resolution of the saliency map is superior to 14x14.\nThe call method returns a dictionary with two entries: ```dauc``` (which correspond to the original Deletion metric by Petsiuk et al.) and ```dc``` (the variant proposed by Gomez et al.).\n\n```\nresult_dic = deletion(model,data,explanations,class_to_explain)\ndauc = result_dic[\"dauc\"]\ndc = result_dic[\"dc\"]\n```\nThe Insertion metric is computed similarly:\n\n```\ninsertion = Insertion(data_shape,explanation_shape,bound_max_step=True)\nresult_dic = insertion(model,data,explanations,class_to_explain)\niauc = result_dic[\"iauc\"]\nic = result_dic[\"ic\"]\n```\n\n## Changing the filling method\n\nThese metrics work by removing parts of the image and replacing it with something else, for e.g. black pixels (Deletion, AD, IIC, ADD) or a blurred version of the input image (Insertion).\nThe default replacing method can be changed with the ```data_replace_method``` argument passed to the constructor:\n\n```\ninsertion = Insertion(data_shape,explanation_shape,bound_max_step=True,data_replace_method=\"black\")\nadd = ADD(data_replace_method=\"blur\")\n```\nCurrently, the autorized values are:\n- \"black\": replaces with black pixels\n- \"blur\": replaces with a blurred version of the input image\n\n## Demonstration\n\nLook at the ```demo.ipynb``` script for a demonstration.\nIf you want to re-run the demo, you should download the [model's weights](https://drive.google.com/file/d/1JdHJjvCb9IAtcwKizo_KGLDR73UmtFYY/view?usp=sharing) pretrained on the CUB dataset and put it on the project's root.\nYou should also download the [CUB test dataset](http://www.vision.caltech.edu/datasets/cub_200_2011/) and put it in a \"data\" folder located at the project's root.\nThe dataset should be formated as expected by the ```torchvision.datasets.ImageFolder``` class from ```torchvision```.\n",
"bugtrack_url": null,
"license": "",
"summary": "This repository implements the faithfulness metrics mentionned in the paper --Computing and evaluating saliency maps for image classification: a tutorial-- in Pytorch.",
"version": "0.0.24",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "849c7dc9e84e4c2746636d2a7ef909aa24e2cc6aa4da83f26c194dd5248f1244",
"md5": "ed18eefdb0903f1bcf16dd1d504c2670",
"sha256": "21bf5f2755c5a15d6683614fe13c9ed75d0317b7c330f12d17ec72f7b62102e4"
},
"downloads": -1,
"filename": "saliency_maps_metrics-0.0.24-py3-none-any.whl",
"has_sig": false,
"md5_digest": "ed18eefdb0903f1bcf16dd1d504c2670",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 8252,
"upload_time": "2023-04-26T07:21:32",
"upload_time_iso_8601": "2023-04-26T07:21:32.944898Z",
"url": "https://files.pythonhosted.org/packages/84/9c/7dc9e84e4c2746636d2a7ef909aa24e2cc6aa4da83f26c194dd5248f1244/saliency_maps_metrics-0.0.24-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "bc544c4c216e911c2aa7e96695aecccb3767fd5efdb6e038dc67c57ebbb4b9e8",
"md5": "69c7ef949527ed9952b6e73dabdad9c0",
"sha256": "aeeeac0b1b3398bf63b3d26c1e56c7df82ce3d1c86e6a69dbe7ffbfbd0ca1ccd"
},
"downloads": -1,
"filename": "saliency_maps_metrics-0.0.24.tar.gz",
"has_sig": false,
"md5_digest": "69c7ef949527ed9952b6e73dabdad9c0",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 7444,
"upload_time": "2023-04-26T07:21:35",
"upload_time_iso_8601": "2023-04-26T07:21:35.048832Z",
"url": "https://files.pythonhosted.org/packages/bc/54/4c4c216e911c2aa7e96695aecccb3767fd5efdb6e038dc67c57ebbb4b9e8/saliency_maps_metrics-0.0.24.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-04-26 07:21:35",
"github": false,
"gitlab": false,
"bitbucket": false,
"lcname": "saliency-maps-metrics"
}
