<p align = "center">
<img src="https://raw.githubusercontent.com/chus-chus/teex/master/docs/images/teex_logo__.png"
alt="Our AI generated logo. Comes from the prompt: 'logo of a t, inspired by an AI that is fair and responsible.'" width="115"/>
# teex: a toolbox for evaluating XAI explanations
[](https://img.shields.io/pypi/v/teex)
[](https://img.shields.io/github/issues/chus-chus/teex)
[](https://codecov.io/gh/chus-chus/teex)
[](https://teex.readthedocs.io/en/latest/?badge=latest)
A Python **t**oolbox for the **e**valuation of machine learning **ex**planations.
This project aims to provide a simple way of **evaluating** individual black box explanations against ground truth. Moreover, it contains a collection of easy-to-access datasets with available g.t. explanations.
## Installation
The teex package is on [PyPI](https://pypi.org/project/teex/). To install it, simply run
```shell
pip install teex
```
**teex** is compatible with Python 3.8 and 3.9.
## Documentation
**teex**'s documentation, in-depth examples and API reference can be found on [Read The Docs](https://teex.readthedocs.io).
## Usage overview
`teex` is divided into subpackages, one for each explanation type. Each subpackage contains two modules, focused on two
distinct functionalities:
- **eval**: contains _**evaluation**_ methods for that particular explanation type. For every subpackage, there is one high-level
function to easily compute all the available metrics for an arbitrary number of explanations.
- **data**: contains _**data**_ classes with available g.t. explanations of that particular
explanation type, both synthetic and real. All of them are objects that need to be instanced and, when sliced,
will return the data, the target and the ground truth explanations, respectively.
### Evaluation (with feature importance as an example)
**What are feature importance vectors?** They are vectors with one entry per feature. Each entry contains a weight that
represents a feature's importance for the observation's outcome. Weights are usually in the range $[-1, 1]$.
Suppose that we have a dataset with available g.t. explanations (``gtExps``) and a model trained with it (``model``):
```python
from teex.featureImportance.eval import feature_importance_scores
# get individual feature importance explanations with any method
predictedExps = get_explanations(model, X)
# evaluate predicted explanations against ground truths
feature_importance_scores(gtExps, predictedExps, metrics=['fscore', 'cs', 'auc'])
```
This basic syntax is followed by the main evaluation APIs of all 4 explanation types:
- **Feature Importance**: ``feature_importance_scores``
- **Saliency Maps**: ``saliency_map_scores``
- **Decision Rules**: ``rule_scores``
- **Word Importance**: ``word_importance_scores``
Other functionalities are included in each evaluation module. More about each explanation type can be found in the example notebooks and the documentation.
#### Metrics supported:
Metrics available as of `v1.0.0` are
- **Feature Importance**
- **Cosine Similarity**: similarity between the two vectors is measured in an inner product space in terms of orientation.
- **ROC AUC**: where the ground truth is binarized in order for it to represent a class and the predicted vector entries are interpreted as classification scores or likelihood.
- **F1 Score**: where both ground truth and prediction are binarized according to a user-defined threshold.
- **Precision**: g.t. and prediction treated as in F1 Score
- **Recall**: g.t. and prediction treated as in F1 Score
- **Saliency Maps**
- Same metrics as in feature importance. Each pixel in an image is considered to be a feature.
- **Decision Rules**
- **Complete Rule Quality**: Proportion of lower and upper bounds in a rule explanation whose that are $\epsilon$-close to the respective lower and upper bounds (same feature) in the ground truth rule explanation amongst those that are not infinity.
- All metrics in feature importance, where a transformation of the rule into feature importance vectors is performed first. See doc. for details.
- **Word Importance**:
- All metrics in feature importance, where a vocabulary is considered the feature space and a word importance explanation may or may not contain words from the vocabulary.
Note how in **teex**, feature importance vectors are a universal representation: we 'translate' all other explanation types
to feature importance vectors to allow a wider metric space.
### Data
**teex** also provides an easy way to get and use data with available ground truth explanations. It contains real datasets and can generate synthetic ones.
All of them are instanced as objects, and can be sliced as usual. For example:
```python
from teex.saliencyMap.data import Kahikatea
X, y, exps = Kahikatea()[:]
```
downloads and assigns data from the Kahikatea dataset:
<p align = "center">
<img src="https://raw.githubusercontent.com/chus-chus/teex/master/docs/images/kahikatea_sample.png"
alt="drawing" width="200"/>
<img src="https://raw.githubusercontent.com/chus-chus/teex/master/docs/images/kahikatea_gt.png" alt="drawing" width="200"/>
</p>
<body>
<p align = "center">Fig. 1 A <a href="https://zenodo.org/record/5059769#.YN7KKegzZPZ">Kahikatea</a> dataset sample. </p>
</body>
Other datasets, such as [CUB-200-2011](https://www.vision.caltech.edu/datasets/cub_200_2011/) and the [Oxford-IIIT Pet Dataset](https://www.robots.ox.ac.uk/~vgg/data/pets/), are available on **teex**, with over 19000 images and 230 distinct classes:
```python
from teex.saliencyMap.data import CUB200
X, y, exps = CUB200()[:]
```
<p align = "center">
<img src="https://raw.githubusercontent.com/chus-chus/teex/master/docs/images/cub_sample.jpg"
alt="drawing" width="200"/>
<img src="https://raw.githubusercontent.com/chus-chus/teex/master/docs/images/cub_gt.png" alt="drawing" width="200"/>
</p>
<body>
<p align = "center">Fig. 2 A <a href="https://www.vision.caltech.edu/datasets/cub_200_2011/">CUB-200-2011</a> dataset sample. </p>
</body>
```python
from teex.saliencyMap.data import OxfordIIIT
X, y, exps = OxfordIIIT()[:]
```
<p align = "center">
<img src="https://raw.githubusercontent.com/chus-chus/teex/master/docs/images/ox_sample.jpg"
alt="drawing" width="200"/>
<img src="https://raw.githubusercontent.com/chus-chus/teex/master/docs/images/ox_gt.png" alt="drawing" width="200"/>
</p>
<body>
<p align = "center">Fig. 3 An <a href="https://www.robots.ox.ac.uk/~vgg/data/pets/">Oxford-IIIT Pet Dataset</a> sample. </p>
</body>
Synthetic datasets can also be easily generated:
```python
from teex.saliencyMap.data import SenecaSM
X, y, exps = SenecaSM()[:]
```
<p align = "center">
<img src="https://raw.githubusercontent.com/chus-chus/teex/master/docs/images/seneca_sm_sample.png"
alt="drawing" width="200"/>
<img src="https://raw.githubusercontent.com/chus-chus/teex/master/docs/images/seneca_sm_gt.png" alt="drawing" width="200"/>
</p>
<body>
<p align = "center">Fig. 4 Artificial image and its g.t. saliency map explanation.
</p>
</body>
Datasets for all other explanation types are available too.
## Tutorials and demos
---
*Saliency maps*
- [Improving model selection with explanation quality](https://teex.readthedocs.io/en/latest/demos/model_selection/model_selection_nb.html)
- [Retrieving image data with g.t. saliency map explanations](https://teex.readthedocs.io/en/latest/demos/saliency_map/gen_saliency_map_nb.html)
- [Evaluating Captum saliency map explanations](https://teex.readthedocs.io/en/latest/demos/saliency_map/eval_saliency_map_nb.html)
---
*Feature importance vectors*
- [Retrieving tabular data with g.t. feature importance explanations](https://teex.readthedocs.io/en/latest/demos/feature_importance/gen_feature_importance_nb.html)
- [Evaluating LIME feature importance explanations](https://teex.readthedocs.io/en/latest/demos/feature_importance/eval_feature_importance_nb.html)
---
*Decision rules*
- [Retrieving tabular data with g.t. decision rule explanations](https://teex.readthedocs.io/en/latest/demos/decision_rule/gen_decision_rule_nb.html)
- [Evaluating decision rule explanations](https://teex.readthedocs.io/en/latest/demos/decision_rule/eval_decision_rule_nb.html)
---
*Word importance vectors*
- [Retrieving language data with g.t. word importance explanations](https://teex.readthedocs.io/en/latest/demos/word_importance/gen_word_importance_nb.html)
- [Evaluating word importance explanations](https://teex.readthedocs.io/en/latest/demos/word_importance/eval_word_importance_nb.html)
## Contributing
There is still work to do and we would really appreciate your help. Before contributing to **teex**, please take a moment to read the [manual](https://github.com/chus-chus/teex/blob/main/CONTRIBUTING.md).
## Acknowledgements
This work has been made possible by the [University of Waikato](https://www.waikato.ac.nz/) under the scope of
the [TAIAO](https://taiao.ai/) project.
<p align = "center">
<a href="https://taiao.ai">
<img src="https://raw.githubusercontent.com/chus-chus/teex/main/docs/images/TAIAO_logo.png" alt="drawing" width="150"/>
</a>
<a href="https://www.waikato.ac.nz/">
<img src="https://upload.wikimedia.org/wikipedia/en/thumb/b/bd/University_of_Waikato_logo.svg/1200px-University_of_Waikato_logo.svg.png" alt="drawing" width="45"/>
</a>
<a href="https://www.upc.edu/en">
<img src="https://upload.wikimedia.org/wikipedia/commons/thumb/9/97/Logo_UPC.svg/2048px-Logo_UPC.svg.png" alt="drawing" width="50"/>
</a>
</p>
Raw data
{
"_id": null,
"home_page": "https://github.com/chus-chus/teex",
"name": "teex",
"maintainer": "",
"docs_url": null,
"requires_python": "<3.10,>=3.8",
"maintainer_email": "",
"keywords": "",
"author": "Jesus Antonanzas",
"author_email": "jesus.maria.antonanzas@estudiantat.upc.edu",
"download_url": "https://files.pythonhosted.org/packages/c0/59/c48b66552d6fbaac8a950a1cc7102237a9e8db703fbe2cfa3e162644ce74/teex-1.1.3.tar.gz",
"platform": null,
"description": "<p align = \"center\">\n <img src=\"https://raw.githubusercontent.com/chus-chus/teex/master/docs/images/teex_logo__.png\" \n alt=\"Our AI generated logo. Comes from the prompt: 'logo of a t, inspired by an AI that is fair and responsible.'\" width=\"115\"/>\n\n# teex: a toolbox for evaluating XAI explanations\n\n[](https://img.shields.io/pypi/v/teex)\n[](https://img.shields.io/github/issues/chus-chus/teex)\n[](https://codecov.io/gh/chus-chus/teex)\n\n[](https://teex.readthedocs.io/en/latest/?badge=latest)\n\nA Python **t**oolbox for the **e**valuation of machine learning **ex**planations.\n\nThis project aims to provide a simple way of **evaluating** individual black box explanations against ground truth. Moreover, it contains a collection of easy-to-access datasets with available g.t. explanations.\n\n## Installation\n\nThe teex package is on [PyPI](https://pypi.org/project/teex/). To install it, simply run\n\n```shell\npip install teex\n```\n**teex** is compatible with Python 3.8 and 3.9.\n\n## Documentation\n\n**teex**'s documentation, in-depth examples and API reference can be found on [Read The Docs](https://teex.readthedocs.io).\n\n## Usage overview\n\n`teex` is divided into subpackages, one for each explanation type. Each subpackage contains two modules, focused on two\ndistinct functionalities:\n\n- **eval**: contains _**evaluation**_ methods for that particular explanation type. For every subpackage, there is one high-level\n function to easily compute all the available metrics for an arbitrary number of explanations.\n- **data**: contains _**data**_ classes with available g.t. explanations of that particular \n explanation type, both synthetic and real. All of them are objects that need to be instanced and, when sliced,\n will return the data, the target and the ground truth explanations, respectively.\n \n### Evaluation (with feature importance as an example)\n\n**What are feature importance vectors?** They are vectors with one entry per feature. Each entry contains a weight that \nrepresents a feature's importance for the observation's outcome. Weights are usually in the range $[-1, 1]$.\n\nSuppose that we have a dataset with available g.t. explanations (``gtExps``) and a model trained with it (``model``):\n\n```python\nfrom teex.featureImportance.eval import feature_importance_scores\n\n# get individual feature importance explanations with any method\npredictedExps = get_explanations(model, X)\n\n# evaluate predicted explanations against ground truths\nfeature_importance_scores(gtExps, predictedExps, metrics=['fscore', 'cs', 'auc'])\n```\n\nThis basic syntax is followed by the main evaluation APIs of all 4 explanation types:\n\n- **Feature Importance**: ``feature_importance_scores``\n- **Saliency Maps**: ``saliency_map_scores``\n- **Decision Rules**: ``rule_scores``\n- **Word Importance**: ``word_importance_scores``\n\nOther functionalities are included in each evaluation module. More about each explanation type can be found in the example notebooks and the documentation.\n\n#### Metrics supported:\n\nMetrics available as of `v1.0.0` are\n\n- **Feature Importance**\n - **Cosine Similarity**: similarity between the two vectors is measured in an inner product space in terms of orientation.\n - **ROC AUC**: where the ground truth is binarized in order for it to represent a class and the predicted vector entries are interpreted as classification scores or likelihood.\n - **F1 Score**: where both ground truth and prediction are binarized according to a user-defined threshold.\n - **Precision**: g.t. and prediction treated as in F1 Score\n - **Recall**: g.t. and prediction treated as in F1 Score\n- **Saliency Maps**\n - Same metrics as in feature importance. Each pixel in an image is considered to be a feature.\n- **Decision Rules**\n - **Complete Rule Quality**: Proportion of lower and upper bounds in a rule explanation whose that are $\\epsilon$-close to the respective lower and upper bounds (same feature) in the ground truth rule explanation amongst those that are not infinity.\n - All metrics in feature importance, where a transformation of the rule into feature importance vectors is performed first. See doc. for details.\n- **Word Importance**:\n - All metrics in feature importance, where a vocabulary is considered the feature space and a word importance explanation may or may not contain words from the vocabulary.\n\nNote how in **teex**, feature importance vectors are a universal representation: we 'translate' all other explanation types\nto feature importance vectors to allow a wider metric space. \n\n### Data\n\n**teex** also provides an easy way to get and use data with available ground truth explanations. It contains real datasets and can generate synthetic ones.\nAll of them are instanced as objects, and can be sliced as usual. For example:\n\n```python\nfrom teex.saliencyMap.data import Kahikatea\nX, y, exps = Kahikatea()[:]\n```\n\ndownloads and assigns data from the Kahikatea dataset: \n\n<p align = \"center\">\n <img src=\"https://raw.githubusercontent.com/chus-chus/teex/master/docs/images/kahikatea_sample.png\" \n alt=\"drawing\" width=\"200\"/>\n <img src=\"https://raw.githubusercontent.com/chus-chus/teex/master/docs/images/kahikatea_gt.png\" alt=\"drawing\" width=\"200\"/>\n</p>\n<body>\n <p align = \"center\">Fig. 1 A <a href=\"https://zenodo.org/record/5059769#.YN7KKegzZPZ\">Kahikatea</a> dataset sample. </p>\n</body>\n\nOther datasets, such as [CUB-200-2011](https://www.vision.caltech.edu/datasets/cub_200_2011/) and the [Oxford-IIIT Pet Dataset](https://www.robots.ox.ac.uk/~vgg/data/pets/), are available on **teex**, with over 19000 images and 230 distinct classes:\n\n```python\n from teex.saliencyMap.data import CUB200\n X, y, exps = CUB200()[:]\n```\n\n<p align = \"center\">\n <img src=\"https://raw.githubusercontent.com/chus-chus/teex/master/docs/images/cub_sample.jpg\" \n alt=\"drawing\" width=\"200\"/>\n <img src=\"https://raw.githubusercontent.com/chus-chus/teex/master/docs/images/cub_gt.png\" alt=\"drawing\" width=\"200\"/>\n</p>\n<body>\n <p align = \"center\">Fig. 2 A <a href=\"https://www.vision.caltech.edu/datasets/cub_200_2011/\">CUB-200-2011</a> dataset sample. </p>\n</body>\n \n```python\n from teex.saliencyMap.data import OxfordIIIT\n X, y, exps = OxfordIIIT()[:]\n```\n\n<p align = \"center\">\n <img src=\"https://raw.githubusercontent.com/chus-chus/teex/master/docs/images/ox_sample.jpg\" \n alt=\"drawing\" width=\"200\"/>\n <img src=\"https://raw.githubusercontent.com/chus-chus/teex/master/docs/images/ox_gt.png\" alt=\"drawing\" width=\"200\"/>\n</p>\n<body>\n <p align = \"center\">Fig. 3 An <a href=\"https://www.robots.ox.ac.uk/~vgg/data/pets/\">Oxford-IIIT Pet Dataset</a> sample. </p>\n</body>\n\n\nSynthetic datasets can also be easily generated:\n\n```python\nfrom teex.saliencyMap.data import SenecaSM\nX, y, exps = SenecaSM()[:]\n```\n\n<p align = \"center\">\n <img src=\"https://raw.githubusercontent.com/chus-chus/teex/master/docs/images/seneca_sm_sample.png\" \n alt=\"drawing\" width=\"200\"/>\n <img src=\"https://raw.githubusercontent.com/chus-chus/teex/master/docs/images/seneca_sm_gt.png\" alt=\"drawing\" width=\"200\"/>\n</p>\n<body>\n <p align = \"center\">Fig. 4 Artificial image and its g.t. saliency map explanation.\n </p>\n</body>\n\nDatasets for all other explanation types are available too.\n\n## Tutorials and demos\n\n---\n*Saliency maps*\n- [Improving model selection with explanation quality](https://teex.readthedocs.io/en/latest/demos/model_selection/model_selection_nb.html)\n- [Retrieving image data with g.t. saliency map explanations](https://teex.readthedocs.io/en/latest/demos/saliency_map/gen_saliency_map_nb.html)\n- [Evaluating Captum saliency map explanations](https://teex.readthedocs.io/en/latest/demos/saliency_map/eval_saliency_map_nb.html)\n---\n*Feature importance vectors*\n- [Retrieving tabular data with g.t. feature importance explanations](https://teex.readthedocs.io/en/latest/demos/feature_importance/gen_feature_importance_nb.html)\n- [Evaluating LIME feature importance explanations](https://teex.readthedocs.io/en/latest/demos/feature_importance/eval_feature_importance_nb.html)\n---\n*Decision rules*\n- [Retrieving tabular data with g.t. decision rule explanations](https://teex.readthedocs.io/en/latest/demos/decision_rule/gen_decision_rule_nb.html)\n- [Evaluating decision rule explanations](https://teex.readthedocs.io/en/latest/demos/decision_rule/eval_decision_rule_nb.html)\n---\n*Word importance vectors*\n- [Retrieving language data with g.t. word importance explanations](https://teex.readthedocs.io/en/latest/demos/word_importance/gen_word_importance_nb.html)\n- [Evaluating word importance explanations](https://teex.readthedocs.io/en/latest/demos/word_importance/eval_word_importance_nb.html)\n\n\n## Contributing\n\nThere is still work to do and we would really appreciate your help. Before contributing to **teex**, please take a moment to read the [manual](https://github.com/chus-chus/teex/blob/main/CONTRIBUTING.md).\n\n## Acknowledgements\nThis work has been made possible by the [University of Waikato](https://www.waikato.ac.nz/) under the scope of \nthe [TAIAO](https://taiao.ai/) project.\n\n<p align = \"center\">\n <a href=\"https://taiao.ai\">\n <img src=\"https://raw.githubusercontent.com/chus-chus/teex/main/docs/images/TAIAO_logo.png\" alt=\"drawing\" width=\"150\"/>\n </a>\n <a href=\"https://www.waikato.ac.nz/\">\n <img src=\"https://upload.wikimedia.org/wikipedia/en/thumb/b/bd/University_of_Waikato_logo.svg/1200px-University_of_Waikato_logo.svg.png\" alt=\"drawing\" width=\"45\"/>\n </a> \n <a href=\"https://www.upc.edu/en\">\n <img src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/9/97/Logo_UPC.svg/2048px-Logo_UPC.svg.png\" alt=\"drawing\" width=\"50\"/>\n </a>\n</p>\n",
"bugtrack_url": null,
"license": "",
"summary": "A Toolbox for the Evaluation of Explanations",
"version": "1.1.3",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "afca66eeaae88f0d7e80fa6b12e199849fac64c54792b4a163da4cf3c6c1861c",
"md5": "20540fee06079579faba030d639cf0ff",
"sha256": "6c32cfa187336ad25fe9c3be7184a7fd7d136721cbc05bffa43b4522a278435d"
},
"downloads": -1,
"filename": "teex-1.1.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "20540fee06079579faba030d639cf0ff",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.10,>=3.8",
"size": 57172,
"upload_time": "2023-03-26T14:36:35",
"upload_time_iso_8601": "2023-03-26T14:36:35.076560Z",
"url": "https://files.pythonhosted.org/packages/af/ca/66eeaae88f0d7e80fa6b12e199849fac64c54792b4a163da4cf3c6c1861c/teex-1.1.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c059c48b66552d6fbaac8a950a1cc7102237a9e8db703fbe2cfa3e162644ce74",
"md5": "31882ce2d324c0fbc83f91325427558b",
"sha256": "6c6aa677497c6a3ad36dce8fd4a0f631269fa95cb61f03eb6347d15d2e8e0032"
},
"downloads": -1,
"filename": "teex-1.1.3.tar.gz",
"has_sig": false,
"md5_digest": "31882ce2d324c0fbc83f91325427558b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.10,>=3.8",
"size": 50344,
"upload_time": "2023-03-26T14:36:36",
"upload_time_iso_8601": "2023-03-26T14:36:36.958746Z",
"url": "https://files.pythonhosted.org/packages/c0/59/c48b66552d6fbaac8a950a1cc7102237a9e8db703fbe2cfa3e162644ce74/teex-1.1.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-03-26 14:36:36",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "chus-chus",
"github_project": "teex",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"requirements": [],
"lcname": "teex"
}
JSON
