# ExplainaBoard: An Explainable Leaderboard for NLP
<img src="./fig/logo-full-v2.png" width="800" class="center">
[](https://github.com/neulab/ExplainaBoard/blob/main/LICENSE)
[](https://github.com/neulab/ExplainaBoard/stargazers)
[](https://pypi.org/project/explainaboard/)
[](https://github.com/psf/black)
[](.github/workflows/ci.yml)
## What is ExplainaBoard?
When developing a natural language processing (NLP or AI) system, often one of the
hardest things is to understand where your system is working and where it is failing,
and deciding what to do next. ExplainaBoard is a tool that *inspects your system
outputs*, *identifies what is working and what is not working*, and helps *inspire you
with ideas of where to go next*.
<img src="./fig/intro.png" width="300" align="right">
It offers a number of different ways with which you can evaluate and understand your systems:
1. *Single-system Analysis*: What is a system good or bad at?
2. *Pairwise Analysis*: Where is one system better (worse) than another?
3. *Fine-grained Error Analysis*: On what examples do errors occur?
4. *Holistic Leaderboards and Benchmarks*: Which systems perform best for a particular task?
## Using Explainaboard
ExplainaBoard can be used online or offline. For most users, we recommend using the
online interface, as it is more interactive and easier to get started.
### Online Usage
Browse the [**web interface**](https://explainaboard.inspiredco.ai), which gives you the
ability to browse outputs and evaluate and analyze your own system outputs.
<a href="https://explainaboard.inspiredco.ai"><img src="./fig/demo-v2.png" width="500" class="center"></a>
If you would like to evaluate and analyze your own systems programmatically, you can
use the
<a href="https://github.com/neulab/explainaboard_client">ExplainaBoard client</a>.
### Offline Usage
For power-users who want to use ExplainaBoard offline, first, follow the installation directions
below, then take a look at our [**CLI examples**](docs/cli_interface.md).
**Install Method 1 - Standard Use:** Simple installation from PyPI (Python 3 only)
```bash
pip install --upgrade pip # recommending the newest version of pip.
pip install explainaboard
python -m spacy download en_core_web_sm # if you plan to use the AspectBasedSentimentClassificationProcessor
```
**Install Method 2 - Development:** Install from the source and develop locally (Python
3 only)
```bash
# Clone current repo
git clone https://github.com/neulab/ExplainaBoard.git
cd ExplainaBoard
# Install the required dependencies and dev dependencies
pip install ."[dev]"
python -m spacy download en_core_web_sm
pre-commit install
```
- **Testing:** To run tests, you can run `python -m unittest`.
- **Linting and Code Style:** This project uses flake8 (linter) and black (formatter).
They are enforced in the pre-commit hook and in the CI pipeline.
- run `python -m black .` to format code
- run `flake8` to lint code
- You can also configure your IDE to automatically format and lint the files as you
are writing code.
After trying things out in the [CLI](docs/cli_interface.md), you can read how to add
new [features](docs/add_new_features.md), [tasks](docs/add_new_tasks.md), or
[file formats](docs/add_new_formats.md).
## Acknowledgement
ExplainaBoard is developed by Carnegie Mellon University, Inspired Cognition Inc., and
other collaborators. If you find it useful in research, you can cite it in papers:
```text
@inproceedings{liu-etal-2021-explainaboard,
title = "{E}xplaina{B}oard: An Explainable Leaderboard for {NLP}",
author = "Liu, Pengfei and Fu, Jinlan and Xiao, Yang and Yuan, Weizhe and Chang, Shuaichen and Dai, Junqi and Liu, Yixin and Ye, Zihuiwen and Neubig, Graham",
booktitle = "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations",
month = aug,
year = "2021",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.acl-demo.34",
doi = "10.18653/v1/2021.acl-demo.34",
pages = "280--289",
}
```
We thanks all authors who shared their system outputs with us: Ikuya Yamada, Stefan Schweter,
Colin Raffel, Yang Liu, Li Dong. We also thank
Vijay Viswanathan, Yiran Chen, Hiroaki Hayashi for useful discussion and feedback about ExplainaBoard.
Raw data
{
"_id": null,
"home_page": "https://github.com/neulab/ExplainaBoard",
"name": "explainaboard",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "",
"author": "Pengfei Liu",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/36/72/928a57bac44391e9e6b68a4c91ae0f2ef85e0be8ecd3c5c9490a226b10f2/explainaboard-0.12.7.tar.gz",
"platform": null,
"description": "# ExplainaBoard: An Explainable Leaderboard for NLP\n\n<img src=\"./fig/logo-full-v2.png\" width=\"800\" class=\"center\">\n\n[](https://github.com/neulab/ExplainaBoard/blob/main/LICENSE)\n[](https://github.com/neulab/ExplainaBoard/stargazers)\n[](https://pypi.org/project/explainaboard/)\n[](https://github.com/psf/black)\n[](.github/workflows/ci.yml)\n\n## What is ExplainaBoard?\n\nWhen developing a natural language processing (NLP or AI) system, often one of the\nhardest things is to understand where your system is working and where it is failing,\nand deciding what to do next. ExplainaBoard is a tool that *inspects your system\noutputs*, *identifies what is working and what is not working*, and helps *inspire you\nwith ideas of where to go next*.\n\n<img src=\"./fig/intro.png\" width=\"300\" align=\"right\">\n\nIt offers a number of different ways with which you can evaluate and understand your systems:\n\n1. *Single-system Analysis*: What is a system good or bad at?\n2. *Pairwise Analysis*: Where is one system better (worse) than another?\n3. *Fine-grained Error Analysis*: On what examples do errors occur?\n4. *Holistic Leaderboards and Benchmarks*: Which systems perform best for a particular task?\n\n## Using Explainaboard\n\nExplainaBoard can be used online or offline. For most users, we recommend using the\nonline interface, as it is more interactive and easier to get started.\n\n### Online Usage\n\nBrowse the [**web interface**](https://explainaboard.inspiredco.ai), which gives you the\nability to browse outputs and evaluate and analyze your own system outputs.\n\n<a href=\"https://explainaboard.inspiredco.ai\"><img src=\"./fig/demo-v2.png\" width=\"500\" class=\"center\"></a>\n\nIf you would like to evaluate and analyze your own systems programmatically, you can\nuse the\n<a href=\"https://github.com/neulab/explainaboard_client\">ExplainaBoard client</a>.\n\n### Offline Usage\n\nFor power-users who want to use ExplainaBoard offline, first, follow the installation directions\nbelow, then take a look at our [**CLI examples**](docs/cli_interface.md).\n\n**Install Method 1 - Standard Use:** Simple installation from PyPI (Python 3 only)\n\n```bash\npip install --upgrade pip # recommending the newest version of pip.\npip install explainaboard\npython -m spacy download en_core_web_sm # if you plan to use the AspectBasedSentimentClassificationProcessor\n```\n\n**Install Method 2 - Development:** Install from the source and develop locally (Python\n3 only)\n\n```bash\n# Clone current repo\ngit clone https://github.com/neulab/ExplainaBoard.git\ncd ExplainaBoard\n\n# Install the required dependencies and dev dependencies\npip install .\"[dev]\"\npython -m spacy download en_core_web_sm\npre-commit install\n```\n\n- **Testing:** To run tests, you can run `python -m unittest`.\n- **Linting and Code Style:** This project uses flake8 (linter) and black (formatter).\n They are enforced in the pre-commit hook and in the CI pipeline.\n - run `python -m black .` to format code\n - run `flake8` to lint code\n - You can also configure your IDE to automatically format and lint the files as you\n are writing code.\n\nAfter trying things out in the [CLI](docs/cli_interface.md), you can read how to add\nnew [features](docs/add_new_features.md), [tasks](docs/add_new_tasks.md), or\n[file formats](docs/add_new_formats.md).\n\n## Acknowledgement\n\nExplainaBoard is developed by Carnegie Mellon University, Inspired Cognition Inc., and\nother collaborators. If you find it useful in research, you can cite it in papers:\n\n```text\n@inproceedings{liu-etal-2021-explainaboard,\n title = \"{E}xplaina{B}oard: An Explainable Leaderboard for {NLP}\",\n author = \"Liu, Pengfei and Fu, Jinlan and Xiao, Yang and Yuan, Weizhe and Chang, Shuaichen and Dai, Junqi and Liu, Yixin and Ye, Zihuiwen and Neubig, Graham\",\n booktitle = \"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations\",\n month = aug,\n year = \"2021\",\n address = \"Online\",\n publisher = \"Association for Computational Linguistics\",\n url = \"https://aclanthology.org/2021.acl-demo.34\",\n doi = \"10.18653/v1/2021.acl-demo.34\",\n pages = \"280--289\",\n}\n```\n\nWe thanks all authors who shared their system outputs with us: Ikuya Yamada, Stefan Schweter,\nColin Raffel, Yang Liu, Li Dong. We also thank\nVijay Viswanathan, Yiran Chen, Hiroaki Hayashi for useful discussion and feedback about ExplainaBoard.\n\n\n",
"bugtrack_url": null,
"license": "MIT License",
"summary": "Explainable Leaderboards for Natural Language Processing",
"version": "0.12.7",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2b13e936c93fad94fce9f29689a4fc3465973d151b2e8820d636d58c72583e93",
"md5": "3753c2531929f338ec8d5a8acb8c8c0a",
"sha256": "cf8ebacd9feb18908e837daae866a213180ff70791ca6ef3cd60b1ab235db163"
},
"downloads": -1,
"filename": "explainaboard-0.12.7-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "3753c2531929f338ec8d5a8acb8c8c0a",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": null,
"size": 278570,
"upload_time": "2023-03-10T13:42:16",
"upload_time_iso_8601": "2023-03-10T13:42:16.099268Z",
"url": "https://files.pythonhosted.org/packages/2b/13/e936c93fad94fce9f29689a4fc3465973d151b2e8820d636d58c72583e93/explainaboard-0.12.7-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3672928a57bac44391e9e6b68a4c91ae0f2ef85e0be8ecd3c5c9490a226b10f2",
"md5": "0557d8b9a6f85bece44821245f3e72d6",
"sha256": "eb710a8f1c3f8ac368cefd1c0ddac59a8db9dfad0f57aa0ef35a9ba3a717cc15"
},
"downloads": -1,
"filename": "explainaboard-0.12.7.tar.gz",
"has_sig": false,
"md5_digest": "0557d8b9a6f85bece44821245f3e72d6",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 177846,
"upload_time": "2023-03-10T13:42:17",
"upload_time_iso_8601": "2023-03-10T13:42:17.717133Z",
"url": "https://files.pythonhosted.org/packages/36/72/928a57bac44391e9e6b68a4c91ae0f2ef85e0be8ecd3c5c9490a226b10f2/explainaboard-0.12.7.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-03-10 13:42:17",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "neulab",
"github_project": "ExplainaBoard",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "explainaboard"
}
JSON
