# :robot::speech_balloon::question: nlp-uncertainty-zoo
This repository contains implementations of several models used for uncertainty estimation in Natural Language processing,
implemented in PyTorch. You can install the repository using pip:
pip3 install nlp-uncertainty-zoo
If you are using the repository in your academic research, please cite the paper below:
@inproceedings{ulmer-etal-2022-exploring,
title = "Exploring Predictive Uncertainty and Calibration in {NLP}: A Study on the Impact of Method {\&} Data Scarcity",
author = "Ulmer, Dennis and
Frellsen, Jes and
Hardmeier, Christian",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2022",
month = dec,
year = "2022",
address = "Abu Dhabi, United Arab Emirates",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.findings-emnlp.198",
pages = "2707--2735",
}
To learn more about the package, consult the documentation [here](http://dennisulmer.eu/nlp-uncertainty-zoo/),
check a Jupyter notebook demo [here](https://github.com/Kaleidophon/nlp-uncertainty-zoo/blob/main/demo.ipynb) or a Google
collab [here](https://colab.research.google.com/drive/1-Pl5lvcnpbGL2ZXLGDDNqvJB7Ew8uIsS?usp=sharing).
### Included models
The following models are implemented in the repository. They can all be imported by using `from nlp-uncertainty-zoo import <MODEL>`.
For transformer-based model, furthermore a version of a model is available that uses a pre-trained BERT from the HuggingFace `transformers`.
| Name | Description | Implementation | Paper |
|---|---|---|---|
| LSTM | Vanilla LSTM | `LSTM` | [Hochreiter & Schmidhuber, 1997](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.676.4320&rep=rep1&type=pdf) |
| LSTM Ensemble | Ensemble of LSTMs | `LSTMEnsemble` | [Lakshminarayanan et al., 2017](https://proceedings.neurips.cc/paper/2017/file/9ef2ed4b7fd2c810847ffa5fa85bce38-Paper.pdf) |
| Bayesian LSTM | LSTM implementing Bayes-by-backprop [Blundell et al, 2015](http://proceedings.mlr.press/v37/blundell15.pdf) | `BayesianLSTM` | [Fortunato et al, 2017](https://arxiv.org/pdf/1704.02798.pdf) |
| ST-tau LSTM | LSTM modelling transitions of a finite-state-automaton | `STTauLSTM` | [Wang et al., 2021](https://openreview.net/pdf?id=9EKHN1jOlA) |
| Variational LSTM | LSTM with MC Dropout [(Gal & Ghahramani, 2016a)](http://proceedings.mlr.press/v48/gal16.pdf) | `VariationalLSTM` | [Gal & Ghahramani, 2016b](https://proceedings.neurips.cc/paper/2016/file/076a0c97d09cf1a0ec3e19c7f2529f2b-Paper.pdf) |
| DDU Transformer, DDU BERT | Transformer / BERT with Gaussian Mixture Model fit to hidden features | `DDUTransformer`, `DDUBert` | [Mukhoti et al, 2021](https://arxiv.org/pdf/2102.11582.pdf) |
| Variational Transformer, Variational BERT | Transformer / BERT with MC Dropout [(Gal & Ghahramani, 2016a)](http://proceedings.mlr.press/v48/gal16.pdf) | `VariationalTransformer`, `VariationalBert` | [Xiao et al., 2021](https://arxiv.org/pdf/2006.08344.pdf) |
| DPP Transformer, DPP Bert | Transformer / BERT using determinantal point process dropout | `DPPTransformer`, `DPPBert` | [Shelmanov et al., 2021](https://aclanthology.org/2021.eacl-main.157) |
| SNGP Transformer, SNGP BERT | Spectrally-normalized transformer / BERT using a Gaussian Process output layer | `SNGPTransformer`, `SNGPBert` | [Liu et al., 2022](http://arxiv.org/abs/2205.00403) |
Contributions to include even more approaches are much appreciated!
### Usage
Each model comes in two versions, for instance `LSTMEnsemble` and `LSTMEnsembleModule`. The first one is supposed to be
used as an out-of-the-box solution, encapsulating all training logic and convenience functions. These include fitting
the model, prediction, getting the uncertainty for an input batch using a specific metric.
```python
model = LSTMEnsemble(**network_params, ensemble_size=10, is_sequence_classifer=False)
model.fit(train_split=train_dataloader)
model.get_logits(X)
model.get_predictions(X)
model.get_sequence_representation(X)
model.available_uncertainty_metrics
model.get_uncertainty(X)
model.get_uncertainty(X, metric_name="mutual_information")
```
In comparison, the `-Module` class is supposed to me more simple and bare-bones, only containing the core model logic.
It is intended for research purposes, and for others who would like to embed the model into their own code base. While
the model class (e.g. `LSTMEnsemble`) inherits from `Model` and would require to implement certain methods, any `Module` class
sticks closely to `torch.nn.Module`.
To check what arguments are required to initialize and use different models, check [the documentation here](http://nlpuncertaintyzoo.dennisulmer.eu/).
Also, check out the demo provided as a Jupyter notebook [here](https://github.com/Kaleidophon/nlp-uncertainty-zoo/blob/main/demo.ipynb) or a Google
collab [here](https://colab.research.google.com/drive/1-Pl5lvcnpbGL2ZXLGDDNqvJB7Ew8uIsS?usp=sharing).
### Repository structure
The repository has the following structure:
* `models`: All model implementations.
* `tests`: Unit tests. So far, only contains rudimentary tests to check that all output shapes are consistent between models and functions.
* `utils`: Utility code (see below)
* `utils/custom_types.py`: Custom types used in the repository for type annotations.
* `utils/data.py`: Module containing data collators, and data builders - which build the dataloaders for a type of task and a specific dataset. Currently, language modelling, sequence labeling and sequence classification are supported.
* `utils/metrics.py`: Implementations of uncertainty metrics.
* `utils/samplers.py`: Dataset subsamplers for language modelling, sequence labelling and sequence classification.
* `utils/task_eval.py`: Functions used to evaluate task performance.
* `utils/uncertainty_eval.py`: Function used to evaluate uncertainty quality.
* `utils/calibration_eval.py`: Function used to evaluate calibration quality.
* `config.py`: Define available datasets, model and tasks.
* `defaults.py`: Define default config parameters for sequence classification and language modelling (**Note**: These might not be very good parameters).
### Other features
* **Weights & Biases integration**: You can track your experiments easily with weights & biases by passing a `wandb_run` argument to `model.fit()`!
* **Easy fine-tuning via HuggingFace**: You can fine-tune arbitrary BERT models using their name from HuggingFace's `transformers`.
### Contributing
This repository is by no means perfect nor complete. If you find any bugs, please report them using the issue template,
and, if you also happen to provide a fix, create a pull request! A GitHub template is provided for that as well.
You would like to make a new addition to the repository? Follow the steps below:
* **Adding a new model**: To add a new model, add a new module in the `models` directory. You will also need to implement
a corresponding `Model` and `Module` class, inheriting from the classes of the same name in `models/model.py` and implementing all
required functions. `Model` is supposed to be an out-of-the-box solution that you can start experimenting right away, whil
`Module` should only include the most basic model logic in order to be easy to integrate into other codebases and allow tinkering.
* **Adding a new uncertainty metric**: To add a new uncertainty metric, add the function to `utils/metrics.py`. The function should take
the logits of a model and output an uncertainty score (the higher the score, the more uncertain the model). The function should output
a batch_size x sequence_length matrix, with batch_size x 1 for sequence classification tasks. After finishing the implementation, you can
add the metric to the `single_prediction_uncertainty_metrics` of the `models.model.Model` class and `multi_prediction_uncertainty_metrics` of `models.model.MultiPredictionMixin` (if applicable).
You would like to add something else? Create an issue or contact me at dennis {dot} ulmer {at} mailbox {dot} org!
Raw data
{
"_id": null,
"home_page": "https://github.com/Kaleidophon/nlp-uncertainty-zoo",
"name": "nlp-uncertainty-zoo",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.5.3",
"maintainer_email": "",
"keywords": "machine learning,deep learning,nlp,uncertainty,uncertainty estimationpytorch",
"author": "Dennis Ulmer",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/10/d4/0adbbb2c4d9383026ba1f073d90f8a521a87583d9b248aecd91ff88c278e/nlp-uncertainty-zoo-1.0.3.tar.gz",
"platform": null,
"description": "# :robot::speech_balloon::question: nlp-uncertainty-zoo\n\nThis repository contains implementations of several models used for uncertainty estimation in Natural Language processing,\nimplemented in PyTorch. You can install the repository using pip:\n\n pip3 install nlp-uncertainty-zoo\n\nIf you are using the repository in your academic research, please cite the paper below:\n\n @inproceedings{ulmer-etal-2022-exploring,\n title = \"Exploring Predictive Uncertainty and Calibration in {NLP}: A Study on the Impact of Method {\\&} Data Scarcity\",\n author = \"Ulmer, Dennis and\n Frellsen, Jes and\n Hardmeier, Christian\",\n booktitle = \"Findings of the Association for Computational Linguistics: EMNLP 2022\",\n month = dec,\n year = \"2022\",\n address = \"Abu Dhabi, United Arab Emirates\",\n publisher = \"Association for Computational Linguistics\",\n url = \"https://aclanthology.org/2022.findings-emnlp.198\",\n pages = \"2707--2735\",\n }\n\nTo learn more about the package, consult the documentation [here](http://dennisulmer.eu/nlp-uncertainty-zoo/),\ncheck a Jupyter notebook demo [here](https://github.com/Kaleidophon/nlp-uncertainty-zoo/blob/main/demo.ipynb) or a Google \ncollab [here](https://colab.research.google.com/drive/1-Pl5lvcnpbGL2ZXLGDDNqvJB7Ew8uIsS?usp=sharing).\n\n### Included models\n\nThe following models are implemented in the repository. They can all be imported by using `from nlp-uncertainty-zoo import <MODEL>`.\nFor transformer-based model, furthermore a version of a model is available that uses a pre-trained BERT from the HuggingFace `transformers`.\n\n| Name | Description | Implementation | Paper |\n|---|---|---|---|\n| LSTM | Vanilla LSTM | `LSTM` | [Hochreiter & Schmidhuber, 1997](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.676.4320&rep=rep1&type=pdf) |\n| LSTM Ensemble | Ensemble of LSTMs | `LSTMEnsemble` | [Lakshminarayanan et al., 2017](https://proceedings.neurips.cc/paper/2017/file/9ef2ed4b7fd2c810847ffa5fa85bce38-Paper.pdf) | \n| Bayesian LSTM | LSTM implementing Bayes-by-backprop [Blundell et al, 2015](http://proceedings.mlr.press/v37/blundell15.pdf) | `BayesianLSTM` | [Fortunato et al, 2017](https://arxiv.org/pdf/1704.02798.pdf) |\n| ST-tau LSTM | LSTM modelling transitions of a finite-state-automaton | `STTauLSTM` | [Wang et al., 2021](https://openreview.net/pdf?id=9EKHN1jOlA) |\n| Variational LSTM | LSTM with MC Dropout [(Gal & Ghahramani, 2016a)](http://proceedings.mlr.press/v48/gal16.pdf) | `VariationalLSTM` | [Gal & Ghahramani, 2016b](https://proceedings.neurips.cc/paper/2016/file/076a0c97d09cf1a0ec3e19c7f2529f2b-Paper.pdf) |\n| DDU Transformer, DDU BERT | Transformer / BERT with Gaussian Mixture Model fit to hidden features | `DDUTransformer`, `DDUBert` | [Mukhoti et al, 2021](https://arxiv.org/pdf/2102.11582.pdf) |\n| Variational Transformer, Variational BERT | Transformer / BERT with MC Dropout [(Gal & Ghahramani, 2016a)](http://proceedings.mlr.press/v48/gal16.pdf) | `VariationalTransformer`, `VariationalBert` | [Xiao et al., 2021](https://arxiv.org/pdf/2006.08344.pdf) |\n| DPP Transformer, DPP Bert | Transformer / BERT using determinantal point process dropout | `DPPTransformer`, `DPPBert` | [Shelmanov et al., 2021](https://aclanthology.org/2021.eacl-main.157) |\n| SNGP Transformer, SNGP BERT | Spectrally-normalized transformer / BERT using a Gaussian Process output layer | `SNGPTransformer`, `SNGPBert` | [Liu et al., 2022](http://arxiv.org/abs/2205.00403) |\n\nContributions to include even more approaches are much appreciated!\n\n### Usage\n\nEach model comes in two versions, for instance `LSTMEnsemble` and `LSTMEnsembleModule`. The first one is supposed to be \nused as an out-of-the-box solution, encapsulating all training logic and convenience functions. These include fitting \nthe model, prediction, getting the uncertainty for an input batch using a specific metric.\n\n```python\nmodel = LSTMEnsemble(**network_params, ensemble_size=10, is_sequence_classifer=False)\nmodel.fit(train_split=train_dataloader)\nmodel.get_logits(X)\nmodel.get_predictions(X)\nmodel.get_sequence_representation(X)\nmodel.available_uncertainty_metrics\nmodel.get_uncertainty(X)\nmodel.get_uncertainty(X, metric_name=\"mutual_information\")\n```\n\nIn comparison, the `-Module` class is supposed to me more simple and bare-bones, only containing the core model logic. \nIt is intended for research purposes, and for others who would like to embed the model into their own code base. While \nthe model class (e.g. `LSTMEnsemble`) inherits from `Model` and would require to implement certain methods, any `Module` class\nsticks closely to `torch.nn.Module`.\n\nTo check what arguments are required to initialize and use different models, check [the documentation here](http://nlpuncertaintyzoo.dennisulmer.eu/).\n\nAlso, check out the demo provided as a Jupyter notebook [here](https://github.com/Kaleidophon/nlp-uncertainty-zoo/blob/main/demo.ipynb) or a Google \ncollab [here](https://colab.research.google.com/drive/1-Pl5lvcnpbGL2ZXLGDDNqvJB7Ew8uIsS?usp=sharing).\n\n### Repository structure\n\nThe repository has the following structure:\n\n* `models`: All model implementations.\n* `tests`: Unit tests. So far, only contains rudimentary tests to check that all output shapes are consistent between models and functions.\n* `utils`: Utility code (see below)\n * `utils/custom_types.py`: Custom types used in the repository for type annotations.\n * `utils/data.py`: Module containing data collators, and data builders - which build the dataloaders for a type of task and a specific dataset. Currently, language modelling, sequence labeling and sequence classification are supported.\n * `utils/metrics.py`: Implementations of uncertainty metrics.\n * `utils/samplers.py`: Dataset subsamplers for language modelling, sequence labelling and sequence classification.\n * `utils/task_eval.py`: Functions used to evaluate task performance.\n * `utils/uncertainty_eval.py`: Function used to evaluate uncertainty quality.\n * `utils/calibration_eval.py`: Function used to evaluate calibration quality.\n* `config.py`: Define available datasets, model and tasks.\n* `defaults.py`: Define default config parameters for sequence classification and language modelling (**Note**: These might not be very good parameters).\n\n### Other features\n\n* **Weights & Biases integration**: You can track your experiments easily with weights & biases by passing a `wandb_run` argument to `model.fit()`!\n* **Easy fine-tuning via HuggingFace**: You can fine-tune arbitrary BERT models using their name from HuggingFace's `transformers`.\n\n### Contributing\n\nThis repository is by no means perfect nor complete. If you find any bugs, please report them using the issue template,\nand, if you also happen to provide a fix, create a pull request! A GitHub template is provided for that as well.\n\nYou would like to make a new addition to the repository? Follow the steps below:\n\n* **Adding a new model**: To add a new model, add a new module in the `models` directory. You will also need to implement\na corresponding `Model` and `Module` class, inheriting from the classes of the same name in `models/model.py` and implementing all \n required functions. `Model` is supposed to be an out-of-the-box solution that you can start experimenting right away, whil \n `Module` should only include the most basic model logic in order to be easy to integrate into other codebases and allow tinkering.\n\n* **Adding a new uncertainty metric**: To add a new uncertainty metric, add the function to `utils/metrics.py`. The function should take\nthe logits of a model and output an uncertainty score (the higher the score, the more uncertain the model). The function should output \n a batch_size x sequence_length matrix, with batch_size x 1 for sequence classification tasks. After finishing the implementation, you can \n add the metric to the `single_prediction_uncertainty_metrics` of the `models.model.Model` class and `multi_prediction_uncertainty_metrics` of `models.model.MultiPredictionMixin` (if applicable).\n\nYou would like to add something else? Create an issue or contact me at dennis {dot} ulmer {at} mailbox {dot} org!\n\n\n",
"bugtrack_url": null,
"license": "GPL",
"summary": "PyTorch Implementation of Models used for Uncertainty Estimation in Natural Language Processing.",
"version": "1.0.3",
"project_urls": {
"Homepage": "https://github.com/Kaleidophon/nlp-uncertainty-zoo"
},
"split_keywords": [
"machine learning",
"deep learning",
"nlp",
"uncertainty",
"uncertainty estimationpytorch"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "93bb3bdb0df0aa20209e276520ce4a401d8a588324b625cde6ff1dc1ddf519fe",
"md5": "0479203eb77e814e7a981fe1a4c5042b",
"sha256": "63825ac61c4e659db9eed56c74ffa5405e34dfeeb04473c0b38018ffeac052ba"
},
"downloads": -1,
"filename": "nlp_uncertainty_zoo-1.0.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0479203eb77e814e7a981fe1a4c5042b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.5.3",
"size": 76354,
"upload_time": "2023-05-05T13:18:41",
"upload_time_iso_8601": "2023-05-05T13:18:41.058470Z",
"url": "https://files.pythonhosted.org/packages/93/bb/3bdb0df0aa20209e276520ce4a401d8a588324b625cde6ff1dc1ddf519fe/nlp_uncertainty_zoo-1.0.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "10d40adbbb2c4d9383026ba1f073d90f8a521a87583d9b248aecd91ff88c278e",
"md5": "debabd7b2570e5107d21d710d0af372d",
"sha256": "372e6b6d324beffe9a68ca0eaacba9e5d70b2ee0c5de6463d194eb389ebd7a2e"
},
"downloads": -1,
"filename": "nlp-uncertainty-zoo-1.0.3.tar.gz",
"has_sig": false,
"md5_digest": "debabd7b2570e5107d21d710d0af372d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.5.3",
"size": 56264,
"upload_time": "2023-05-05T13:18:49",
"upload_time_iso_8601": "2023-05-05T13:18:49.698826Z",
"url": "https://files.pythonhosted.org/packages/10/d4/0adbbb2c4d9383026ba1f073d90f8a521a87583d9b248aecd91ff88c278e/nlp-uncertainty-zoo-1.0.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-05-05 13:18:49",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Kaleidophon",
"github_project": "nlp-uncertainty-zoo",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "nlp-uncertainty-zoo"
}
JSON
