# MAUVE
This is a library built on PyTorch and HuggingFace Transformers to measure the gap between neural text and human text
with the MAUVE measure,
introduced in [this NeurIPS 2021 paper](https://arxiv.org/pdf/2102.01454.pdf) (Outstanding Paper Award) and [this JMLR 2023 paper](https://arxiv.org/pdf/2212.14578.pdf).
MAUVE is a measure of the gap between neural text and human text. It is computed using the Kullback–Leibler (KL) divergences between the two text distributions in a quantized embedding space of a large language model. MAUVE can identify differences in quality arising from model sizes and decoding algorithms.
### [Documentation Link](https://krishnap25.github.io/mauve/) [GitHub Link](https://github.com/krishnap25/mauve/)
### _MAUVE is also available via [HuggingFace Evaluate](https://huggingface.co/spaces/evaluate-metric/mauve)!_
**Features**:
- MAUVE with quantization using *k*-means.
- Adaptive selection of *k*-means hyperparameters.
- Compute MAUVE using pre-computed GPT-2 features (i.e., terminal hidden state),
or featurize raw text using HuggingFace transformers + PyTorch.
- Use with other modalities (e.g. images or audio) by directly passing in pre-computed feature embeddings to our API.
More details can be found in the [documentation](https://krishnap25.github.io/mauve).
## Installation
For a direct install, run this command from your terminal:
```
pip install mauve-text
```
## Citation
If you find this package useful, or you use it in your research, please cite the following papers:
```
@article{pillutla-etal:mauve:jmlr2023,
title={{MAUVE Scores for Generative Models: Theory and Practice}},
author={Pillutla, Krishna and Liu, Lang and Thickstun, John and Welleck, Sean and Swayamdipta, Swabha and Zellers, Rowan and Oh, Sewoong and Choi, Yejin and Harchaoui, Zaid},
journal={JMLR},
year={2023}
}
@inproceedings{pillutla-etal:mauve:neurips2021,
title={MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers},
author={Pillutla, Krishna and Swayamdipta, Swabha and Zellers, Rowan and Thickstun, John and Welleck, Sean and Choi, Yejin and Harchaoui, Zaid},
booktitle = {NeurIPS},
year = {2021}
}
@inproceedings{liu-etal:mauve-theory:neurips2021,
title={{Divergence Frontiers for Generative Models: Sample Complexity, Quantization Effects, and Frontier Integrals}},
author={Liu, Lang and Pillutla, Krishna and Welleck, Sean and Oh, Sewoong and Choi, Yejin and Harchaoui, Zaid},
booktitle={NeurIPS},
year={2021}
}
```
## Acknowledgements
This work was supported by NSF DMS-2134012, NSF CCF-2019844, NSF DMS-2023166, the DARPA MCS program through NIWC Pacific (N66001-19-2-4031), the CIFAR "Learning in Machines & Brains" program, a Qualcomm Innovation Fellowship, and faculty research awards.
Raw data
{
"_id": null,
"home_page": "https://github.com/krishnap25/mauve",
"name": "mauve-text",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": null,
"keywords": null,
"author": "Krishna Pillutla",
"author_email": "pillutla@cs.washington.edu",
"download_url": "https://files.pythonhosted.org/packages/53/f1/790ce5858951689e17cf2d767a951fdd40dd22b33b8ae01aecee182d2ad2/mauve-text-0.4.0.tar.gz",
"platform": null,
"description": "# MAUVE\n\nThis is a library built on PyTorch and HuggingFace Transformers to measure the gap between neural text and human text\nwith the MAUVE measure, \nintroduced in [this NeurIPS 2021 paper](https://arxiv.org/pdf/2102.01454.pdf) (Outstanding Paper Award) and [this JMLR 2023 paper](https://arxiv.org/pdf/2212.14578.pdf).\n\n\nMAUVE is a measure of the gap between neural text and human text. It is computed using the Kullback\u2013Leibler (KL) divergences between the two text distributions in a quantized embedding space of a large language model. MAUVE can identify differences in quality arising from model sizes and decoding algorithms.\n\n### [Documentation Link](https://krishnap25.github.io/mauve/) [GitHub Link](https://github.com/krishnap25/mauve/)\n\n### _MAUVE is also available via [HuggingFace Evaluate](https://huggingface.co/spaces/evaluate-metric/mauve)!_\n\n\n**Features**:\n- MAUVE with quantization using *k*-means. \n- Adaptive selection of *k*-means hyperparameters. \n- Compute MAUVE using pre-computed GPT-2 features (i.e., terminal hidden state), \n or featurize raw text using HuggingFace transformers + PyTorch.\n- Use with other modalities (e.g. images or audio) by directly passing in pre-computed feature embeddings to our API.\n\nMore details can be found in the [documentation](https://krishnap25.github.io/mauve).\n\n## Installation\n\nFor a direct install, run this command from your terminal:\n```\npip install mauve-text\n``` \n\n## Citation\nIf you find this package useful, or you use it in your research, please cite the following papers:\n```\n@article{pillutla-etal:mauve:jmlr2023,\n title={{MAUVE Scores for Generative Models: Theory and Practice}},\n author={Pillutla, Krishna and Liu, Lang and Thickstun, John and Welleck, Sean and Swayamdipta, Swabha and Zellers, Rowan and Oh, Sewoong and Choi, Yejin and Harchaoui, Zaid},\n journal={JMLR},\n year={2023}\n}\n\n@inproceedings{pillutla-etal:mauve:neurips2021,\n title={MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers},\n author={Pillutla, Krishna and Swayamdipta, Swabha and Zellers, Rowan and Thickstun, John and Welleck, Sean and Choi, Yejin and Harchaoui, Zaid},\n booktitle = {NeurIPS},\n year = {2021}\n}\n\n@inproceedings{liu-etal:mauve-theory:neurips2021,\n title={{Divergence Frontiers for Generative Models: Sample Complexity, Quantization Effects, and Frontier Integrals}},\n author={Liu, Lang and Pillutla, Krishna and Welleck, Sean and Oh, Sewoong and Choi, Yejin and Harchaoui, Zaid},\n booktitle={NeurIPS},\n year={2021}\n}\n\n```\n \n## Acknowledgements\nThis work was supported by NSF DMS-2134012, NSF CCF-2019844, NSF DMS-2023166, the DARPA MCS program through NIWC Pacific (N66001-19-2-4031), the CIFAR \"Learning in Machines & Brains\" program, a Qualcomm Innovation Fellowship, and faculty research awards.\n\n\n",
"bugtrack_url": null,
"license": null,
"summary": "Implementation of the MAUVE to evaluate text generation",
"version": "0.4.0",
"project_urls": {
"Bug Tracker": "https://github.com/krishnap25/mauve/issues",
"Homepage": "https://github.com/krishnap25/mauve"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "1f0a7fa7d797479b762e800b57064c5fe861743fe12722292c36de220fc964a2",
"md5": "d812928b4ac853f0e38496881d1e504d",
"sha256": "ceafe978fd66adf4a2a8bad8c47be417e7306f43a4a4ab9121de01f81fc7e47b"
},
"downloads": -1,
"filename": "mauve_text-0.4.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d812928b4ac853f0e38496881d1e504d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 21534,
"upload_time": "2024-07-03T11:24:20",
"upload_time_iso_8601": "2024-07-03T11:24:20.451223Z",
"url": "https://files.pythonhosted.org/packages/1f/0a/7fa7d797479b762e800b57064c5fe861743fe12722292c36de220fc964a2/mauve_text-0.4.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "53f1790ce5858951689e17cf2d767a951fdd40dd22b33b8ae01aecee182d2ad2",
"md5": "359d0785b105aef681df2a278ca32516",
"sha256": "a9cd29587d1acdfeb006274839c44ac65aec378fb89cceb368094f4a264fd94f"
},
"downloads": -1,
"filename": "mauve-text-0.4.0.tar.gz",
"has_sig": false,
"md5_digest": "359d0785b105aef681df2a278ca32516",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 22281,
"upload_time": "2024-07-03T11:24:21",
"upload_time_iso_8601": "2024-07-03T11:24:21.650284Z",
"url": "https://files.pythonhosted.org/packages/53/f1/790ce5858951689e17cf2d767a951fdd40dd22b33b8ae01aecee182d2ad2/mauve-text-0.4.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-07-03 11:24:21",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "krishnap25",
"github_project": "mauve",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "mauve-text"
}
JSON
