[](https://github.com/asahi417/lmppl/blob/master/LICENSE.txt)
[](https://badge.fury.io/py/lmppl)
[](https://pypi.python.org/pypi/lmppl/)
[](https://pypi.python.org/pypi/lmppl/)
# Language Model Perplexity (LM-PPL)
Perplexity measures how predictable a text is by a language model (LM), and it is often used to evaluate fluency or proto-typicality of the text
(lower the perplexity is, more fluent or proto-typical the text is).
LM-PPL is a python library to calculate perplexity on a text with any types of pre-trained LMs.
We compute an ordinary perplexity for recurrent LMs such as [GPT3 (Brown et al., 2020)](https://arxiv.org/abs/2005.14165) and the perplexity of the decoder for encoder-decoder
LMs such as [BART (Lewis et al., 2020)](https://aclanthology.org/2020.acl-main.703/) or [T5 (Raffel et al., 2020)](https://arxiv.org/abs/1910.10683),
while we compute [pseudo-perplexity (Wang and Cho, 2018)](https://aclanthology.org/W19-2304/) for masked LMs.
## Get Started
Install via pip.
```shell
pip install lmppl
```
### Example
Let's solve sentiment analysis with perplexity as an example! Remember the text with lower perplexity is better, so we
compare two texts (positive and negative) and choose the one with lower perplexity as the model prediction.
1. ***Recurrent LM*** including variants of GPT.
```python3
import lmppl
scorer = lmppl.LM('gpt2')
text = [
'sentiment classification: I dropped my laptop on my knee, and someone stole my coffee. I am happy.',
'sentiment classification: I dropped my laptop on my knee, and someone stole my coffee. I am sad.'
]
ppl = scorer.get_perplexity(text)
print(list(zip(text, ppl)))
>>> [
('sentiment classification: I dropped my laptop on my knee, and someone stole my coffee. I am happy.', 136.64255272925908),
('sentiment classification: I dropped my laptop on my knee, and someone stole my coffee. I am sad.', 139.2400838400971)
]
print(f"prediction: {text[ppl.index(min(ppl))]}")
>>> "prediction: sentiment classification: I dropped my laptop on my knee, and someone stole my coffee. I am happy."
```
2. ***Masked LM*** including variants of BERT.
```python3
import lmppl
scorer = lmppl.MaskedLM('microsoft/deberta-v3-small')
text = [
'sentiment classification: I dropped my laptop on my knee, and someone stole my coffee. I am happy.',
'sentiment classification: I dropped my laptop on my knee, and someone stole my coffee. I am sad.'
]
ppl = scorer.get_perplexity(text)
print(list(zip(text, ppl)))
>>> [
('sentiment classification: I dropped my laptop on my knee, and someone stole my coffee. I am happy.', 1190212.1699246117),
('sentiment classification: I dropped my laptop on my knee, and someone stole my coffee. I am sad.', 1152767.482071837)
]
print(f"prediction: {text[ppl.index(min(ppl))]}")
>>> "prediction: sentiment classification: I dropped my laptop on my knee, and someone stole my coffee. I am sad."
```
3. ***Encoder-Decoder LM*** including variants of T5 and BART.
```python3
import lmppl
scorer = lmppl.EncoderDecoderLM('google/flan-t5-small')
inputs = [
'sentiment classification: I dropped my laptop on my knee, and someone stole my coffee.',
'sentiment classification: I dropped my laptop on my knee, and someone stole my coffee.'
]
outputs = [
'I am happy.',
'I am sad.'
]
ppl = scorer.get_perplexity(input_texts=inputs, output_texts=outputs)
print(list(zip(outputs, ppl)))
>>> [
('I am happy.', 4138.748977714201),
('I am sad.', 2991.629250051472)
]
print(f"prediction: {outputs[ppl.index(min(ppl))]}")
>>> "prediction: I am sad."
```
### Models
Below are some examples of popular models and the corresponding model type to use within the lmppl package.
| Model | HuggingFace ID | Model Type |
|-----------|----------------------------------|------------------|
| BERT | google-bert/bert-base-uncased | MaskedLM |
| Roberta | roberta-large | MaskedLM |
| GPT 2 | gpt2-xl | LM |
| flan-ul2 | google/flan-ul2 | EncoderDecoderLM |
| GPT-NeoX | EleutherAI/gpt-neox-20b | LM |
| OPT | facebook/opt-30b | LM |
| Mixtral | mistralai/Mixtral-8x22B-v0.1 | LM |
| Llama 3 | meta-llama/Meta-Llama-3-8B | LM |
### Tips
- **Max Token Length**: Each LM has its own max-token length (`max_length` for recurrent/masked LMs, and `max_length_encoder` and `max_length_decoder` for encoder-decoder LMs).
Limiting those max-token will reduce the time to process the text, but it may affect the accuracy of the perplexity, so please experiment on your texts and decide
an optimal token length.
- **Batch Size**: One can pass batch size to the function `get_perplexity` (eg. `get_perplexity(text, batch_size=32)`).
As default, it will process all the text once, that may cause memory error if the number of texts is too large.
Raw data
{
"_id": null,
"home_page": "https://github.com/asahi417/lmppl",
"name": "lmppl",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": null,
"keywords": "language model, t5, gpt3, bert, perplexity, nlp",
"author": "Asahi Ushio",
"author_email": "asahi1992ushio@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/02/ee/6d281f9c166846af1a91b13805b102a954e444f46cfc7bf175b96f8dd2cc/lmppl-0.3.2.tar.gz",
"platform": null,
"description": "[](https://github.com/asahi417/lmppl/blob/master/LICENSE.txt)\n[](https://badge.fury.io/py/lmppl)\n[](https://pypi.python.org/pypi/lmppl/)\n[](https://pypi.python.org/pypi/lmppl/)\n\n# Language Model Perplexity (LM-PPL) \nPerplexity measures how predictable a text is by a language model (LM), and it is often used to evaluate fluency or proto-typicality of the text\n(lower the perplexity is, more fluent or proto-typical the text is).\nLM-PPL is a python library to calculate perplexity on a text with any types of pre-trained LMs.\nWe compute an ordinary perplexity for recurrent LMs such as [GPT3 (Brown et al., 2020)](https://arxiv.org/abs/2005.14165) and the perplexity of the decoder for encoder-decoder \nLMs such as [BART (Lewis et al., 2020)](https://aclanthology.org/2020.acl-main.703/) or [T5 (Raffel et al., 2020)](https://arxiv.org/abs/1910.10683), \nwhile we compute [pseudo-perplexity (Wang and Cho, 2018)](https://aclanthology.org/W19-2304/) for masked LMs. \n\n\n## Get Started\nInstall via pip.\n```shell\npip install lmppl\n```\n\n### Example \nLet's solve sentiment analysis with perplexity as an example! Remember the text with lower perplexity is better, so we \ncompare two texts (positive and negative) and choose the one with lower perplexity as the model prediction.\n\n\n1. ***Recurrent LM*** including variants of GPT.\n```python3\nimport lmppl\n\nscorer = lmppl.LM('gpt2')\ntext = [\n 'sentiment classification: I dropped my laptop on my knee, and someone stole my coffee. I am happy.',\n 'sentiment classification: I dropped my laptop on my knee, and someone stole my coffee. I am sad.'\n]\nppl = scorer.get_perplexity(text)\nprint(list(zip(text, ppl)))\n>>> [\n ('sentiment classification: I dropped my laptop on my knee, and someone stole my coffee. I am happy.', 136.64255272925908),\n ('sentiment classification: I dropped my laptop on my knee, and someone stole my coffee. I am sad.', 139.2400838400971)\n]\nprint(f\"prediction: {text[ppl.index(min(ppl))]}\")\n>>> \"prediction: sentiment classification: I dropped my laptop on my knee, and someone stole my coffee. I am happy.\"\n```\n\n2. ***Masked LM*** including variants of BERT.\n```python3\nimport lmppl\n\nscorer = lmppl.MaskedLM('microsoft/deberta-v3-small')\ntext = [\n 'sentiment classification: I dropped my laptop on my knee, and someone stole my coffee. I am happy.',\n 'sentiment classification: I dropped my laptop on my knee, and someone stole my coffee. I am sad.'\n]\nppl = scorer.get_perplexity(text)\nprint(list(zip(text, ppl)))\n>>> [\n ('sentiment classification: I dropped my laptop on my knee, and someone stole my coffee. I am happy.', 1190212.1699246117),\n ('sentiment classification: I dropped my laptop on my knee, and someone stole my coffee. I am sad.', 1152767.482071837)\n]\nprint(f\"prediction: {text[ppl.index(min(ppl))]}\")\n>>> \"prediction: sentiment classification: I dropped my laptop on my knee, and someone stole my coffee. I am sad.\"\n```\n\n\n3. ***Encoder-Decoder LM*** including variants of T5 and BART.\n```python3\nimport lmppl\n\nscorer = lmppl.EncoderDecoderLM('google/flan-t5-small')\ninputs = [\n 'sentiment classification: I dropped my laptop on my knee, and someone stole my coffee.',\n 'sentiment classification: I dropped my laptop on my knee, and someone stole my coffee.'\n]\noutputs = [\n 'I am happy.',\n 'I am sad.'\n]\nppl = scorer.get_perplexity(input_texts=inputs, output_texts=outputs)\nprint(list(zip(outputs, ppl)))\n>>> [\n ('I am happy.', 4138.748977714201),\n ('I am sad.', 2991.629250051472)\n]\nprint(f\"prediction: {outputs[ppl.index(min(ppl))]}\")\n>>> \"prediction: I am sad.\"\n```\n\n### Models\nBelow are some examples of popular models and the corresponding model type to use within the lmppl package.\n\n| Model | HuggingFace ID | Model Type |\n|-----------|----------------------------------|------------------|\n| BERT | google-bert/bert-base-uncased | MaskedLM |\n| Roberta | roberta-large | MaskedLM |\n| GPT 2 | gpt2-xl | LM |\n| flan-ul2 | google/flan-ul2 | EncoderDecoderLM |\n| GPT-NeoX | EleutherAI/gpt-neox-20b | LM |\n| OPT | facebook/opt-30b | LM |\n| Mixtral | mistralai/Mixtral-8x22B-v0.1 | LM |\n| Llama 3 | meta-llama/Meta-Llama-3-8B | LM |\n\n### Tips\n- **Max Token Length**: Each LM has its own max-token length (`max_length` for recurrent/masked LMs, and `max_length_encoder` and `max_length_decoder` for encoder-decoder LMs).\nLimiting those max-token will reduce the time to process the text, but it may affect the accuracy of the perplexity, so please experiment on your texts and decide\nan optimal token length.\n \n- **Batch Size**: One can pass batch size to the function `get_perplexity` (eg. `get_perplexity(text, batch_size=32)`).\nAs default, it will process all the text once, that may cause memory error if the number of texts is too large.\n\n\n\n",
"bugtrack_url": null,
"license": "MIT License",
"summary": "Calculate perplexity on the text with pre-trained language models.",
"version": "0.3.2",
"project_urls": {
"Download": "https://github.com/asahi417/lmppl/archive/v0.3.2.tar.gz",
"Homepage": "https://github.com/asahi417/lmppl"
},
"split_keywords": [
"language model",
" t5",
" gpt3",
" bert",
" perplexity",
" nlp"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "02ee6d281f9c166846af1a91b13805b102a954e444f46cfc7bf175b96f8dd2cc",
"md5": "eed78e25fac53a66db9e828643751733",
"sha256": "1c9d968211e0650aee179bc56c185fbc505c4f168eda1a557ba0239fdd397730"
},
"downloads": -1,
"filename": "lmppl-0.3.2.tar.gz",
"has_sig": false,
"md5_digest": "eed78e25fac53a66db9e828643751733",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 12107,
"upload_time": "2024-08-20T14:45:07",
"upload_time_iso_8601": "2024-08-20T14:45:07.019562Z",
"url": "https://files.pythonhosted.org/packages/02/ee/6d281f9c166846af1a91b13805b102a954e444f46cfc7bf175b96f8dd2cc/lmppl-0.3.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-20 14:45:07",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "asahi417",
"github_project": "lmppl",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "lmppl"
}
JSON
