# bleuscore
[](https://codecov.io/gh/shenxiangzhuang/bleuscore)
[](./LICENSE)
[](https://crates.io/crates/bleuscore)
[](https://pypi.org/project/bleuscore/)
[`bleuscore`](https://github.com/shenxiangzhuang/bleuscore)
is a fast BLEU score calculator written in rust.
## Installation
The python package has been published to [pypi](https://pypi.org/project/bleuscore/),
so we can install it directly with many ways:
- `pip`
```bash
pip install bleuscore
```
- `poetry`
```bash
poetry add bleuscore
```
- `uv`
```bash
uv pip install bleuscore
```
## Quick Start
The usage is exactly same with [huggingface evaluate](https://huggingface.co/spaces/evaluate-metric/bleu):
```diff
- import evaluate
+ import bleuscore
predictions = ["hello there general kenobi", "foo bar foobar"]
references = [
["hello there general kenobi", "hello there !"],
["foo bar foobar"]
]
- bleu = evaluate.load("bleu")
- results = bleu.compute(predictions=predictions, references=references)
+ results = bleuscore.compute(predictions=predictions, references=references)
print(results)
# {'bleu': 1.0, 'precisions': [1.0, 1.0, 1.0, 1.0], 'brevity_penalty': 1.0,
# 'length_ratio': 1.1666666666666667, 'translation_length': 7, 'reference_length': 6}
```
## Benchmark
**TLDR: We got more than 10x speedup when the corpus size beyond 100K**
<p align="center">
<img src="./benchmark/bench.png" alt="Benchmark" width="400" height="300">
</p>
We use the demo data shown in quick start to do this simple benchmark.
You can check the [benchmark/simple](./benchmark/simple) for the benchmark source code.
- `rs_bleuscore`: bleuscore python library
- `local_hf_bleu`: huggingface evaluate bleu algorithm in **local**
- `sacre_bleu`: sacrebleu
- Note that we got different result with sacrebleu in the simple demo data and all the rests have same result
- `hf_evaluate`: huggingface evaluate bleu algorithm with **evaluate** package
The `N` is used to enlarge the predictions/references size by simply duplication the demo data as shown before.
We can see that as `N` increase, the bleuscore gets better performance.
You can navigate [benchmark](./benchmark/README.md) for more benchmark details.
### N=100
```bash
hyhyperfine --warmup 5 --runs 10 \
"python simple/rs_bleuscore.py 100" \
"python simple/local_hf_bleu.py 100" \
"python simple/sacre_bleu.py 100" \
"python simple/hf_evaluate.py 100"
Benchmark 1: python simple/rs_bleuscore.py 100
Time (mean ± σ): 19.0 ms ± 2.6 ms [User: 17.8 ms, System: 5.3 ms]
Range (min … max): 14.8 ms … 23.2 ms 10 runs
Benchmark 2: python simple/local_hf_bleu.py 100
Time (mean ± σ): 21.5 ms ± 2.2 ms [User: 19.0 ms, System: 2.5 ms]
Range (min … max): 16.8 ms … 24.1 ms 10 runs
Benchmark 3: python simple/sacre_bleu.py 100
Time (mean ± σ): 45.9 ms ± 2.2 ms [User: 38.7 ms, System: 7.1 ms]
Range (min … max): 43.5 ms … 50.9 ms 10 runs
Benchmark 4: python simple/hf_evaluate.py 100
Time (mean ± σ): 4.504 s ± 0.429 s [User: 0.762 s, System: 0.823 s]
Range (min … max): 4.163 s … 5.446 s 10 runs
Summary
python simple/rs_bleuscore.py 100 ran
1.13 ± 0.20 times faster than python simple/local_hf_bleu.py 100
2.42 ± 0.35 times faster than python simple/sacre_bleu.py 100
237.68 ± 39.88 times faster than python simple/hf_evaluate.py 100
```
### N = 1K ~ 1M
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|:-----------------------------------------|----------------:|---------:|---------:|----------------:|
| `python simple/rs_bleuscore.py 1000` | 20.3 ± 1.3 | 18.2 | 21.4 | 1.00 |
| `python simple/local_hf_bleu.py 1000` | 45.8 ± 1.2 | 44.2 | 47.5 | 2.26 ± 0.16 |
| `python simple/rs_bleuscore.py 10000` | 37.8 ± 1.5 | 35.9 | 39.5 | 1.87 ± 0.14 |
| `python simple/local_hf_bleu.py 10000` | 295.0 ± 5.9 | 288.6 | 304.2 | 14.55 ± 0.98 |
| `python simple/rs_bleuscore.py 100000` | 219.6 ± 3.3 | 215.3 | 224.0 | 10.83 ± 0.72 |
| `python simple/local_hf_bleu.py 100000` | 2781.4 ± 42.2 | 2723.1 | 2833.0 | 137.13 ± 9.10 |
| `python simple/rs_bleuscore.py 1000000` | 2048.8 ± 31.4 | 2013.2 | 2090.3 | 101.01 ± 6.71 |
| `python simple/local_hf_bleu.py 1000000` | 28285.3 ± 100.9 | 28182.1 | 28396.1 | 1394.51 ± 90.21 |
Raw data
{
"_id": null,
"home_page": null,
"name": "bleuscore",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "NLP, Tokenizer, BLEU, DeepLearning",
"author": "Mathew Shen <datahonor@gmail.com>",
"author_email": "Mathew Shen <datahonor@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/9f/4f/5213bfb3e7f5a5f383a2a49279fbee95ecbb573eb2a53b00b3018d4faf24/bleuscore-0.1.3.tar.gz",
"platform": null,
"description": "# bleuscore\n\n[](https://codecov.io/gh/shenxiangzhuang/bleuscore)\n[](./LICENSE)\n[](https://crates.io/crates/bleuscore)\n[](https://pypi.org/project/bleuscore/)\n\n\n\n[`bleuscore`](https://github.com/shenxiangzhuang/bleuscore)\nis a fast BLEU score calculator written in rust.\n\n## Installation\nThe python package has been published to [pypi](https://pypi.org/project/bleuscore/),\nso we can install it directly with many ways: \n\n- `pip`\n ```bash\n pip install bleuscore\n ```\n\n- `poetry`\n ```bash\n poetry add bleuscore\n ```\n\n- `uv`\n ```bash\n uv pip install bleuscore\n ```\n\n## Quick Start\nThe usage is exactly same with [huggingface evaluate](https://huggingface.co/spaces/evaluate-metric/bleu):\n\n```diff\n- import evaluate\n+ import bleuscore\n\npredictions = [\"hello there general kenobi\", \"foo bar foobar\"]\nreferences = [\n [\"hello there general kenobi\", \"hello there !\"],\n [\"foo bar foobar\"]\n]\n\n- bleu = evaluate.load(\"bleu\")\n- results = bleu.compute(predictions=predictions, references=references)\n+ results = bleuscore.compute(predictions=predictions, references=references)\n\nprint(results)\n# {'bleu': 1.0, 'precisions': [1.0, 1.0, 1.0, 1.0], 'brevity_penalty': 1.0, \n# 'length_ratio': 1.1666666666666667, 'translation_length': 7, 'reference_length': 6}\n\n```\n\n## Benchmark\n\n**TLDR: We got more than 10x speedup when the corpus size beyond 100K**\n\n<p align=\"center\">\n <img src=\"./benchmark/bench.png\" alt=\"Benchmark\" width=\"400\" height=\"300\">\n</p>\n\nWe use the demo data shown in quick start to do this simple benchmark.\nYou can check the [benchmark/simple](./benchmark/simple) for the benchmark source code.\n\n- `rs_bleuscore`: bleuscore python library\n- `local_hf_bleu`: huggingface evaluate bleu algorithm in **local**\n- `sacre_bleu`: sacrebleu\n - Note that we got different result with sacrebleu in the simple demo data and all the rests have same result\n- `hf_evaluate`: huggingface evaluate bleu algorithm with **evaluate** package\n\n\nThe `N` is used to enlarge the predictions/references size by simply duplication the demo data as shown before.\nWe can see that as `N` increase, the bleuscore gets better performance.\nYou can navigate [benchmark](./benchmark/README.md) for more benchmark details.\n\n\n\n### N=100\n\n```bash\nhyhyperfine --warmup 5 --runs 10 \\\n \"python simple/rs_bleuscore.py 100\" \\\n \"python simple/local_hf_bleu.py 100\" \\\n \"python simple/sacre_bleu.py 100\" \\\n \"python simple/hf_evaluate.py 100\"\n\nBenchmark 1: python simple/rs_bleuscore.py 100\n Time (mean \u00b1 \u03c3): 19.0 ms \u00b1 2.6 ms [User: 17.8 ms, System: 5.3 ms]\n Range (min \u2026 max): 14.8 ms \u2026 23.2 ms 10 runs\n\nBenchmark 2: python simple/local_hf_bleu.py 100\n Time (mean \u00b1 \u03c3): 21.5 ms \u00b1 2.2 ms [User: 19.0 ms, System: 2.5 ms]\n Range (min \u2026 max): 16.8 ms \u2026 24.1 ms 10 runs\n\nBenchmark 3: python simple/sacre_bleu.py 100\n Time (mean \u00b1 \u03c3): 45.9 ms \u00b1 2.2 ms [User: 38.7 ms, System: 7.1 ms]\n Range (min \u2026 max): 43.5 ms \u2026 50.9 ms 10 runs\n\nBenchmark 4: python simple/hf_evaluate.py 100\n Time (mean \u00b1 \u03c3): 4.504 s \u00b1 0.429 s [User: 0.762 s, System: 0.823 s]\n Range (min \u2026 max): 4.163 s \u2026 5.446 s 10 runs\n\nSummary\n python simple/rs_bleuscore.py 100 ran\n 1.13 \u00b1 0.20 times faster than python simple/local_hf_bleu.py 100\n 2.42 \u00b1 0.35 times faster than python simple/sacre_bleu.py 100\n 237.68 \u00b1 39.88 times faster than python simple/hf_evaluate.py 100\n```\n\n### N = 1K ~ 1M\n\n| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |\n|:-----------------------------------------|----------------:|---------:|---------:|----------------:|\n| `python simple/rs_bleuscore.py 1000` | 20.3 \u00b1 1.3 | 18.2 | 21.4 | 1.00 |\n| `python simple/local_hf_bleu.py 1000` | 45.8 \u00b1 1.2 | 44.2 | 47.5 | 2.26 \u00b1 0.16 |\n| `python simple/rs_bleuscore.py 10000` | 37.8 \u00b1 1.5 | 35.9 | 39.5 | 1.87 \u00b1 0.14 |\n| `python simple/local_hf_bleu.py 10000` | 295.0 \u00b1 5.9 | 288.6 | 304.2 | 14.55 \u00b1 0.98 |\n| `python simple/rs_bleuscore.py 100000` | 219.6 \u00b1 3.3 | 215.3 | 224.0 | 10.83 \u00b1 0.72 |\n| `python simple/local_hf_bleu.py 100000` | 2781.4 \u00b1 42.2 | 2723.1 | 2833.0 | 137.13 \u00b1 9.10 |\n| `python simple/rs_bleuscore.py 1000000` | 2048.8 \u00b1 31.4 | 2013.2 | 2090.3 | 101.01 \u00b1 6.71 |\n| `python simple/local_hf_bleu.py 1000000` | 28285.3 \u00b1 100.9 | 28182.1 | 28396.1 | 1394.51 \u00b1 90.21 |\n\n\n\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A fast bleu score calculator",
"version": "0.1.3",
"project_urls": {
"Homepage": "https://github.com/shenxiangzhuang/bleuscore",
"Source": "https://github.com/shenxiangzhuang/bleuscore"
},
"split_keywords": [
"nlp",
" tokenizer",
" bleu",
" deeplearning"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "88054191cd3a8900e9e154b08c8e8263e5d7240e81ce712a95cfcd68cfb73809",
"md5": "8c60518f976c37750d3f810d7ea9b501",
"sha256": "d41d4010fc6ab2429cbde3c66dfec93d94d6936a3f7c79f3ab0055548c09e237"
},
"downloads": -1,
"filename": "bleuscore-0.1.3-cp38-abi3-macosx_10_12_x86_64.whl",
"has_sig": false,
"md5_digest": "8c60518f976c37750d3f810d7ea9b501",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.8",
"size": 894696,
"upload_time": "2024-05-27T03:28:34",
"upload_time_iso_8601": "2024-05-27T03:28:34.267485Z",
"url": "https://files.pythonhosted.org/packages/88/05/4191cd3a8900e9e154b08c8e8263e5d7240e81ce712a95cfcd68cfb73809/bleuscore-0.1.3-cp38-abi3-macosx_10_12_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "6585eab7e8983df1ca4810bb0f8aef0b03afcc81e2f4fe66234434db21810267",
"md5": "f56f941678570952f2cedcbb11bf8ff2",
"sha256": "91c1037ef7324dc6f80defa9aa49902b46a69bba874ac3d2623b1f95495f0be3"
},
"downloads": -1,
"filename": "bleuscore-0.1.3-cp38-abi3-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "f56f941678570952f2cedcbb11bf8ff2",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.8",
"size": 846590,
"upload_time": "2024-05-27T03:28:32",
"upload_time_iso_8601": "2024-05-27T03:28:32.502225Z",
"url": "https://files.pythonhosted.org/packages/65/85/eab7e8983df1ca4810bb0f8aef0b03afcc81e2f4fe66234434db21810267/bleuscore-0.1.3-cp38-abi3-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "bc6453e57f6504c1a48c2191aae7eca1dfbec7ebb728c5f86f98e10eda8cc42f",
"md5": "5c8241cae3d474121ef22a78b7afc29e",
"sha256": "4ad55f76434fe01e0569bec6c053573e370f13b1a8a246d6a0bb9738c728608f"
},
"downloads": -1,
"filename": "bleuscore-0.1.3-cp38-abi3-manylinux_2_17_i686.manylinux2014_i686.whl",
"has_sig": false,
"md5_digest": "5c8241cae3d474121ef22a78b7afc29e",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.8",
"size": 1003903,
"upload_time": "2024-05-27T03:28:29",
"upload_time_iso_8601": "2024-05-27T03:28:29.110210Z",
"url": "https://files.pythonhosted.org/packages/bc/64/53e57f6504c1a48c2191aae7eca1dfbec7ebb728c5f86f98e10eda8cc42f/bleuscore-0.1.3-cp38-abi3-manylinux_2_17_i686.manylinux2014_i686.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "849f78628ea11dbd4b1cf4e90aedd738bcb1dd07cd14e281c9b8d7ced1832cff",
"md5": "5f3ed9d595f75fa23725fdebfd501f03",
"sha256": "a83796a6a6a7e1ce87dd4b4c48ef8dbdd24084704b8469def0eb6f4f4e87060f"
},
"downloads": -1,
"filename": "bleuscore-0.1.3-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "5f3ed9d595f75fa23725fdebfd501f03",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.8",
"size": 1001445,
"upload_time": "2024-05-27T03:28:31",
"upload_time_iso_8601": "2024-05-27T03:28:31.252703Z",
"url": "https://files.pythonhosted.org/packages/84/9f/78628ea11dbd4b1cf4e90aedd738bcb1dd07cd14e281c9b8d7ced1832cff/bleuscore-0.1.3-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "3aa79a0790c7e1e782b22b6ac46b5f27e65058c55d2a091e5d391bb8a52847fd",
"md5": "edb0839406ac64ea88c3aa758dbabe32",
"sha256": "4d012285e6be18ca4621b44f56d1ee9b9e6daa1e76baab22331b242d7c1a35aa"
},
"downloads": -1,
"filename": "bleuscore-0.1.3-cp38-abi3-win_amd64.whl",
"has_sig": false,
"md5_digest": "edb0839406ac64ea88c3aa758dbabe32",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.8",
"size": 734710,
"upload_time": "2024-05-27T03:28:37",
"upload_time_iso_8601": "2024-05-27T03:28:37.242089Z",
"url": "https://files.pythonhosted.org/packages/3a/a7/9a0790c7e1e782b22b6ac46b5f27e65058c55d2a091e5d391bb8a52847fd/bleuscore-0.1.3-cp38-abi3-win_amd64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "9f4f5213bfb3e7f5a5f383a2a49279fbee95ecbb573eb2a53b00b3018d4faf24",
"md5": "7f7e47b3b67162cba01581a770960f15",
"sha256": "d78531815b0b8f7c66adecbf8097a3759eab8915764b744a1ce41ad7a5821e5d"
},
"downloads": -1,
"filename": "bleuscore-0.1.3.tar.gz",
"has_sig": false,
"md5_digest": "7f7e47b3b67162cba01581a770960f15",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 1081142,
"upload_time": "2024-05-27T03:28:36",
"upload_time_iso_8601": "2024-05-27T03:28:36.138167Z",
"url": "https://files.pythonhosted.org/packages/9f/4f/5213bfb3e7f5a5f383a2a49279fbee95ecbb573eb2a53b00b3018d4faf24/bleuscore-0.1.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-27 03:28:36",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "shenxiangzhuang",
"github_project": "bleuscore",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "bleuscore"
}
JSON
