bleuscore


Namebleuscore JSON
Version 0.1.3 PyPI version JSON
download
home_pageNone
SummaryA fast bleu score calculator
upload_time2024-05-27 03:28:36
maintainerNone
docs_urlNone
authorMathew Shen <datahonor@gmail.com>
requires_python>=3.8
licenseMIT
keywords nlp tokenizer bleu deeplearning
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # bleuscore

[![codecov](https://codecov.io/gh/shenxiangzhuang/bleuscore/graph/badge.svg?token=ckgU5oGbxf)](https://codecov.io/gh/shenxiangzhuang/bleuscore)
[![MIT licensed](https://img.shields.io/badge/license-MIT-blue.svg)](./LICENSE)
[![Crates.io](https://img.shields.io/crates/v/bleuscore)](https://crates.io/crates/bleuscore)
[![PyPI - Version](https://img.shields.io/pypi/v/bleuscore)](https://pypi.org/project/bleuscore/)
![docs.rs](https://img.shields.io/docsrs/bleuscore)


[`bleuscore`](https://github.com/shenxiangzhuang/bleuscore)
is a fast BLEU score calculator written in rust.

## Installation
The python package has been published to [pypi](https://pypi.org/project/bleuscore/),
so we can install it directly with many ways: 

- `pip`
    ```bash
    pip install bleuscore
    ```

- `poetry`
    ```bash
    poetry add bleuscore
    ```

- `uv`
    ```bash
    uv pip install bleuscore
    ```

## Quick Start
The usage is exactly same with [huggingface evaluate](https://huggingface.co/spaces/evaluate-metric/bleu):

```diff
- import evaluate
+ import bleuscore

predictions = ["hello there general kenobi", "foo bar foobar"]
references = [
    ["hello there general kenobi", "hello there !"],
    ["foo bar foobar"]
]

- bleu = evaluate.load("bleu")
- results = bleu.compute(predictions=predictions, references=references)
+ results = bleuscore.compute(predictions=predictions, references=references)

print(results)
# {'bleu': 1.0, 'precisions': [1.0, 1.0, 1.0, 1.0], 'brevity_penalty': 1.0, 
# 'length_ratio': 1.1666666666666667, 'translation_length': 7, 'reference_length': 6}

```

## Benchmark

**TLDR: We got more than 10x speedup when the corpus size beyond 100K**

<p align="center">
  <img src="./benchmark/bench.png" alt="Benchmark" width="400" height="300">
</p>

We use the demo data shown in quick start to do this simple benchmark.
You can check the [benchmark/simple](./benchmark/simple) for the benchmark source code.

- `rs_bleuscore`: bleuscore python library
- `local_hf_bleu`: huggingface evaluate bleu algorithm in **local**
- `sacre_bleu`: sacrebleu
  - Note that we got different result with sacrebleu in the simple demo data and all the rests have same result
- `hf_evaluate`: huggingface evaluate bleu algorithm with **evaluate** package


The `N` is used to enlarge the predictions/references size by simply duplication the demo data as shown before.
We can see that as `N` increase, the bleuscore gets better performance.
You can navigate [benchmark](./benchmark/README.md) for more benchmark details.



### N=100

```bash
hyhyperfine --warmup 5 --runs 10   \
  "python simple/rs_bleuscore.py 100" \
  "python simple/local_hf_bleu.py 100" \
  "python simple/sacre_bleu.py 100"   \
  "python simple/hf_evaluate.py 100"

Benchmark 1: python simple/rs_bleuscore.py 100
  Time (mean ± σ):      19.0 ms ±   2.6 ms    [User: 17.8 ms, System: 5.3 ms]
  Range (min … max):    14.8 ms …  23.2 ms    10 runs

Benchmark 2: python simple/local_hf_bleu.py 100
  Time (mean ± σ):      21.5 ms ±   2.2 ms    [User: 19.0 ms, System: 2.5 ms]
  Range (min … max):    16.8 ms …  24.1 ms    10 runs

Benchmark 3: python simple/sacre_bleu.py 100
  Time (mean ± σ):      45.9 ms ±   2.2 ms    [User: 38.7 ms, System: 7.1 ms]
  Range (min … max):    43.5 ms …  50.9 ms    10 runs

Benchmark 4: python simple/hf_evaluate.py 100
  Time (mean ± σ):      4.504 s ±  0.429 s    [User: 0.762 s, System: 0.823 s]
  Range (min … max):    4.163 s …  5.446 s    10 runs

Summary
  python simple/rs_bleuscore.py 100 ran
    1.13 ± 0.20 times faster than python simple/local_hf_bleu.py 100
    2.42 ± 0.35 times faster than python simple/sacre_bleu.py 100
  237.68 ± 39.88 times faster than python simple/hf_evaluate.py 100
```

### N = 1K ~ 1M

| Command                                  |       Mean [ms] | Min [ms] | Max [ms] |        Relative |
|:-----------------------------------------|----------------:|---------:|---------:|----------------:|
| `python simple/rs_bleuscore.py 1000`     |      20.3 ± 1.3 |     18.2 |     21.4 |            1.00 |
| `python simple/local_hf_bleu.py 1000`    |      45.8 ± 1.2 |     44.2 |     47.5 |     2.26 ± 0.16 |
| `python simple/rs_bleuscore.py 10000`    |      37.8 ± 1.5 |     35.9 |     39.5 |     1.87 ± 0.14 |
| `python simple/local_hf_bleu.py 10000`   |     295.0 ± 5.9 |    288.6 |    304.2 |    14.55 ± 0.98 |
| `python simple/rs_bleuscore.py 100000`   |     219.6 ± 3.3 |    215.3 |    224.0 |    10.83 ± 0.72 |
| `python simple/local_hf_bleu.py 100000`  |   2781.4 ± 42.2 |   2723.1 |   2833.0 |   137.13 ± 9.10 |
| `python simple/rs_bleuscore.py 1000000`  |   2048.8 ± 31.4 |   2013.2 |   2090.3 |   101.01 ± 6.71 |
| `python simple/local_hf_bleu.py 1000000` | 28285.3 ± 100.9 |  28182.1 |  28396.1 | 1394.51 ± 90.21 |





            

Raw data

            {
    "_id": null,
    "home_page": null,
    "name": "bleuscore",
    "maintainer": null,
    "docs_url": null,
    "requires_python": ">=3.8",
    "maintainer_email": null,
    "keywords": "NLP, Tokenizer, BLEU, DeepLearning",
    "author": "Mathew Shen <datahonor@gmail.com>",
    "author_email": "Mathew Shen <datahonor@gmail.com>",
    "download_url": "https://files.pythonhosted.org/packages/9f/4f/5213bfb3e7f5a5f383a2a49279fbee95ecbb573eb2a53b00b3018d4faf24/bleuscore-0.1.3.tar.gz",
    "platform": null,
    "description": "# bleuscore\n\n[![codecov](https://codecov.io/gh/shenxiangzhuang/bleuscore/graph/badge.svg?token=ckgU5oGbxf)](https://codecov.io/gh/shenxiangzhuang/bleuscore)\n[![MIT licensed](https://img.shields.io/badge/license-MIT-blue.svg)](./LICENSE)\n[![Crates.io](https://img.shields.io/crates/v/bleuscore)](https://crates.io/crates/bleuscore)\n[![PyPI - Version](https://img.shields.io/pypi/v/bleuscore)](https://pypi.org/project/bleuscore/)\n![docs.rs](https://img.shields.io/docsrs/bleuscore)\n\n\n[`bleuscore`](https://github.com/shenxiangzhuang/bleuscore)\nis a fast BLEU score calculator written in rust.\n\n## Installation\nThe python package has been published to [pypi](https://pypi.org/project/bleuscore/),\nso we can install it directly with many ways: \n\n- `pip`\n    ```bash\n    pip install bleuscore\n    ```\n\n- `poetry`\n    ```bash\n    poetry add bleuscore\n    ```\n\n- `uv`\n    ```bash\n    uv pip install bleuscore\n    ```\n\n## Quick Start\nThe usage is exactly same with [huggingface evaluate](https://huggingface.co/spaces/evaluate-metric/bleu):\n\n```diff\n- import evaluate\n+ import bleuscore\n\npredictions = [\"hello there general kenobi\", \"foo bar foobar\"]\nreferences = [\n    [\"hello there general kenobi\", \"hello there !\"],\n    [\"foo bar foobar\"]\n]\n\n- bleu = evaluate.load(\"bleu\")\n- results = bleu.compute(predictions=predictions, references=references)\n+ results = bleuscore.compute(predictions=predictions, references=references)\n\nprint(results)\n# {'bleu': 1.0, 'precisions': [1.0, 1.0, 1.0, 1.0], 'brevity_penalty': 1.0, \n# 'length_ratio': 1.1666666666666667, 'translation_length': 7, 'reference_length': 6}\n\n```\n\n## Benchmark\n\n**TLDR: We got more than 10x speedup when the corpus size beyond 100K**\n\n<p align=\"center\">\n  <img src=\"./benchmark/bench.png\" alt=\"Benchmark\" width=\"400\" height=\"300\">\n</p>\n\nWe use the demo data shown in quick start to do this simple benchmark.\nYou can check the [benchmark/simple](./benchmark/simple) for the benchmark source code.\n\n- `rs_bleuscore`: bleuscore python library\n- `local_hf_bleu`: huggingface evaluate bleu algorithm in **local**\n- `sacre_bleu`: sacrebleu\n  - Note that we got different result with sacrebleu in the simple demo data and all the rests have same result\n- `hf_evaluate`: huggingface evaluate bleu algorithm with **evaluate** package\n\n\nThe `N` is used to enlarge the predictions/references size by simply duplication the demo data as shown before.\nWe can see that as `N` increase, the bleuscore gets better performance.\nYou can navigate [benchmark](./benchmark/README.md) for more benchmark details.\n\n\n\n### N=100\n\n```bash\nhyhyperfine --warmup 5 --runs 10   \\\n  \"python simple/rs_bleuscore.py 100\" \\\n  \"python simple/local_hf_bleu.py 100\" \\\n  \"python simple/sacre_bleu.py 100\"   \\\n  \"python simple/hf_evaluate.py 100\"\n\nBenchmark 1: python simple/rs_bleuscore.py 100\n  Time (mean \u00b1 \u03c3):      19.0 ms \u00b1   2.6 ms    [User: 17.8 ms, System: 5.3 ms]\n  Range (min \u2026 max):    14.8 ms \u2026  23.2 ms    10 runs\n\nBenchmark 2: python simple/local_hf_bleu.py 100\n  Time (mean \u00b1 \u03c3):      21.5 ms \u00b1   2.2 ms    [User: 19.0 ms, System: 2.5 ms]\n  Range (min \u2026 max):    16.8 ms \u2026  24.1 ms    10 runs\n\nBenchmark 3: python simple/sacre_bleu.py 100\n  Time (mean \u00b1 \u03c3):      45.9 ms \u00b1   2.2 ms    [User: 38.7 ms, System: 7.1 ms]\n  Range (min \u2026 max):    43.5 ms \u2026  50.9 ms    10 runs\n\nBenchmark 4: python simple/hf_evaluate.py 100\n  Time (mean \u00b1 \u03c3):      4.504 s \u00b1  0.429 s    [User: 0.762 s, System: 0.823 s]\n  Range (min \u2026 max):    4.163 s \u2026  5.446 s    10 runs\n\nSummary\n  python simple/rs_bleuscore.py 100 ran\n    1.13 \u00b1 0.20 times faster than python simple/local_hf_bleu.py 100\n    2.42 \u00b1 0.35 times faster than python simple/sacre_bleu.py 100\n  237.68 \u00b1 39.88 times faster than python simple/hf_evaluate.py 100\n```\n\n### N = 1K ~ 1M\n\n| Command                                  |       Mean [ms] | Min [ms] | Max [ms] |        Relative |\n|:-----------------------------------------|----------------:|---------:|---------:|----------------:|\n| `python simple/rs_bleuscore.py 1000`     |      20.3 \u00b1 1.3 |     18.2 |     21.4 |            1.00 |\n| `python simple/local_hf_bleu.py 1000`    |      45.8 \u00b1 1.2 |     44.2 |     47.5 |     2.26 \u00b1 0.16 |\n| `python simple/rs_bleuscore.py 10000`    |      37.8 \u00b1 1.5 |     35.9 |     39.5 |     1.87 \u00b1 0.14 |\n| `python simple/local_hf_bleu.py 10000`   |     295.0 \u00b1 5.9 |    288.6 |    304.2 |    14.55 \u00b1 0.98 |\n| `python simple/rs_bleuscore.py 100000`   |     219.6 \u00b1 3.3 |    215.3 |    224.0 |    10.83 \u00b1 0.72 |\n| `python simple/local_hf_bleu.py 100000`  |   2781.4 \u00b1 42.2 |   2723.1 |   2833.0 |   137.13 \u00b1 9.10 |\n| `python simple/rs_bleuscore.py 1000000`  |   2048.8 \u00b1 31.4 |   2013.2 |   2090.3 |   101.01 \u00b1 6.71 |\n| `python simple/local_hf_bleu.py 1000000` | 28285.3 \u00b1 100.9 |  28182.1 |  28396.1 | 1394.51 \u00b1 90.21 |\n\n\n\n\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "A fast bleu score calculator",
    "version": "0.1.3",
    "project_urls": {
        "Homepage": "https://github.com/shenxiangzhuang/bleuscore",
        "Source": "https://github.com/shenxiangzhuang/bleuscore"
    },
    "split_keywords": [
        "nlp",
        " tokenizer",
        " bleu",
        " deeplearning"
    ],
    "urls": [
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "88054191cd3a8900e9e154b08c8e8263e5d7240e81ce712a95cfcd68cfb73809",
                "md5": "8c60518f976c37750d3f810d7ea9b501",
                "sha256": "d41d4010fc6ab2429cbde3c66dfec93d94d6936a3f7c79f3ab0055548c09e237"
            },
            "downloads": -1,
            "filename": "bleuscore-0.1.3-cp38-abi3-macosx_10_12_x86_64.whl",
            "has_sig": false,
            "md5_digest": "8c60518f976c37750d3f810d7ea9b501",
            "packagetype": "bdist_wheel",
            "python_version": "cp38",
            "requires_python": ">=3.8",
            "size": 894696,
            "upload_time": "2024-05-27T03:28:34",
            "upload_time_iso_8601": "2024-05-27T03:28:34.267485Z",
            "url": "https://files.pythonhosted.org/packages/88/05/4191cd3a8900e9e154b08c8e8263e5d7240e81ce712a95cfcd68cfb73809/bleuscore-0.1.3-cp38-abi3-macosx_10_12_x86_64.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "6585eab7e8983df1ca4810bb0f8aef0b03afcc81e2f4fe66234434db21810267",
                "md5": "f56f941678570952f2cedcbb11bf8ff2",
                "sha256": "91c1037ef7324dc6f80defa9aa49902b46a69bba874ac3d2623b1f95495f0be3"
            },
            "downloads": -1,
            "filename": "bleuscore-0.1.3-cp38-abi3-macosx_11_0_arm64.whl",
            "has_sig": false,
            "md5_digest": "f56f941678570952f2cedcbb11bf8ff2",
            "packagetype": "bdist_wheel",
            "python_version": "cp38",
            "requires_python": ">=3.8",
            "size": 846590,
            "upload_time": "2024-05-27T03:28:32",
            "upload_time_iso_8601": "2024-05-27T03:28:32.502225Z",
            "url": "https://files.pythonhosted.org/packages/65/85/eab7e8983df1ca4810bb0f8aef0b03afcc81e2f4fe66234434db21810267/bleuscore-0.1.3-cp38-abi3-macosx_11_0_arm64.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "bc6453e57f6504c1a48c2191aae7eca1dfbec7ebb728c5f86f98e10eda8cc42f",
                "md5": "5c8241cae3d474121ef22a78b7afc29e",
                "sha256": "4ad55f76434fe01e0569bec6c053573e370f13b1a8a246d6a0bb9738c728608f"
            },
            "downloads": -1,
            "filename": "bleuscore-0.1.3-cp38-abi3-manylinux_2_17_i686.manylinux2014_i686.whl",
            "has_sig": false,
            "md5_digest": "5c8241cae3d474121ef22a78b7afc29e",
            "packagetype": "bdist_wheel",
            "python_version": "cp38",
            "requires_python": ">=3.8",
            "size": 1003903,
            "upload_time": "2024-05-27T03:28:29",
            "upload_time_iso_8601": "2024-05-27T03:28:29.110210Z",
            "url": "https://files.pythonhosted.org/packages/bc/64/53e57f6504c1a48c2191aae7eca1dfbec7ebb728c5f86f98e10eda8cc42f/bleuscore-0.1.3-cp38-abi3-manylinux_2_17_i686.manylinux2014_i686.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "849f78628ea11dbd4b1cf4e90aedd738bcb1dd07cd14e281c9b8d7ced1832cff",
                "md5": "5f3ed9d595f75fa23725fdebfd501f03",
                "sha256": "a83796a6a6a7e1ce87dd4b4c48ef8dbdd24084704b8469def0eb6f4f4e87060f"
            },
            "downloads": -1,
            "filename": "bleuscore-0.1.3-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
            "has_sig": false,
            "md5_digest": "5f3ed9d595f75fa23725fdebfd501f03",
            "packagetype": "bdist_wheel",
            "python_version": "cp38",
            "requires_python": ">=3.8",
            "size": 1001445,
            "upload_time": "2024-05-27T03:28:31",
            "upload_time_iso_8601": "2024-05-27T03:28:31.252703Z",
            "url": "https://files.pythonhosted.org/packages/84/9f/78628ea11dbd4b1cf4e90aedd738bcb1dd07cd14e281c9b8d7ced1832cff/bleuscore-0.1.3-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "3aa79a0790c7e1e782b22b6ac46b5f27e65058c55d2a091e5d391bb8a52847fd",
                "md5": "edb0839406ac64ea88c3aa758dbabe32",
                "sha256": "4d012285e6be18ca4621b44f56d1ee9b9e6daa1e76baab22331b242d7c1a35aa"
            },
            "downloads": -1,
            "filename": "bleuscore-0.1.3-cp38-abi3-win_amd64.whl",
            "has_sig": false,
            "md5_digest": "edb0839406ac64ea88c3aa758dbabe32",
            "packagetype": "bdist_wheel",
            "python_version": "cp38",
            "requires_python": ">=3.8",
            "size": 734710,
            "upload_time": "2024-05-27T03:28:37",
            "upload_time_iso_8601": "2024-05-27T03:28:37.242089Z",
            "url": "https://files.pythonhosted.org/packages/3a/a7/9a0790c7e1e782b22b6ac46b5f27e65058c55d2a091e5d391bb8a52847fd/bleuscore-0.1.3-cp38-abi3-win_amd64.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "9f4f5213bfb3e7f5a5f383a2a49279fbee95ecbb573eb2a53b00b3018d4faf24",
                "md5": "7f7e47b3b67162cba01581a770960f15",
                "sha256": "d78531815b0b8f7c66adecbf8097a3759eab8915764b744a1ce41ad7a5821e5d"
            },
            "downloads": -1,
            "filename": "bleuscore-0.1.3.tar.gz",
            "has_sig": false,
            "md5_digest": "7f7e47b3b67162cba01581a770960f15",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.8",
            "size": 1081142,
            "upload_time": "2024-05-27T03:28:36",
            "upload_time_iso_8601": "2024-05-27T03:28:36.138167Z",
            "url": "https://files.pythonhosted.org/packages/9f/4f/5213bfb3e7f5a5f383a2a49279fbee95ecbb573eb2a53b00b3018d4faf24/bleuscore-0.1.3.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-05-27 03:28:36",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "shenxiangzhuang",
    "github_project": "bleuscore",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "lcname": "bleuscore"
}
        
Elapsed time: 0.35466s