# SacreCOMET [](https://pypi.org/project/sacrecomet/) [](https://github.com/PinzhenChen/sacreCOMET/actions/workflows/test.yml)
> Since its introduction, the COMET metric has blazed a trail in the machine translation community, given its strong correlation with human judgements of translation quality. Its success stems from being a modified pre-trained multilingual model finetuned for quality assessment. However, it being a machine learning model also gives rise to a new set of pitfalls that may not be widely known. We investigate these unexpected behaviours from three aspects: 1) technical: obsolete software versions and compute precision; 2) data: empty content, language mismatch, and translationese at test time as well as distribution and domain biases in training; 3) usage and reporting: multi-reference support and model referencing in the literature. All of these problems imply that COMET scores is not comparable between papers or even technical setups and we put forward our perspective on fixing each issue. Furthermore, we release the SacreCOMET package that can generate a signature for the software and model configuration as well as an appropriate citation. The goal of this work is to help the community make more sound use of the COMET metric.
Read the full paper [Pitfalls and Outlooks in Using COMET](https://aclanthology.org/2024.wmt-1.121/).
## Tool
The Python tool has two functionalities.
First, it creates a signature with your setup and COMET model:
```
pip install sacrecomet
# Without anything will try to detect the local environment and will
# ask you questions about which COMET model you used.
# Example output: Python3.11.8|Comet2.2.2|fp32|unite-mup
sacrecomet
# Arguments can also be specified non-interactively:
sacrecomet --model unite-mup --prec fp32
```
The other functionality is to find specific citations for COMET models that you're using:
```
sacrecomet cite --model Unbabel/xcomet-xl
https://arxiv.org/abs/2310.10482
@misc{guerreiro2023xcomet,
title={xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection},
...
```
You can also list all the available models:
```
sacrecomet list
unbabel/wmt24-qe-task2-baseline
unbabel/wmt22-cometkiwi-da
unbabel/xcomet-xl
unbabel/xcomet-xxl
unbabel/towerinstruct-13b-v0.1
unbabel/towerinstruct-7b-v0.2
unbabel/towerbase-7b-v0.1
...
```
## Experiments
Documentation TODO
## Paper
Cite as:
```
@inproceedings{zouhar-etal-2024-pitfalls,
title = "Pitfalls and Outlooks in Using {COMET}",
author = "Zouhar, Vil{\'e}m and Chen, Pinzhen and Lam, Tsz Kin and Moghe, Nikita and Haddow, Barry",
editor = "Haddow, Barry and Kocmi, Tom and Koehn, Philipp and Monz, Christof",
booktitle = "Proceedings of the Ninth Conference on Machine Translation",
month = nov,
year = "2024",
address = "Miami, Florida, USA",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.wmt-1.121/",
doi = "10.18653/v1/2024.wmt-1.121",
pages = "1272--1288",
}
```
<img src="https://raw.githubusercontent.com/PinzhenChen/sacreCOMET/main/misc/poster.png" width="900vw">
## YouTube presentation (click image)
[<img src="https://img.youtube.com/vi/jDMvueySuPo/maxresdefault.jpg" width=400px>](https://www.youtube.com/watch?v=jDMvueySuPo)
## Changelog
- v1.0.1 (13 January 2025)
- Stable release
- v0.1.1 (13 January 2025)
- Add `r` in the signature before references.
- Add simple tests.
- v0.1.0 (30 October 2024):
- Add `list` command to list available models
- Add references usage to the SacreCOMET usage.
- Deprecate `sacrecomet cite model_name` positional model name specification. Citations now have to explicitly use the `--model` argument.
- v0.0.1 (7 August 2024)
- First release
Raw data
{
"_id": null,
"home_page": null,
"name": "sacrecomet",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": null,
"keywords": "machine-translation, evaluation, quality-estimaation",
"author": null,
"author_email": "Vil\u00e9m Zouhar <vzouhar@ethz.ch>, Tsz Kin Lam <tlam@ed.ac.uk>, Pinzhen Chen <pinzhen.chen@ed.ac.uk>, Nikita Moghe <nikita.moghe@ed.ac.uk>",
"download_url": "https://files.pythonhosted.org/packages/ed/04/f99aa4b5b609b178dcd68638a5e625ce1ae39d454516305193a712c15107/sacrecomet-1.0.1.tar.gz",
"platform": null,
"description": "# SacreCOMET [](https://pypi.org/project/sacrecomet/) [](https://github.com/PinzhenChen/sacreCOMET/actions/workflows/test.yml)\n\n> Since its introduction, the COMET metric has blazed a trail in the machine translation community, given its strong correlation with human judgements of translation quality. Its success stems from being a modified pre-trained multilingual model finetuned for quality assessment. However, it being a machine learning model also gives rise to a new set of pitfalls that may not be widely known. We investigate these unexpected behaviours from three aspects: 1) technical: obsolete software versions and compute precision; 2) data: empty content, language mismatch, and translationese at test time as well as distribution and domain biases in training; 3) usage and reporting: multi-reference support and model referencing in the literature. All of these problems imply that COMET scores is not comparable between papers or even technical setups and we put forward our perspective on fixing each issue. Furthermore, we release the SacreCOMET package that can generate a signature for the software and model configuration as well as an appropriate citation. The goal of this work is to help the community make more sound use of the COMET metric.\n\nRead the full paper [Pitfalls and Outlooks in Using COMET](https://aclanthology.org/2024.wmt-1.121/).\n\n## Tool\n\nThe Python tool has two functionalities.\nFirst, it creates a signature with your setup and COMET model:\n\n```\npip install sacrecomet\n\n# Without anything will try to detect the local environment and will\n# ask you questions about which COMET model you used.\n# Example output: Python3.11.8|Comet2.2.2|fp32|unite-mup\n\nsacrecomet \n\n# Arguments can also be specified non-interactively:\n\nsacrecomet --model unite-mup --prec fp32\n```\n\nThe other functionality is to find specific citations for COMET models that you're using:\n\n```\nsacrecomet cite --model Unbabel/xcomet-xl\n\nhttps://arxiv.org/abs/2310.10482\n@misc{guerreiro2023xcomet,\n title={xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection}, \n ...\n```\n\nYou can also list all the available models:\n```\nsacrecomet list\n\nunbabel/wmt24-qe-task2-baseline\nunbabel/wmt22-cometkiwi-da\nunbabel/xcomet-xl\nunbabel/xcomet-xxl\nunbabel/towerinstruct-13b-v0.1\nunbabel/towerinstruct-7b-v0.2\nunbabel/towerbase-7b-v0.1\n...\n```\n\n## Experiments\n\nDocumentation TODO\n\n## Paper\n\nCite as:\n\n```\n@inproceedings{zouhar-etal-2024-pitfalls,\n title = \"Pitfalls and Outlooks in Using {COMET}\",\n author = \"Zouhar, Vil{\\'e}m and Chen, Pinzhen and Lam, Tsz Kin and Moghe, Nikita and Haddow, Barry\",\n editor = \"Haddow, Barry and Kocmi, Tom and Koehn, Philipp and Monz, Christof\",\n booktitle = \"Proceedings of the Ninth Conference on Machine Translation\",\n month = nov,\n year = \"2024\",\n address = \"Miami, Florida, USA\",\n publisher = \"Association for Computational Linguistics\",\n url = \"https://aclanthology.org/2024.wmt-1.121/\",\n doi = \"10.18653/v1/2024.wmt-1.121\",\n pages = \"1272--1288\",\n}\n\n```\n\n<img src=\"https://raw.githubusercontent.com/PinzhenChen/sacreCOMET/main/misc/poster.png\" width=\"900vw\">\n\n## YouTube presentation (click image)\n\n[<img src=\"https://img.youtube.com/vi/jDMvueySuPo/maxresdefault.jpg\" width=400px>](https://www.youtube.com/watch?v=jDMvueySuPo)\n\n## Changelog\n\n- v1.0.1 (13 January 2025)\n - Stable release\n- v0.1.1 (13 January 2025)\n - Add `r` in the signature before references.\n - Add simple tests.\n- v0.1.0 (30 October 2024):\n - Add `list` command to list available models\n - Add references usage to the SacreCOMET usage.\n - Deprecate `sacrecomet cite model_name` positional model name specification. Citations now have to explicitly use the `--model` argument.\n- v0.0.1 (7 August 2024)\n - First release\n",
"bugtrack_url": null,
"license": null,
"summary": "Tool to guide you through reporting the use of COMET for machine translation evaluation.",
"version": "1.0.1",
"project_urls": {
"Homepage": "https://github.com/PinzhenChen/sacreCOMET",
"Issues": "https://github.com/PinzhenChen/sacreCOMET/issues"
},
"split_keywords": [
"machine-translation",
" evaluation",
" quality-estimaation"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4e34f1cadff3498259913e43c5bf5ae295442ada9a236d73c38cdd262ed20836",
"md5": "7ffde5a8f14e0c3497d3beb4a7f516de",
"sha256": "0f41d561e84435c92b12bb1e28140798c1ac4489e8567b5fcb548cc1862a9898"
},
"downloads": -1,
"filename": "sacrecomet-1.0.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "7ffde5a8f14e0c3497d3beb4a7f516de",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 10825,
"upload_time": "2025-01-13T10:19:22",
"upload_time_iso_8601": "2025-01-13T10:19:22.291454Z",
"url": "https://files.pythonhosted.org/packages/4e/34/f1cadff3498259913e43c5bf5ae295442ada9a236d73c38cdd262ed20836/sacrecomet-1.0.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ed04f99aa4b5b609b178dcd68638a5e625ce1ae39d454516305193a712c15107",
"md5": "3aa8e3fa3b8320a23d951952c1ea672a",
"sha256": "9cfb004774fea8ceee47468054c1a6c39127c42868e09cfc0498f676f180bb05"
},
"downloads": -1,
"filename": "sacrecomet-1.0.1.tar.gz",
"has_sig": false,
"md5_digest": "3aa8e3fa3b8320a23d951952c1ea672a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 12054,
"upload_time": "2025-01-13T10:19:24",
"upload_time_iso_8601": "2025-01-13T10:19:24.464054Z",
"url": "https://files.pythonhosted.org/packages/ed/04/f99aa4b5b609b178dcd68638a5e625ce1ae39d454516305193a712c15107/sacrecomet-1.0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-13 10:19:24",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "PinzhenChen",
"github_project": "sacreCOMET",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "sacrecomet"
}
JSON
