<p style="text-align: left; width: 20%;">
<a target="_blank">
<img src="figure/logo.png" alt="logo" style="width: 15%; min-width: 75px; display: block; margin-left: 0;">
</a>
</p>
# RaTEScore
RaTEScore: A Metric for Entity-Aware Radiology Text Similarity
<div style='display:flex; gap: 0.25rem; '>
<a href='https://angelakeke.github.io/RaTEScore/'><img src='https://img.shields.io/badge/website-URL-blueviolet'></a>
<a href='https://huggingface.co/Angelakeke/RaTE-NER'><img src='https://img.shields.io/badge/RaTENER-Model&Demo-blue'></a>
<a href='https://huggingface.co/datasets/Angelakeke/RaTE-NER'><img src='https://img.shields.io/badge/RaTENER-Dataset-blue'></a>
<a href='https://huggingface.co/datasets/Angelakeke/RaTE-Eval'><img src='https://img.shields.io/badge/RaTEEval-Benchmark-green'></a>
<a href=''><img src='https://img.shields.io/badge/Paper-PDF-red'></a>
</div>
## Overview
RaTEScore is a novel, entity-aware metric to assess the quality of medical reports generated by AI models. It emphasizes crucial medical entities such as diagnostic outcomes and anatomical details, and is robust against complex medical synonyms and sensitive to negation expressions. The evaluations demonstrate that RaTEScore aligns more closely with human preference than existing metrics.
Here is an illustration of the Computation of RaTEScore.
For more detailed about our pipeline, please refer to our paper.
## Installation
### Environment Preparation
```
pip install RaTEScore
```
## Usage
```python
from RaTEScore import RaTEScore
pred_report = ['There are no intracranial hemorrhages.',
'The musculature and soft tissues are intact.']
gt_report = ['There is no finding to suggest intracranial hemorrhage.',
'The muscle compartments are intact.']
ratescore = RaTEScore()
scores = ratescore.compute_score(pred_report, gt_report)
```
## RaTE-NER
RaTE-NER dataset is a large-scale, radiological named entity recognition (NER) dataset. We set up this dataset to serve our Medical Entity Recognition module of our proposed metric. To download or find out more about our dataset, please refer to [Hugginface](https://huggingface.co/datasets/Angelakeke/RaTE-NER) and our [paper]().
## RaTE-Eval
To effectively measure the alignment between automatic evaluation metrics and radiologists' assessments in medical text generation tasks, we have established a comprehensive benchmark, RaTE-Eval, that encompasses three tasks:
- Sentences-level Human Rating.
- Paragraph-level Human Rating.
- Rating on the Synthetic Reports.
To download or find out more about our dataset, please refer to [Hugginface](https://huggingface.co/datasets/Angelakeke/RaTE-Eval) and our [paper]().
## Default Scheme
For **Medical Entity Recognition module**, we default to use our fine-tuned version of DeBERTa on the RaTE-NER dataset. It is based on whole the IOB scheme.
For **Synonym Disambiguation Encoding module**, we default to ultilize [BioLORD-2023-C](https://huggingface.co/FremyCompany/BioLORD-2023-C). We have discussed this part in our paper. You can also change this to other pretrained bert models.
## Contact
If you have any questions, please feel free to contact zwk0629@sjtu.edu.cn.
## Citation
```
```
Raw data
{
"_id": null,
"home_page": "https://github.com/Angelakeke/RaTEScore",
"name": "RaTEScore",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": null,
"author": "Weike_Zhao",
"author_email": "zwk0629@sjtu.edu.cn",
"download_url": "https://files.pythonhosted.org/packages/64/07/b306e9de413211c12b207cb276594e77be1e1a351a3125b47e9c68c994e9/RaTEScore-0.5.0.tar.gz",
"platform": null,
"description": "<p style=\"text-align: left; width: 20%;\">\n <a target=\"_blank\">\n <img src=\"figure/logo.png\" alt=\"logo\" style=\"width: 15%; min-width: 75px; display: block; margin-left: 0;\">\n </a>\n</p>\n\n\n# RaTEScore\nRaTEScore: A Metric for Entity-Aware Radiology Text Similarity\n\n<div style='display:flex; gap: 0.25rem; '>\n<a href='https://angelakeke.github.io/RaTEScore/'><img src='https://img.shields.io/badge/website-URL-blueviolet'></a>\n<a href='https://huggingface.co/Angelakeke/RaTE-NER'><img src='https://img.shields.io/badge/RaTENER-Model&Demo-blue'></a>\n<a href='https://huggingface.co/datasets/Angelakeke/RaTE-NER'><img src='https://img.shields.io/badge/RaTENER-Dataset-blue'></a>\n<a href='https://huggingface.co/datasets/Angelakeke/RaTE-Eval'><img src='https://img.shields.io/badge/RaTEEval-Benchmark-green'></a> \n<a href=''><img src='https://img.shields.io/badge/Paper-PDF-red'></a>\n</div>\n\n## Overview\n\nRaTEScore is a novel, entity-aware metric to assess the quality of medical reports generated by AI models. It emphasizes crucial medical entities such as diagnostic outcomes and anatomical details, and is robust against complex medical synonyms and sensitive to negation expressions. The evaluations demonstrate that RaTEScore aligns more closely with human preference than existing metrics.\n\n\nHere is an illustration of the Computation of RaTEScore. \n\n\n\nFor more detailed about our pipeline, please refer to our paper.\n\n## Installation\n\n### Environment Preparation\n```\npip install RaTEScore\n```\n\n## Usage\n\n```python\nfrom RaTEScore import RaTEScore\n\npred_report = ['There are no intracranial hemorrhages.',\n 'The musculature and soft tissues are intact.']\n\ngt_report = ['There is no finding to suggest intracranial hemorrhage.',\n 'The muscle compartments are intact.']\n\nratescore = RaTEScore()\nscores = ratescore.compute_score(pred_report, gt_report)\n```\n\n## RaTE-NER\n\n RaTE-NER dataset is a large-scale, radiological named entity recognition (NER) dataset. We set up this dataset to serve our Medical Entity Recognition module of our proposed metric. To download or find out more about our dataset, please refer to [Hugginface](https://huggingface.co/datasets/Angelakeke/RaTE-NER) and our [paper]().\n\n## RaTE-Eval\n\nTo effectively measure the alignment between automatic evaluation metrics and radiologists' assessments in medical text generation tasks, we have established a comprehensive benchmark, RaTE-Eval, that encompasses three tasks:\n\n- Sentences-level Human Rating.\n- Paragraph-level Human Rating.\n- Rating on the Synthetic Reports.\n\nTo download or find out more about our dataset, please refer to [Hugginface](https://huggingface.co/datasets/Angelakeke/RaTE-Eval) and our [paper]().\n\n## Default Scheme\n\nFor **Medical Entity Recognition module**, we default to use our fine-tuned version of DeBERTa on the RaTE-NER dataset. It is based on whole the IOB scheme. \n\nFor **Synonym Disambiguation Encoding module**, we default to ultilize [BioLORD-2023-C](https://huggingface.co/FremyCompany/BioLORD-2023-C). We have discussed this part in our paper. You can also change this to other pretrained bert models.\n\n## Contact\nIf you have any questions, please feel free to contact zwk0629@sjtu.edu.cn.\n\n## Citation\n```\n\n```\n",
"bugtrack_url": null,
"license": null,
"summary": "A Metric for Entity-Aware Radiology Report Generation",
"version": "0.5.0",
"project_urls": {
"Homepage": "https://github.com/Angelakeke/RaTEScore"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "0013e0fbb85ab56d5191219118c2d6d0a335c5b0ee3e63433a54c4a72c036cd9",
"md5": "b1e39ee5dc8d56d8d2199e94a0427a7c",
"sha256": "6d0b7d6caaffe7bf330896dca46c530409e074c24f24e6bbc48494a8650d3e5d"
},
"downloads": -1,
"filename": "RaTEScore-0.5.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "b1e39ee5dc8d56d8d2199e94a0427a7c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 8548,
"upload_time": "2024-09-09T02:41:20",
"upload_time_iso_8601": "2024-09-09T02:41:20.766097Z",
"url": "https://files.pythonhosted.org/packages/00/13/e0fbb85ab56d5191219118c2d6d0a335c5b0ee3e63433a54c4a72c036cd9/RaTEScore-0.5.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6407b306e9de413211c12b207cb276594e77be1e1a351a3125b47e9c68c994e9",
"md5": "62a16552a19dbff2d3cc8cac6f9a7fb4",
"sha256": "35caa81140ede9534695b352af5bb81e4aa56fea97ad8d790eaa177f5723d544"
},
"downloads": -1,
"filename": "RaTEScore-0.5.0.tar.gz",
"has_sig": false,
"md5_digest": "62a16552a19dbff2d3cc8cac6f9a7fb4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 7591,
"upload_time": "2024-09-09T02:41:22",
"upload_time_iso_8601": "2024-09-09T02:41:22.832360Z",
"url": "https://files.pythonhosted.org/packages/64/07/b306e9de413211c12b207cb276594e77be1e1a351a3125b47e9c68c994e9/RaTEScore-0.5.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-09 02:41:22",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Angelakeke",
"github_project": "RaTEScore",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "ratescore"
}
JSON
