RQUGE🤗: Reference-Free Metric for Evaluating Question Generation by Answering the Question
=================
[](#python)
[](https://arxiv.org/abs/2211.01482)
[](https://pypi.python.org/pypi/rquge/)[](https://opensource.org/licenses/MIT)
[](https://github.com/psf/black)
<p align="center">
<img src="meta.jpeg" width="500"/>
</p>
We propose RQUGE, a **R**eference-free **QU**estion **G**eneration **E**valuation metric that can compute the quality of the candidate question without requiring the access to the reference question. Given the corresponding context and answer span, our metric calculates the acceptability score by applying a general question-answering module, followed by a span scorer. You can find more detail in [the paper](https://arxiv.org/abs/2211.01482) (ACL2023).
<p align="center">
<img src="main_model.jpg" width="700"/>
</p>
Contents
---------------
- [Huggingface Evaluate🤗](#hf_evaluate)
- [Installation](#install)
- [Calculation](#calculation)
- [Citation](#citation)
<a name="hf_evaluate"/>
HuggingFace Evaluation 🤗
--------------
RQUGE score is available on [Huggingface Evaluate](https://huggingface.co/docs/evaluate/index). It can be used as:
```
from evaluate import load
rqugescore = load("alirezamsh/rquge")
generated_questions = ["how is the weather?"]
contexts = ["the weather is sunny"]
answers = ["sunny"]
results = rquge.compute(generated_questions=generated_questions, contexts=contexts, answers=answers)
print(results["mean_score"])
>>> [5.05]
```
The demo and further details are also available on [here](https://huggingface.co/spaces/alirezamsh/rquge).
<a name="install"/>
Installation
--------------
You should have the following packages:
- transformers
- pytorch
- sentencepiece
Install from pypi with pip by
```
pip install rquge
```
Install latest unstable version from the master branch on Github by:
```
pip install git+https://github.com/alirezamshi/RQUGE
```
Install it from the source by:
```
git clone https://github.com/alirezamshi/RQUGE
cd RQUGE
pip install .
```
Note: you need to download the pre-trained model for the span scorer module (available on [Huggingface](https://huggingface.co/alirezamsh/quip-512-mocha)🤗 ```alirezamsh/quip-512-mocha```):
```
wget https://storage.googleapis.com/sfr-qafacteval-research/quip-512-mocha.tar.gz
tar -xzvf quip-512-mocha.tar.gz
rm quip-512-mocha.tar.gz
```
<a name="calculation"/>
RQUGE Calculation
--------------
#### Python Function
The RQUGE class is provided in ```rquge_score/scorer.py```. We also provide a python function in ```rquge_score_cli/scorer_cli.py``` to use different features of RQUGE metric.
#### Command Line Interface (CLI)
We provide a command line interface (CLI) of RQUGE, you can use it as follows:
```
rquge --input_type #MODE --sp_scorer_path #PATH_TO_SPAN_SCORER --qa_model_path #PATH_TO_QA_MODEL --context #CONTEXT_FILE --question #QUESTION --answer #ANSWER --output_path #OUTPUT
#MODE: The type of input (sample or offline). In the sample mode, "--context", "--question", and "--answer" commands contain string, while in offline mode, they contain path to files including contexts, corresponding questions and answers
#PATH_TO_SPAN_SCORER: path to the local checkpoint of span scorer model or "alirezamsh/quip-512-mocha"
#PATH_TO_QA_MODEL: name of QA model on Huggingface or local path
#CONTEXT_FILE: a text file containing one context per line (directly in the input in "sample" mode)
#QUESTION_FILE: a text file containing one question per line (directly in the input in "sample" mode)
#ANSWER_FILE: a text file containing one answer per line (directly in the input in "sample" mode)
#OUTPUT: local path for saving RQUGE scores for each (context,question,answer) pair
```
Here is a sample score computation for the interactive mode
```
rquge --input_type sample --sp_scorer_path ckpt/quip-512-mocha --qa_model_path 'allenai/unifiedqa-v2-t5-large-1363200' --context "the weather is sunny" --question "how is the weather?" --answer sunny
```
**Note:** the rquge score is between 1 to 5.
<a name="citation"/>
Citation
-------------
<a name="citations"/>
If you use this code for your research, please cite the following work:
```
@misc{mohammadshahi2022rquge,
title={RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question},
author={Alireza Mohammadshahi and Thomas Scialom and Majid Yazdani and Pouya Yanki and Angela Fan and James Henderson and Marzieh Saeidi},
year={2022},
eprint={2211.01482},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```
Have a question not listed here? Open [a GitHub Issue](https://github.com/alirezamshi/RQUGE/issues) or
send us an [email](alireza.mohammadshahi@idiap.ch).
Raw data
{
"_id": null,
"home_page": "https://github.com/alirezamshi/RQUGE",
"name": "rquge",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "BERT NLP deep learning QG QA metric",
"author": "Alireza Mohammadshahi*, Thomas Scialom, Majid Yazdani, Pouya Yanki, Angela Fan, James Henderson, Marzieh Saeidi",
"author_email": "alireza.msh1373@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/69/e8/50294bedf549f3d61d6d1ecedb259614ea5020088b95025a5a4946dc6ab6/rquge-0.3.tar.gz",
"platform": null,
"description": "RQUGE\ud83e\udd17: Reference-Free Metric for Evaluating Question Generation by Answering the Question\n=================\n[](#python)\n[](https://arxiv.org/abs/2211.01482)\n[](https://pypi.python.org/pypi/rquge/)[](https://opensource.org/licenses/MIT) \n[](https://github.com/psf/black) \n\n\n<p align=\"center\">\n <img src=\"meta.jpeg\" width=\"500\"/>\n</p>\n\nWe propose RQUGE, a **R**eference-free **QU**estion **G**eneration **E**valuation metric that can compute the quality of the candidate question without requiring the access to the reference question. Given the corresponding context and answer span, our metric calculates the acceptability score by applying a general question-answering module, followed by a span scorer. You can find more detail in [the paper](https://arxiv.org/abs/2211.01482) (ACL2023).\n\n<p align=\"center\">\n <img src=\"main_model.jpg\" width=\"700\"/>\n</p>\n\n\nContents\n---------------\n\n- [Huggingface Evaluate\ud83e\udd17](#hf_evaluate)\n- [Installation](#install)\n- [Calculation](#calculation)\n- [Citation](#citation)\n\n<a name=\"hf_evaluate\"/> \n\nHuggingFace Evaluation \ud83e\udd17\n-------------- \n\nRQUGE score is available on [Huggingface Evaluate](https://huggingface.co/docs/evaluate/index). It can be used as:\n\n```\nfrom evaluate import load\nrqugescore = load(\"alirezamsh/rquge\")\ngenerated_questions = [\"how is the weather?\"]\ncontexts = [\"the weather is sunny\"]\nanswers = [\"sunny\"]\nresults = rquge.compute(generated_questions=generated_questions, contexts=contexts, answers=answers)\nprint(results[\"mean_score\"])\n>>> [5.05]\n```\n\nThe demo and further details are also available on [here](https://huggingface.co/spaces/alirezamsh/rquge).\n\n<a name=\"install\"/> \n\nInstallation\n-------------- \n\nYou should have the following packages:\n\n- transformers\n- pytorch\n- sentencepiece\n\nInstall from pypi with pip by\n```\npip install rquge\n```\nInstall latest unstable version from the master branch on Github by:\n```\npip install git+https://github.com/alirezamshi/RQUGE\n```\nInstall it from the source by:\n```\ngit clone https://github.com/alirezamshi/RQUGE\ncd RQUGE\npip install .\n```\n\nNote: you need to download the pre-trained model for the span scorer module (available on [Huggingface](https://huggingface.co/alirezamsh/quip-512-mocha)\ud83e\udd17 ```alirezamsh/quip-512-mocha```):\n```\nwget https://storage.googleapis.com/sfr-qafacteval-research/quip-512-mocha.tar.gz\ntar -xzvf quip-512-mocha.tar.gz\nrm quip-512-mocha.tar.gz\n```\n\n<a name=\"calculation\"/> \n\nRQUGE Calculation\n-------------- \n\n#### Python Function\nThe RQUGE class is provided in ```rquge_score/scorer.py```. We also provide a python function in ```rquge_score_cli/scorer_cli.py``` to use different features of RQUGE metric.\n\n#### Command Line Interface (CLI)\nWe provide a command line interface (CLI) of RQUGE, you can use it as follows:\n```\nrquge --input_type #MODE --sp_scorer_path #PATH_TO_SPAN_SCORER --qa_model_path #PATH_TO_QA_MODEL --context #CONTEXT_FILE --question #QUESTION --answer #ANSWER --output_path #OUTPUT\n\n#MODE: The type of input (sample or offline). In the sample mode, \"--context\", \"--question\", and \"--answer\" commands contain string, while in offline mode, they contain path to files including contexts, corresponding questions and answers\n#PATH_TO_SPAN_SCORER: path to the local checkpoint of span scorer model or \"alirezamsh/quip-512-mocha\"\n#PATH_TO_QA_MODEL: name of QA model on Huggingface or local path \n#CONTEXT_FILE: a text file containing one context per line (directly in the input in \"sample\" mode)\n#QUESTION_FILE: a text file containing one question per line (directly in the input in \"sample\" mode)\n#ANSWER_FILE: a text file containing one answer per line (directly in the input in \"sample\" mode)\n#OUTPUT: local path for saving RQUGE scores for each (context,question,answer) pair\n```\n\nHere is a sample score computation for the interactive mode\n\n```\nrquge --input_type sample --sp_scorer_path ckpt/quip-512-mocha --qa_model_path 'allenai/unifiedqa-v2-t5-large-1363200' --context \"the weather is sunny\" --question \"how is the weather?\" --answer sunny\n```\n\n**Note:** the rquge score is between 1 to 5.\n\n<a name=\"citation\"/> \n\nCitation\n-------------\n\n<a name=\"citations\"/> \n\nIf you use this code for your research, please cite the following work:\n```\n@misc{mohammadshahi2022rquge,\n title={RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question},\n author={Alireza Mohammadshahi and Thomas Scialom and Majid Yazdani and Pouya Yanki and Angela Fan and James Henderson and Marzieh Saeidi},\n year={2022},\n eprint={2211.01482},\n archivePrefix={arXiv},\n primaryClass={cs.CL}\n}\n```\nHave a question not listed here? Open [a GitHub Issue](https://github.com/alirezamshi/RQUGE/issues) or \nsend us an [email](alireza.mohammadshahi@idiap.ch).\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "PyTorch implementation of RQUGE score",
"version": "0.3",
"project_urls": {
"Homepage": "https://github.com/alirezamshi/RQUGE"
},
"split_keywords": [
"bert",
"nlp",
"deep",
"learning",
"qg",
"qa",
"metric"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "52efeda7d5db14e1df70da730fd8fc125142086cf085e41bc6dadfb5f80b2f86",
"md5": "baf1bb7fa5d3f4af0ae591a93aa5ad28",
"sha256": "24c41fb48bfb8263adbe575c030c1bfa622536f28fa9bd89cd170f9637255b71"
},
"downloads": -1,
"filename": "rquge-0.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "baf1bb7fa5d3f4af0ae591a93aa5ad28",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 7101,
"upload_time": "2023-06-15T14:52:59",
"upload_time_iso_8601": "2023-06-15T14:52:59.231691Z",
"url": "https://files.pythonhosted.org/packages/52/ef/eda7d5db14e1df70da730fd8fc125142086cf085e41bc6dadfb5f80b2f86/rquge-0.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "69e850294bedf549f3d61d6d1ecedb259614ea5020088b95025a5a4946dc6ab6",
"md5": "b5095700c917797b14d24149ca388e16",
"sha256": "a08b90cedd3dadb0b6242ce41b0e4b605414a2c051e2cec59c212f18e278a8d5"
},
"downloads": -1,
"filename": "rquge-0.3.tar.gz",
"has_sig": false,
"md5_digest": "b5095700c917797b14d24149ca388e16",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 6485,
"upload_time": "2023-06-15T14:53:01",
"upload_time_iso_8601": "2023-06-15T14:53:01.265425Z",
"url": "https://files.pythonhosted.org/packages/69/e8/50294bedf549f3d61d6d1ecedb259614ea5020088b95025a5a4946dc6ab6/rquge-0.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-06-15 14:53:01",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "alirezamshi",
"github_project": "RQUGE",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "rquge"
}
JSON
