# BCQA (Benchmarking Complex QA)
BCQA is a benchmark for a wide range of complex Qa tasks. It also aims to provide a easy to use framework for evaluating retrieval and reasoning approaches for answering complex multi-hop questions.
# Setup
1) Clone the repo <br />
2) Create a conda environment conda create -n bcqa <br />
3) pip install -e .<br />
# Running Evaluation
The evaluation scripts for retreival and LLMs are in the evaluation folder
For instance to run dpr retreival for Wikimultihopqa run <br/>
python3 evaluation/wikimultihop/run_dpr_inference.py <br />
Before running the above script make sure you have configured the correct paths for the data and corpus files in evaluation/config.ini <br />
Example:
wikimultihopqa = /home/bcqa/BCQA/2wikimultihopQA <br />
wikimultihopqa-corpus = /home/bcqa/BCQA/wiki_musique_corpus.json <br />
## Coding Practices
### Auto-formatting code
1. Install `black`: ```pip install black``` or ```conda install black```
2. In your IDE: Enable formatting on save.
3. Install `isort`: ```pip install isort``` or ```conda install isort```
4. In your IDE: Enable sorting import on save.
In VS Code, you can do this using the following config:
```json
{
"editor.formatOnSave": true,
"editor.codeActionsOnSave": {
"source.organizeImports": true
}
}
```
### Type hints
Use [type hints](https://docs.python.org/3/library/typing.html) for __everything__! No exceptions.
### Docstrings
Write a docstring for __every__ function (except the main function). We use the [Google format](https://github.com/NilsJPWerner/autoDocstring/blob/HEAD/docs/google.md). In VS Code, you can use [autoDocstring](https://marketplace.visualstudio.com/items?itemName=njpwerner.autodocstring).
### Example
```python
def sum(a: float, b: float) -> float:
"""Compute the sum of a and b.
Args:
a (float): First number.
b (float): Second number.
Returns:
float: The sum of a and b.
"""
return a + b
```
Raw data
{
"_id": null,
"home_page": "https://github.com/VenkteshV/BCQA",
"name": "bcqa",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "Information Retrieval Transformer Networks BERT PyTorch Complex Question Answering IR NLP deep learning",
"author": "Venktesh V, Deepali Prabhu",
"author_email": "venkyviswa12@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/f6/36/0e745920ef0437a5e4a2b3ce79968fea448ffa6cd43dc5f327464c08f512/bcqa-1.1.4.tar.gz",
"platform": null,
"description": "# BCQA (Benchmarking Complex QA)\n\nBCQA is a benchmark for a wide range of complex Qa tasks. It also aims to provide a easy to use framework for evaluating retrieval and reasoning approaches for answering complex multi-hop questions.\n\n\n# Setup\n1) Clone the repo <br />\n2) Create a conda environment conda create -n bcqa <br />\n3) pip install -e .<br />\n\n# Running Evaluation\nThe evaluation scripts for retreival and LLMs are in the evaluation folder \n\nFor instance to run dpr retreival for Wikimultihopqa run <br/>\npython3 evaluation/wikimultihop/run_dpr_inference.py <br />\n\nBefore running the above script make sure you have configured the correct paths for the data and corpus files in evaluation/config.ini <br />\n\nExample: \nwikimultihopqa = /home/bcqa/BCQA/2wikimultihopQA <br />\nwikimultihopqa-corpus = /home/bcqa/BCQA/wiki_musique_corpus.json <br />\n\n\n## Coding Practices\n\n### Auto-formatting code\n1. Install `black`: ```pip install black``` or ```conda install black```\n2. In your IDE: Enable formatting on save.\n3. Install `isort`: ```pip install isort``` or ```conda install isort```\n4. In your IDE: Enable sorting import on save.\n\nIn VS Code, you can do this using the following config:\n```json\n{\n \"editor.formatOnSave\": true,\n \"editor.codeActionsOnSave\": {\n \"source.organizeImports\": true\n }\n}\n```\n\n### Type hints\nUse [type hints](https://docs.python.org/3/library/typing.html) for __everything__! No exceptions.\n\n### Docstrings\nWrite a docstring for __every__ function (except the main function). We use the [Google format](https://github.com/NilsJPWerner/autoDocstring/blob/HEAD/docs/google.md). In VS Code, you can use [autoDocstring](https://marketplace.visualstudio.com/items?itemName=njpwerner.autodocstring).\n\n### Example\n```python\ndef sum(a: float, b: float) -> float:\n \"\"\"Compute the sum of a and b.\n\n Args:\n a (float): First number.\n b (float): Second number.\n \n Returns:\n float: The sum of a and b.\n \"\"\"\n\n return a + b\n```\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "A Benchmark for Complex Heterogeneous Question answering",
"version": "1.1.4",
"project_urls": {
"Homepage": "https://github.com/VenkteshV/BCQA"
},
"split_keywords": [
"information",
"retrieval",
"transformer",
"networks",
"bert",
"pytorch",
"complex",
"question",
"answering",
"ir",
"nlp",
"deep",
"learning"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "78d03e83b61fc1ccda5c21998a28c3b27dd8a231ef39bec504ea26194b322c0e",
"md5": "9880934a253fd791a5555815821a663c",
"sha256": "25a03667978202e2a7eca49bd0dd07444f55274566fe10c05d3c902ca40498ce"
},
"downloads": -1,
"filename": "bcqa-1.1.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "9880934a253fd791a5555815821a663c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 144660,
"upload_time": "2024-05-30T23:17:16",
"upload_time_iso_8601": "2024-05-30T23:17:16.353321Z",
"url": "https://files.pythonhosted.org/packages/78/d0/3e83b61fc1ccda5c21998a28c3b27dd8a231ef39bec504ea26194b322c0e/bcqa-1.1.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f6360e745920ef0437a5e4a2b3ce79968fea448ffa6cd43dc5f327464c08f512",
"md5": "af7043230247a9ba84ef16b4811b7ed1",
"sha256": "d31652e2651c600e2c7f89041bb8ab7938828be3841845f030e157116a525a5b"
},
"downloads": -1,
"filename": "bcqa-1.1.4.tar.gz",
"has_sig": false,
"md5_digest": "af7043230247a9ba84ef16b4811b7ed1",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 92612,
"upload_time": "2024-05-30T23:17:18",
"upload_time_iso_8601": "2024-05-30T23:17:18.048835Z",
"url": "https://files.pythonhosted.org/packages/f6/36/0e745920ef0437a5e4a2b3ce79968fea448ffa6cd43dc5f327464c08f512/bcqa-1.1.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-30 23:17:18",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "VenkteshV",
"github_project": "BCQA",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "bcqa"
}
JSON
