# NoMIRACL: A Multilingual Relevance Assessment Dataset for RAG Applications
<p align="center">
<a href="https://aclanthology.org/2024.findings-emnlp.730/">
<img alt="EMNLP2024" src="https://img.shields.io/badge/Citation-EMNLP_2024-orange.svg">
</a>
<a href="https://github.com/project-miracl/nomiracl">
<img alt="Stars" src="https://img.shields.io/github/stars/project-miracl/nomiracl.svg?style=flat&logo=github&colorB=blue&label=stars">
</a>
<a href="https://www.python.org/">
<img alt="Build" src="https://img.shields.io/badge/Made%20with-Python-1f425f.svg?color=purple">
</a>
<a href="https://github.com/project-miracl/nomiracl/blob/main/LICENSE">
<img alt="License" src="https://img.shields.io/github/license/project-miracl/nomiracl.svg?style=flat&colorB=green">
</a>
</p>
<h4 align="center">
<a href="./"><img style="float: middle;" width="800" height="570" src="./images/nomiracl-teaser.png" /></a>
<footer><br clear="all"/>The image has been generated using miramuseai.net and Adobe photoshop.</footer>
</h4>
NoMIRACL [[EMNLP'24 Findings]](https://aclanthology.org/2024.findings-emnlp.730/) is a multilingual relevance assessment dataset for evaluating query \& passage relevancy in large language models (LLMs). This is extremely useful in RAG settings, i.e., when a retrieval systems retrieves a subset of passages or documents which either can or cannot be relevant to the user query. The LLM (as the generator) should assess the relevancy and only answer -- *if* a relevant passage is found within the subset, else abstain from answering.
**This repository provides starter code to evaluate diverse multilingual LLMs using our prompt template on NoMIRACL.**
For more information, checkout out our publication:
- [“Knowing When You Don’t Know”: A Multilingual Relevance Assessment Dataset for Robust Retrieval-Augmented Generation](https://aclanthology.org/2024.findings-emnlp.730/) (Thakur et al., :star: EMNLP 2024 Findings)
## :wrench: Installation
You can install NoMIRACL code repository via pip:
```python
pip install nomiracl
```
Optional dependencies:
```python
pip install nomiracl[hf] # install accelerate, bitsandbytes & peft
pip install nomiracl[vllm] # install vllm & dependencies
```
If you want to build from source, use:
```bash
$ git clone https://github.com/project-miracl/nomiracl.git
$ cd nomiracl
$ pip install -e .
```
## :star: Getting Started
#### 1. Loading NoMIRACL Dataset
- 50\% of relevant examples, 50\% of non-relevant, both maximum capped at 250.
- Full example available in [sample_load_no_miracl.py](./examples/sample_load_no_miracl.py).
```python
from nomiracl.dataset import NoMIRACLDataLoader
data_loader = NoMIRACLDataLoader(language = "english",
split = "test", # or 'dev'
hf_dataset_name="miracl/nomiracl",
load_from_huggingface=True)
corpus, queries, qrels = data_loader.load_data_sample(
relevant_ratio = 0.5, non_relevant_ratio = 0.5, max_sample_pool = 250)
```
#### 2. LLM prompt generation
- Full example available in [sample_model_generation.py](./examples/sample_model_generation.py).
```python
from nomiracl.generation.utils import load_model
model_name = "meta-llama/Meta-Llama-3-8B-Instruct"
# List of techniques supported in nomiracl:
# huggingface (GPUs), vllm (GPUs), cohere (API), openai (API), nvidia (API), azure (API), anyscale (API)
# `cohere` requires COHERE_API_KEY, `openai` requires OPENAI_API_KEY, `nvidia` requires NVIDIA_API_KEY
# `azure` requires AZURE_OPENAI_API_BASE, AZURE_OPENAI_API_VERSION and AZURE_OPENAI_API_KEY
# `anyscale` requires ANYSCALE_BASE_URL and ANYSCALE_API_KEY.
technique = "vllm" # or huggingface or nvidia, anyscale etc.
model = load_model(
technique, # technique
model_name, # model_name
cache_dir="<your-cache-dir>", # extra kwargs
batch_size=2, # extra kwargs
num_gpus=1, # extra kwargs
concurrency=2 # extra kwargs
)
# Sample prompts
prompts = [
"What is the capital of France?",
"What is the capital of Germany?",
"What is the capital of Italy?",
]
model_results = model.call(prompts)
for prompt, result in zip(prompts, model_results):
print("Prompt: {}".format(prompt))
print("{} result: {}".format(model_name, result))
```
#### 3. Loading our paper used prompt templates
- Full example available in [sample_vanilla_prompt_exploration.py](./examples/sample_vanilla_prompt_exploration.py).
```python
from nomiracl.prompts.utils import load_prompt_template
# Options include: vanilla, role, repeat, explanation
prompt_cls = load_prompt_template("vanilla", count = 10) # as we include 10 passages
query = "Which is the best programming language?"
passages = [
"Python is the best programming language.",
"Javascript is the best programming language.",
"Go is the best programming language.",
"Java is the best programming language.",
"C# is the best programming language.",
"Ruby is the best programming language.",
"R is the best programming language.",
"C++ is the best programming language.",
"C is the best programming language.",
"Rust is the best programming language.",
]
prompt = prompt_cls(query=query, passages=passages)
```
Or you can provide your **own** custom prompt template by modifying the `self.template` in `nomiracl.VanillaTemplate`.
```python
from nomiracl.prompts import VanillaTemplate
class CustomTemplate(VanillaTemplate):
def __init__(self, count: int = 1):
super().__init__(count)
self.template = (
"This is a pairwise prompt template. Respond as either "{self.answer}" or "{self.no_answer}".'
+ "\n\nQUESTION:\n{query}\n\n"
+ "CONTEXT:\n"
+ "\n\n".join(
[
"[{}] {}".format(i, "{" + passage + "}")
for i, passage in enumerate(self.passage_variables, 1)
]
)
+ "\n\nOUTPUT:\n"
)
```
## :hugs: NoMIRACL Dataset
The NoMIRACL dataset is available in HuggingFace under: `miracl/nomiracl`.
Languages covered: Arabic (ar), Bengali (bn), German (de), English (en), Spanish (es), Persian (fa), Finnish (fi), French (fr), Hindi (hi), Indonesian (id), Japanese (ja), Korean (ko), Russian (ru), Swahili (sw), Thai (th), Yoruba (yo), Chinese (zh).
HuggingFace Page: [https://huggingface.co/datasets/miracl/nomiracl](https://huggingface.co/datasets/miracl/nomiracl)
```python
import datasets
language = 'german' # or any of the 18 languages
subset = 'relevant' # or 'non_relevant'
split = 'test' # or 'dev' for development split
# four combinations available: 'dev.relevant', 'dev.non_relevant', 'test.relevant' and 'test.non_relevant'
nomiracl = datasets.load_dataset('miracl/nomiracl', language, split=f'{split}.{subset}')
```
### Baseline Accuracy on NoMIRACL non-relevant subset (test split, maximum cap of 250 per language)
Baseline results (250 queries) are available within the repository under `./results/baselines/non_relevant`.
An example datapoint under `./results/baselines/non_relevant/en.test.vanilla_prompt.jsonl`
```
{
"query_id": "842558#0",
"docids": ["2842207#5", "7004944#45", "3310762#14", "47220460#1", "36451733#7", "3310762#20", "4724576#4", "22373402#0", "52203230#0", "23126218#4"],
"prompt": "I will give you a question and several contexts containing information about the question. [ ... ] \n\nOUTPUT:\n",
"template": "vanilla",
"results": {"gpt-4-azure": "Yes, answer is present.",
"llama-2-13b-chat": "\nYes, answer is present in [6].\n\nNo answers found in the other contexts.",
[...]
"aya-101": "Wales"}
}
```
### Baseline Accuracy on NoMIRACL relevant subset (test split, maximum cap of 250 per language)
Baseline results (250 queries) are available within the repository under `./results/baselines/relevant`.
An example datapoint under `./results/baselines/relevant/en.test.vanilla_prompt.jsonl`
```
{
"query_id": "8706103#0",
"docids": ["42057469#2", "4998067#1", "29247933#0", "162619#81", "422315#13", "26790310#4", "41298602#18", "22816#16", "123427#61", "23576525#0"],
"prompt": "I will give you a question and several contexts containing information about the question. [ ... ] \n\nQUESTION:\nWhat is the course that will be discontinued as defined by the National Education Policy? [ ... ] \n\nOUTPUT:\n",
"template": "vanilla",
"results": {"gpt-4-azure": "I don't know.",
"llama-2-13b-chat": "Please answer the question based on the given contexts.",
[...]
"aya-101": "I don't know"}
}
```
## NoMIRACL Dataset Construction
<img src="./images/NoMIRACL-Flowchart.drawio.png" width="1013" height="179" />
NoMIRACL is a multilingual dataset designed to evaluate LLM robustness in relevance assessment to help avoid errors in first-stage retrieval. The dataset covers 18 typologically diverse languages and includes two subsets: non-relevant and relevant.
### Non-Relevant Subset (F)
- Queries with no-known answers within the retrieved oracle passages.
- All top-k passages manually judged as non-relevant (relevancy score = 0).
### Relevant Subset (T)
- Queries with known answers within the retrieved oracle passages.
- At least one of the top-k passages manually judged as relevant (relevancy score = 1).
## Evaluation Metrics
<img src="./images/NoMIRACL-confusion-matrix.png" width="411" height="193"/>
We conduct a robustness evaluation using a binary classification task, comparing LLM predictions against the ground truth provided in NoMIRACL. The metrics used are hallucination rate and error rate.
- **Hallucination Rate:** `FP/(FP + TN)` Measures the model's tendency to hallucinate an answer when no answer is present in the non-relevant subset.
- **Error Rate:** `FN/(FN + TP)` Measures the model's inaccuracy in recognizing relevant passages in the relevant subset.
## :handshake: Collaboration and Acknowledgements
The NoMIRACL dataset has been made possible due to a collaborative effort of the following universities and organizations:
- University of Waterloo
- Huawei Noah's Ark Lab
Parts of the NoMIRACL code structure has been inspired by:
- [https://github.com/McGill-NLP/instruct-qa](https://github.com/McGill-NLP/instruct-qa)
## :scroll: Citations
If you use NoMIRACL or parts in a research paper, please cite our work as follows:
```
@article{thakur:2024,
author = {Nandan Thakur and
Luiz Bonifacio and
Xinyu Zhang and
Odunayo Ogundepo and
Ehsan Kamalloo and
David Alfonso{-}Hermelo and
Xiaoguang Li and
Qun Liu and
Boxing Chen and
Mehdi Rezagholizadeh and
Jimmy Lin},
title = {NoMIRACL: Knowing When You Don't Know for Robust Multilingual Retrieval-Augmented
Generation},
journal = {CoRR},
volume = {abs/2312.11361},
year = {2023},
url = {https://doi.org/10.48550/arXiv.2312.11361},
doi = {10.48550/ARXIV.2312.11361},
eprinttype = {arXiv},
eprint = {2312.11361},
timestamp = {Tue, 16 Jan 2024 11:57:42 +0100},
biburl = {https://dblp.org/rec/journals/corr/abs-2312-11361.bib},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
```
---
Contact person: Nandan Thakur, [nandan.thakur@uwaterloo.co](mailto:nandan.thakur@uwaterloo.ca)
> This repository contains experimental software and is published for the sole purpose of giving additional background details on the respective publication.
Raw data
{
"_id": null,
"home_page": null,
"name": "nomiracl",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "Nandan Thakur <nandant@gmail.com>",
"keywords": "Transformer Networks, PyTorch, NLP, deep learning, LLM, Hallucination, Relevance, Multilingual",
"author": null,
"author_email": "Nandan Thakur <nandant@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/26/d8/7e43a68b1da5245f657dea7e25c02d001a6b91b99ca324394f1d5eb9611f/nomiracl-1.0.0.tar.gz",
"platform": null,
"description": "# NoMIRACL: A Multilingual Relevance Assessment Dataset for RAG Applications\n<p align=\"center\">\n <a href=\"https://aclanthology.org/2024.findings-emnlp.730/\">\n <img alt=\"EMNLP2024\" src=\"https://img.shields.io/badge/Citation-EMNLP_2024-orange.svg\">\n </a>\n <a href=\"https://github.com/project-miracl/nomiracl\">\n <img alt=\"Stars\" src=\"https://img.shields.io/github/stars/project-miracl/nomiracl.svg?style=flat&logo=github&colorB=blue&label=stars\">\n </a>\n <a href=\"https://www.python.org/\">\n <img alt=\"Build\" src=\"https://img.shields.io/badge/Made%20with-Python-1f425f.svg?color=purple\">\n </a>\n <a href=\"https://github.com/project-miracl/nomiracl/blob/main/LICENSE\">\n <img alt=\"License\" src=\"https://img.shields.io/github/license/project-miracl/nomiracl.svg?style=flat&colorB=green\">\n </a>\n</p>\n\n<h4 align=\"center\">\n <a href=\"./\"><img style=\"float: middle;\" width=\"800\" height=\"570\" src=\"./images/nomiracl-teaser.png\" /></a>\n <footer><br clear=\"all\"/>The image has been generated using miramuseai.net and Adobe photoshop.</footer>\n</h4>\n\nNoMIRACL [[EMNLP'24 Findings]](https://aclanthology.org/2024.findings-emnlp.730/) is a multilingual relevance assessment dataset for evaluating query \\& passage relevancy in large language models (LLMs). This is extremely useful in RAG settings, i.e., when a retrieval systems retrieves a subset of passages or documents which either can or cannot be relevant to the user query. The LLM (as the generator) should assess the relevancy and only answer -- *if* a relevant passage is found within the subset, else abstain from answering.\n\n**This repository provides starter code to evaluate diverse multilingual LLMs using our prompt template on NoMIRACL.**\n\nFor more information, checkout out our publication:\n- [\u201cKnowing When You Don\u2019t Know\u201d: A Multilingual Relevance Assessment Dataset for Robust Retrieval-Augmented Generation](https://aclanthology.org/2024.findings-emnlp.730/) (Thakur et al., :star: EMNLP 2024 Findings)\n\n\n## :wrench: Installation\nYou can install NoMIRACL code repository via pip:\n\n```python\npip install nomiracl\n```\n\nOptional dependencies:\n```python\npip install nomiracl[hf] # install accelerate, bitsandbytes & peft\npip install nomiracl[vllm] # install vllm & dependencies \n```\n\nIf you want to build from source, use:\n\n```bash\n$ git clone https://github.com/project-miracl/nomiracl.git\n$ cd nomiracl\n$ pip install -e .\n```\n\n## :star: Getting Started\n\n#### 1. Loading NoMIRACL Dataset \n- 50\\% of relevant examples, 50\\% of non-relevant, both maximum capped at 250. \n- Full example available in [sample_load_no_miracl.py](./examples/sample_load_no_miracl.py).\n```python\nfrom nomiracl.dataset import NoMIRACLDataLoader\n\ndata_loader = NoMIRACLDataLoader(language = \"english\", \n split = \"test\", # or 'dev' \n hf_dataset_name=\"miracl/nomiracl\", \n load_from_huggingface=True)\n \ncorpus, queries, qrels = data_loader.load_data_sample(\n relevant_ratio = 0.5, non_relevant_ratio = 0.5, max_sample_pool = 250)\n```\n\n#### 2. LLM prompt generation\n- Full example available in [sample_model_generation.py](./examples/sample_model_generation.py).\n```python\nfrom nomiracl.generation.utils import load_model\n\nmodel_name = \"meta-llama/Meta-Llama-3-8B-Instruct\"\n\n# List of techniques supported in nomiracl: \n# huggingface (GPUs), vllm (GPUs), cohere (API), openai (API), nvidia (API), azure (API), anyscale (API)\n# `cohere` requires COHERE_API_KEY, `openai` requires OPENAI_API_KEY, `nvidia` requires NVIDIA_API_KEY\n# `azure` requires AZURE_OPENAI_API_BASE, AZURE_OPENAI_API_VERSION and AZURE_OPENAI_API_KEY\n# `anyscale` requires ANYSCALE_BASE_URL and ANYSCALE_API_KEY.\n\ntechnique = \"vllm\" # or huggingface or nvidia, anyscale etc.\n\nmodel = load_model(\n technique, # technique\n model_name, # model_name\n cache_dir=\"<your-cache-dir>\", # extra kwargs\n batch_size=2, # extra kwargs\n num_gpus=1, # extra kwargs\n concurrency=2 # extra kwargs\n)\n\n# Sample prompts\nprompts = [\n \"What is the capital of France?\",\n \"What is the capital of Germany?\",\n \"What is the capital of Italy?\",\n]\n\nmodel_results = model.call(prompts)\n\nfor prompt, result in zip(prompts, model_results):\n print(\"Prompt: {}\".format(prompt))\n print(\"{} result: {}\".format(model_name, result))\n```\n\n#### 3. Loading our paper used prompt templates\n- Full example available in [sample_vanilla_prompt_exploration.py](./examples/sample_vanilla_prompt_exploration.py).\n\n```python\nfrom nomiracl.prompts.utils import load_prompt_template\n\n# Options include: vanilla, role, repeat, explanation \nprompt_cls = load_prompt_template(\"vanilla\", count = 10) # as we include 10 passages\n\nquery = \"Which is the best programming language?\"\n\npassages = [\n \"Python is the best programming language.\",\n \"Javascript is the best programming language.\",\n \"Go is the best programming language.\",\n \"Java is the best programming language.\",\n \"C# is the best programming language.\",\n \"Ruby is the best programming language.\",\n \"R is the best programming language.\",\n \"C++ is the best programming language.\",\n \"C is the best programming language.\",\n \"Rust is the best programming language.\",\n]\n\nprompt = prompt_cls(query=query, passages=passages)\n```\n\nOr you can provide your **own** custom prompt template by modifying the `self.template` in `nomiracl.VanillaTemplate`.\n\n```python\nfrom nomiracl.prompts import VanillaTemplate\n\nclass CustomTemplate(VanillaTemplate):\n def __init__(self, count: int = 1):\n super().__init__(count)\n self.template = (\n \"This is a pairwise prompt template. Respond as either \"{self.answer}\" or \"{self.no_answer}\".'\n + \"\\n\\nQUESTION:\\n{query}\\n\\n\"\n + \"CONTEXT:\\n\"\n + \"\\n\\n\".join(\n [\n \"[{}] {}\".format(i, \"{\" + passage + \"}\")\n for i, passage in enumerate(self.passage_variables, 1)\n ]\n )\n + \"\\n\\nOUTPUT:\\n\"\n )\n```\n\n## :hugs: NoMIRACL Dataset\n\nThe NoMIRACL dataset is available in HuggingFace under: `miracl/nomiracl`.\n\nLanguages covered: Arabic (ar), Bengali (bn), German (de), English (en), Spanish (es), Persian (fa), Finnish (fi), French (fr), Hindi (hi), Indonesian (id), Japanese (ja), Korean (ko), Russian (ru), Swahili (sw), Thai (th), Yoruba (yo), Chinese (zh).\n\nHuggingFace Page: [https://huggingface.co/datasets/miracl/nomiracl](https://huggingface.co/datasets/miracl/nomiracl) \n\n```python\nimport datasets\n\nlanguage = 'german' # or any of the 18 languages\nsubset = 'relevant' # or 'non_relevant'\nsplit = 'test' # or 'dev' for development split\n\n# four combinations available: 'dev.relevant', 'dev.non_relevant', 'test.relevant' and 'test.non_relevant'\nnomiracl = datasets.load_dataset('miracl/nomiracl', language, split=f'{split}.{subset}')\n```\n\n### Baseline Accuracy on NoMIRACL non-relevant subset (test split, maximum cap of 250 per language)\n\nBaseline results (250 queries) are available within the repository under `./results/baselines/non_relevant`.\n\nAn example datapoint under `./results/baselines/non_relevant/en.test.vanilla_prompt.jsonl`\n```\n{\n \"query_id\": \"842558#0\", \n \"docids\": [\"2842207#5\", \"7004944#45\", \"3310762#14\", \"47220460#1\", \"36451733#7\", \"3310762#20\", \"4724576#4\", \"22373402#0\", \"52203230#0\", \"23126218#4\"], \n \"prompt\": \"I will give you a question and several contexts containing information about the question. [ ... ] \\n\\nOUTPUT:\\n\", \n \"template\": \"vanilla\", \n \"results\": {\"gpt-4-azure\": \"Yes, answer is present.\", \n \"llama-2-13b-chat\": \"\\nYes, answer is present in [6].\\n\\nNo answers found in the other contexts.\",\n [...]\n \"aya-101\": \"Wales\"}\n}\n```\n\n### Baseline Accuracy on NoMIRACL relevant subset (test split, maximum cap of 250 per language)\n\nBaseline results (250 queries) are available within the repository under `./results/baselines/relevant`.\n\nAn example datapoint under `./results/baselines/relevant/en.test.vanilla_prompt.jsonl`\n```\n{\n \"query_id\": \"8706103#0\", \n \"docids\": [\"42057469#2\", \"4998067#1\", \"29247933#0\", \"162619#81\", \"422315#13\", \"26790310#4\", \"41298602#18\", \"22816#16\", \"123427#61\", \"23576525#0\"], \n \"prompt\": \"I will give you a question and several contexts containing information about the question. [ ... ] \\n\\nQUESTION:\\nWhat is the course that will be discontinued as defined by the National Education Policy? [ ... ] \\n\\nOUTPUT:\\n\", \n \"template\": \"vanilla\", \n \"results\": {\"gpt-4-azure\": \"I don't know.\", \n \"llama-2-13b-chat\": \"Please answer the question based on the given contexts.\",\n [...]\n \"aya-101\": \"I don't know\"}\n}\n```\n\n## NoMIRACL Dataset Construction\n\n<img src=\"./images/NoMIRACL-Flowchart.drawio.png\" width=\"1013\" height=\"179\" />\n\nNoMIRACL is a multilingual dataset designed to evaluate LLM robustness in relevance assessment to help avoid errors in first-stage retrieval. The dataset covers 18 typologically diverse languages and includes two subsets: non-relevant and relevant.\n\n### Non-Relevant Subset (F)\n- Queries with no-known answers within the retrieved oracle passages.\n- All top-k passages manually judged as non-relevant (relevancy score = 0).\n\n### Relevant Subset (T)\n- Queries with known answers within the retrieved oracle passages.\n- At least one of the top-k passages manually judged as relevant (relevancy score = 1).\n\n## Evaluation Metrics\n\n<img src=\"./images/NoMIRACL-confusion-matrix.png\" width=\"411\" height=\"193\"/>\n\nWe conduct a robustness evaluation using a binary classification task, comparing LLM predictions against the ground truth provided in NoMIRACL. The metrics used are hallucination rate and error rate.\n\n- **Hallucination Rate:** `FP/(FP + TN)` Measures the model's tendency to hallucinate an answer when no answer is present in the non-relevant subset.\n\n- **Error Rate:** `FN/(FN + TP)` Measures the model's inaccuracy in recognizing relevant passages in the relevant subset.\n\n## :handshake: Collaboration and Acknowledgements\n\nThe NoMIRACL dataset has been made possible due to a collaborative effort of the following universities and organizations:\n\n- University of Waterloo\n- Huawei Noah's Ark Lab\n\nParts of the NoMIRACL code structure has been inspired by:\n- [https://github.com/McGill-NLP/instruct-qa](https://github.com/McGill-NLP/instruct-qa)\n\n## :scroll: Citations\n\nIf you use NoMIRACL or parts in a research paper, please cite our work as follows:\n\n```\n@article{thakur:2024,\n author = {Nandan Thakur and\n Luiz Bonifacio and\n Xinyu Zhang and\n Odunayo Ogundepo and\n Ehsan Kamalloo and\n David Alfonso{-}Hermelo and\n Xiaoguang Li and\n Qun Liu and\n Boxing Chen and\n Mehdi Rezagholizadeh and\n Jimmy Lin},\n title = {NoMIRACL: Knowing When You Don't Know for Robust Multilingual Retrieval-Augmented\n Generation},\n journal = {CoRR},\n volume = {abs/2312.11361},\n year = {2023},\n url = {https://doi.org/10.48550/arXiv.2312.11361},\n doi = {10.48550/ARXIV.2312.11361},\n eprinttype = {arXiv},\n eprint = {2312.11361},\n timestamp = {Tue, 16 Jan 2024 11:57:42 +0100},\n biburl = {https://dblp.org/rec/journals/corr/abs-2312-11361.bib},\n bibsource = {dblp computer science bibliography, https://dblp.org}\n}\n\n```\n\n---\nContact person: Nandan Thakur, [nandan.thakur@uwaterloo.co](mailto:nandan.thakur@uwaterloo.ca)\n\n> This repository contains experimental software and is published for the sole purpose of giving additional background details on the respective publication.\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "Multilingual Relevance Assessment for RAG Applications",
"version": "1.0.0",
"project_urls": {
"Homepage": "https://nomiracl.github.io",
"Repository": "https://github.com/project-miracl/nomiracl"
},
"split_keywords": [
"transformer networks",
" pytorch",
" nlp",
" deep learning",
" llm",
" hallucination",
" relevance",
" multilingual"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a1693d9a4e7deb987caf9f9fda0d1aaeb50162044d08fe165f2c0945826394e9",
"md5": "4f08d984a3da5598277812b1fefe4c94",
"sha256": "fefc7eaab7909a680deac0e1c103fd93bbc9e0a1671a8a1c1a43e6f046921d51"
},
"downloads": -1,
"filename": "nomiracl-1.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "4f08d984a3da5598277812b1fefe4c94",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 32078,
"upload_time": "2024-11-29T06:11:07",
"upload_time_iso_8601": "2024-11-29T06:11:07.132511Z",
"url": "https://files.pythonhosted.org/packages/a1/69/3d9a4e7deb987caf9f9fda0d1aaeb50162044d08fe165f2c0945826394e9/nomiracl-1.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "26d87e43a68b1da5245f657dea7e25c02d001a6b91b99ca324394f1d5eb9611f",
"md5": "519c18d643966c9b385ba2cb0385d35e",
"sha256": "ba06b1510610b52fcd4b4b48acff13f2994255e500e5d8c8ecc2f7c2104874b7"
},
"downloads": -1,
"filename": "nomiracl-1.0.0.tar.gz",
"has_sig": false,
"md5_digest": "519c18d643966c9b385ba2cb0385d35e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 30875,
"upload_time": "2024-11-29T06:11:08",
"upload_time_iso_8601": "2024-11-29T06:11:08.574028Z",
"url": "https://files.pythonhosted.org/packages/26/d8/7e43a68b1da5245f657dea7e25c02d001a6b91b99ca324394f1d5eb9611f/nomiracl-1.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-29 06:11:08",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "project-miracl",
"github_project": "nomiracl",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "nomiracl"
}
JSON
