<div align="center">
<img src="https://repository-images.githubusercontent.com/268892956/750228ec-f3f2-465d-9c17-420c688ba2bc">
</div>
<p align="center">
<!-- Python -->
<a href="https://www.python.org" alt="Python"><img src="https://badges.aleen42.com/src/python.svg"></a>
<!-- Version -->
<a href="https://pypi.org/project/ranx/"><img src="https://img.shields.io/pypi/v/ranx?color=light-green" alt="PyPI version"></a>
<!-- Downloads -->
<a href="https://pepy.tech/project/ranx"><img src="https://static.pepy.tech/personalized-badge/ranx?period=total&units=international_system&left_color=grey&right_color=blue&left_text=downloads" alt="Download counter"></a>
<!-- Docs -->
<a href="https://amenra.github.io/ranx"><img src="https://img.shields.io/badge/docs-passing-<COLOR>.svg" alt="Documentation Status"></a>
<!-- Black -->
<a href="https://github.com/psf/black" alt="Code style: black"><img src="https://img.shields.io/badge/code%20style-black-000000.svg"></a>
<!-- License -->
<a href="https://lbesson.mit-license.org/"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="License: MIT"></a>
<!-- Google Colab -->
<a href="https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/1_overview.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open in Colab"/></a>
</p>
## π₯ News
- [August 3 2023] `ranx` `0.3.16` is out!
This release adds support for importing Qrels and Runs from `parquet` files, exporting them as `pandas.DataFrame` and save them as `parquet` files.
Any dependence on `trec_eval` have been removed to make `ranx` truly MIT-compliant.
## β‘οΈ Introduction
[ranx](https://github.com/AmenRa/ranx) ([raΕks]) is a library of fast ranking evaluation metrics implemented in [Python](https://en.wikipedia.org/wiki/Python_(programming_language)), leveraging [Numba](https://github.com/numba/numba) for high-speed [vector operations](https://en.wikipedia.org/wiki/Automatic_vectorization) and [automatic parallelization](https://en.wikipedia.org/wiki/Automatic_parallelization).
It offers a user-friendly interface to evaluate and compare [Information Retrieval](https://en.wikipedia.org/wiki/Information_retrieval) and [Recommender Systems](https://en.wikipedia.org/wiki/Recommender_system).
[ranx](https://github.com/AmenRa/ranx) allows you to perform statistical tests and export [LaTeX](https://en.wikipedia.org/wiki/LaTeX) tables for your scientific publications.
Moreover, [ranx](https://github.com/AmenRa/ranx) provides several [fusion algorithms](https://amenra.github.io/ranx/fusion) and [normalization strategies](https://amenra.github.io/ranx/normalization), and an automatic [fusion optimization](https://amenra.github.io/ranx/fusion/#optimize-fusion) functionality.
[ranx](https://github.com/AmenRa/ranx) also have a companion repository of pre-computed runs to facilitated model comparisons called [ranxhub](https://amenra.github.io/ranxhub).
On [ranxhub](https://amenra.github.io/ranxhub), you can download and share pre-computed runs for Information Retrieval datasets, such as [MSMARCO Passage Ranking](https://arxiv.org/abs/1611.09268).
[ranx](https://github.com/AmenRa/ranx) was featured in [ECIR 2022](https://ecir2022.org), [CIKM 2022](https://www.cikm2022.org), and [SIGIR 2023](https://sigir.org/sigir2023).
If you use [ranx](https://github.com/AmenRa/ranx) to evaluate results or conducting experiments involving fusion for your scientific publication, please consider citing it: [evaluation bibtex](https://dblp.org/rec/conf/ecir/Bassani22.html?view=bibtex), [fusion bibtex](https://dblp.org/rec/conf/cikm/BassaniR22.html?view=bibtex), [ranxhub bibtex](https://dblp.org/rec/conf/sigir/Bassani23.html?view=bibtex).
NB: [ranx](https://github.com/AmenRa/ranx) is not suited for evaluating classifiers. Please, refer to the [FAQ](https://amenra.github.io/ranx/faq) for further details.
For a quick overview, follow the [Usage](#-usage) section.
For a in-depth overview, follow the [Examples](#-examples) section.
## β¨ Features
### Metrics
* [Hits](https://amenra.github.io/ranx/metrics/#hits)
* [Hit Rate](https://amenra.github.io/ranx/metrics/#hit-rate-success)
* [Precision](https://amenra.github.io/ranx/metrics/#precision)
* [Recall](https://amenra.github.io/ranx/metrics/#recall)
* [F1](https://amenra.github.io/ranx/metrics/#f1)
* [r-Precision](https://amenra.github.io/ranx/metrics/#r-precision)
* [Bpref](https://amenra.github.io/ranx/metrics/#bpref)
* [Rank-biased Precision (RBP)](https://amenra.github.io/ranx/metrics/#rank-biased-precision)
* [Mean Reciprocal Rank (MRR)](https://amenra.github.io/ranx/metrics/#mean-reciprocal-rank)
* [Mean Average Precision (MAP)](https://amenra.github.io/ranx/metrics/#mean-average-precision)
* [Discounted Cumulative Gain (DCG)](https://amenra.github.io/ranx/metrics/#dcg)
* [Normalized Discounted Cumulative Gain (NDCG)](https://amenra.github.io/ranx/metrics/#ndcg)
The metrics have been tested against [TREC Eval](https://github.com/usnistgov/trec_eval) for correctness.
### Statistical Tests
* [Paired Student's t-Test](https://www.itl.nist.gov/div898/software/dataplot/refman1/auxillar/t_test.htm) (default)
* [Fisher's Randomization Test](https://www.itl.nist.gov/div898/software/dataplot/refman1/auxillar/fishrand.htm)
* [Tukey's HSD Test](https://www.itl.nist.gov/div898/handbook/prc/section4/prc471.htm)
Please, refer to [Smucker et al.](https://dl.acm.org/doi/10.1145/1321440.1321528), [Carterette](https://dl.acm.org/doi/10.1145/2094072.2094076), and [Fuhr](http://www.sigir.org/wp-content/uploads/2018/01/p032.pdf) for additional information on statistical tests for Information Retrieval.
### Off-the-shelf Qrels
You can load qrels from [ir-datasets](https://ir-datasets.com) as simply as:
```python
qrels = Qrels.from_ir_datasets("msmarco-document/dev")
```
A full list of the available qrels is provided [here](https://ir-datasets.com).
### Off-the-shelf Runs
You can load runs from [ranxhub](https://amenra.github.io/ranxhub/) as simply as:
```python
run = Run.from_ranxhub("run-id")
```
A full list of the available runs is provided [here](https://amenra.github.io/ranxhub//browse).
### Fusion Algorithms
| **Name** | **Name** | **Name** | **Name** | **Name** |
| -------------------------------------------------------- | ---------------------------------------------------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------ |
| [CombMIN](https://amenra.github.io/ranx/fusion/#combmin) | [CombMNZ](https://amenra.github.io/ranx/fusion/#combmnz) | [RRF](https://amenra.github.io/ranx/fusion/#reciprocal-rank-fusion-rrf) | [MAPFuse](https://amenra.github.io/ranx/fusion/#mapfuse) | [BordaFuse](https://amenra.github.io/ranx/fusion/#bordafuse) |
| [CombMED](https://amenra.github.io/ranx/fusion/#combmed) | [CombGMNZ](https://amenra.github.io/ranx/fusion/#combgmnz) | [RBC](https://amenra.github.io/ranx/fusion/#rank-biased-centroids-rbc) | [PosFuse](https://amenra.github.io/ranx/fusion/#posfuse) | [Weighted BordaFuse](https://amenra.github.io/ranx/fusion/#weighted-bordafuse) |
| [CombANZ](https://amenra.github.io/ranx/fusion/#combanz) | [ISR](https://amenra.github.io/ranx/fusion/#isr) | [WMNZ](https://amenra.github.io/ranx/fusion/#wmnz) | [ProbFuse](https://amenra.github.io/ranx/fusion/#probfuse) | [Condorcet](https://amenra.github.io/ranx/fusion/#condorcet) |
| [CombMAX](https://amenra.github.io/ranx/fusion/#combmax) | [Log_ISR](https://amenra.github.io/ranx/fusion/#log_isr) | [Mixed](https://amenra.github.io/ranx/fusion/#mixed) | [SegFuse](https://amenra.github.io/ranx/fusion/#segfuse) | [Weighted Condorcet](https://amenra.github.io/ranx/fusion/#weighted-condorcet) |
| [CombSUM](https://amenra.github.io/ranx/fusion/#combsum) | [LogN_ISR](https://amenra.github.io/ranx/fusion/#logn_isr) | [BayesFuse](https://amenra.github.io/ranx/fusion/#bayesfuse) | [SlideFuse](https://amenra.github.io/ranx/fusion/#slidefuse) | [Weighted Sum](https://amenra.github.io/ranx/fusion/#wighted-sum) |
Please, refer to the [documentation](https://amenra.github.io/ranx/fusion) for further details.
### Normalization Strategies
* [Min-Max Norm](https://amenra.github.io/ranx/normalization/#min-max-norm)
* [Max Norm](https://amenra.github.io/ranx/normalization/#sum-norm)
* [Sum Norm](https://amenra.github.io/ranx/normalization/#rank-norm)
* [ZMUV Norm](https://amenra.github.io/ranx/normalization/#max-norm)
* [Rank Norm](https://amenra.github.io/ranx/normalization/#zmuv-norm)
* [Borda Norm](https://amenra.github.io/ranx/normalization/#borda-norm)
Please, refer to the [documentation](https://amenra.github.io/ranx/fusion) for further details.
## π Requirements
```bash
python>=3.8
```
As of `v.0.3.5`, [ranx](https://github.com/AmenRa/ranx) requires `python>=3.8`.
## πΎ Installation
```bash
pip install ranx
```
## π‘ Usage
### Create Qrels and Run
```python
from ranx import Qrels, Run
qrels_dict = { "q_1": { "d_12": 5, "d_25": 3 },
"q_2": { "d_11": 6, "d_22": 1 } }
run_dict = { "q_1": { "d_12": 0.9, "d_23": 0.8, "d_25": 0.7,
"d_36": 0.6, "d_32": 0.5, "d_35": 0.4 },
"q_2": { "d_12": 0.9, "d_11": 0.8, "d_25": 0.7,
"d_36": 0.6, "d_22": 0.5, "d_35": 0.4 } }
qrels = Qrels(qrels_dict)
run = Run(run_dict)
```
### Evaluate
```python
from ranx import evaluate
# Compute score for a single metric
evaluate(qrels, run, "ndcg@5")
>>> 0.7861
# Compute scores for multiple metrics at once
evaluate(qrels, run, ["map@5", "mrr"])
>>> {"map@5": 0.6416, "mrr": 0.75}
```
### Compare
```python
from ranx import compare
# Compare different runs and perform Two-sided Paired Student's t-Test
report = compare(
qrels=qrels,
runs=[run_1, run_2, run_3, run_4, run_5],
metrics=["map@100", "mrr@100", "ndcg@10"],
max_p=0.01 # P-value threshold
)
```
Output:
```python
print(report)
```
```
# Model MAP@100 MRR@100 NDCG@10
--- ------- -------- -------- ---------
a model_1 0.320α΅ 0.320α΅ 0.368α΅αΆ
b model_2 0.233 0.234 0.239
c model_3 0.308α΅ 0.309α΅ 0.330α΅
d model_4 0.366α΅α΅αΆ 0.367α΅α΅αΆ 0.408α΅α΅αΆ
e model_5 0.405α΅α΅αΆα΅ 0.406α΅α΅αΆα΅ 0.451α΅α΅αΆα΅
```
### Fusion
```python
from ranx import fuse, optimize_fusion
best_params = optimize_fusion(
qrels=train_qrels,
runs=[train_run_1, train_run_2, train_run_3],
norm="min-max", # The norm. to apply before fusion
method="wsum", # The fusion algorithm to use (Weighted Sum)
metric="ndcg@100", # The metric to maximize
)
combined_test_run = fuse(
runs=[test_run_1, test_run_2, test_run_3],
norm="min-max",
method="wsum",
params=best_params,
)
```
## π Examples
| Name | Link |
| ---------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Overview | [](https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/1_overview.ipynb) |
| Qrels and Run | [](https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/2_qrels_and_run.ipynb) |
| Evaluation | [](https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/3_evaluation.ipynb) |
| Comparison and Report | [](https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/4_comparison_and_report.ipynb) |
| Fusion | [](https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/5_fusion.ipynb) |
| Plot | [](https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/7_plot.ipynb) |
| Share your runs with [ranxhub](https://amenra.github.io/ranxhub) | [](https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/6_ranxhub.ipynb) |
## π Documentation
Browse the [documentation](https://amenra.github.io/ranx) for more details and examples.
## π Citation
If you use [ranx](https://github.com/AmenRa/ranx) to evaluate results for your scientific publication, please consider citing our [ECIR 2022](https://ecir2022.org) paper:
<details>
<summary>BibTeX</summary>
```bibtex
@inproceedings{ranx,
author = {Elias Bassani},
title = {ranx: {A} Blazing-Fast Python Library for Ranking Evaluation and Comparison},
booktitle = {{ECIR} {(2)}},
series = {Lecture Notes in Computer Science},
volume = {13186},
pages = {259--264},
publisher = {Springer},
year = {2022},
doi = {10.1007/978-3-030-99739-7\_30}
}
```
</details>
If you use the fusion functionalities provided by [ranx](https://github.com/AmenRa/ranx) for conducting the experiments of your scientific publication, please consider citing our [CIKM 2022](https://www.cikm2022.org) paper:
<details>
<summary>BibTeX</summary>
```bibtex
@inproceedings{ranx.fuse,
author = {Elias Bassani and
Luca Romelli},
title = {ranx.fuse: {A} Python Library for Metasearch},
booktitle = {{CIKM}},
pages = {4808--4812},
publisher = {{ACM}},
year = {2022},
doi = {10.1145/3511808.3557207}
}
```
</details>
If you use pre-computed runs from [ranxhub](https://amenra.github.io/ranxhub) to make comparison for your scientific publication, please consider citing our [SIGIR 2023](https://sigir.org/sigir2023) paper:
<details>
<summary>BibTeX</summary>
```bibtex
@inproceedings{ranxhub,
author = {Elias Bassani},
title = {ranxhub: An Online Repository for Information Retrieval Runs},
booktitle = {{SIGIR}},
pages = {3210--3214},
publisher = {{ACM}},
year = {2023},
doi = {10.1145/3539618.3591823}
}
```
</details>
## π Feature Requests
Would you like to see other features implemented? Please, open a [feature request](https://github.com/AmenRa/ranx/issues/new?assignees=&labels=enhancement&template=feature_request.md&title=%5BFeature+Request%5D+title).
## π€ Want to contribute?
Would you like to contribute? Please, drop me an [e-mail](mailto:elias.bssn@gmail.com?subject=[GitHub]%20ranx).
## π License
[ranx](https://github.com/AmenRa/ranx) is an open-sourced software licensed under the [MIT license](LICENSE).
Raw data
{
"_id": null,
"home_page": "https://github.com/AmenRa/ranx",
"name": "ranx",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "trec_eval,information retrieval,recommender systems,evaluation,ranking,fusion,metasearch,numba",
"author": "Elias Bassani",
"author_email": "elias.bssn@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/a0/42/89bf658aca91dc2987456eb31edc970362ee7870dba1f58ce3d4d6c3ce1b/ranx-0.3.19.tar.gz",
"platform": null,
"description": "<div align=\"center\">\n <img src=\"https://repository-images.githubusercontent.com/268892956/750228ec-f3f2-465d-9c17-420c688ba2bc\">\n</div>\n\n<p align=\"center\">\n <!-- Python -->\n <a href=\"https://www.python.org\" alt=\"Python\"><img src=\"https://badges.aleen42.com/src/python.svg\"></a>\n <!-- Version -->\n <a href=\"https://pypi.org/project/ranx/\"><img src=\"https://img.shields.io/pypi/v/ranx?color=light-green\" alt=\"PyPI version\"></a>\n <!-- Downloads -->\n <a href=\"https://pepy.tech/project/ranx\"><img src=\"https://static.pepy.tech/personalized-badge/ranx?period=total&units=international_system&left_color=grey&right_color=blue&left_text=downloads\" alt=\"Download counter\"></a>\n <!-- Docs -->\n <a href=\"https://amenra.github.io/ranx\"><img src=\"https://img.shields.io/badge/docs-passing-<COLOR>.svg\" alt=\"Documentation Status\"></a>\n <!-- Black -->\n <a href=\"https://github.com/psf/black\" alt=\"Code style: black\"><img src=\"https://img.shields.io/badge/code%20style-black-000000.svg\"></a>\n <!-- License -->\n <a href=\"https://lbesson.mit-license.org/\"><img src=\"https://img.shields.io/badge/license-MIT-blue.svg\" alt=\"License: MIT\"></a>\n <!-- Google Colab -->\n <a href=\"https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/1_overview.ipynb\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>\n</p>\n\n## \ud83d\udd25 News\n\n- [August 3 2023] `ranx` `0.3.16` is out! \nThis release adds support for importing Qrels and Runs from `parquet` files, exporting them as `pandas.DataFrame` and save them as `parquet` files.\nAny dependence on `trec_eval` have been removed to make `ranx` truly MIT-compliant.\n\n## \u26a1\ufe0f Introduction\n\n[ranx](https://github.com/AmenRa/ranx) ([ra\u014bks]) is a library of fast ranking evaluation metrics implemented in [Python](https://en.wikipedia.org/wiki/Python_(programming_language)), leveraging [Numba](https://github.com/numba/numba) for high-speed [vector operations](https://en.wikipedia.org/wiki/Automatic_vectorization) and [automatic parallelization](https://en.wikipedia.org/wiki/Automatic_parallelization).\nIt offers a user-friendly interface to evaluate and compare [Information Retrieval](https://en.wikipedia.org/wiki/Information_retrieval) and [Recommender Systems](https://en.wikipedia.org/wiki/Recommender_system).\n[ranx](https://github.com/AmenRa/ranx) allows you to perform statistical tests and export [LaTeX](https://en.wikipedia.org/wiki/LaTeX) tables for your scientific publications.\nMoreover, [ranx](https://github.com/AmenRa/ranx) provides several [fusion algorithms](https://amenra.github.io/ranx/fusion) and [normalization strategies](https://amenra.github.io/ranx/normalization), and an automatic [fusion optimization](https://amenra.github.io/ranx/fusion/#optimize-fusion) functionality.\n[ranx](https://github.com/AmenRa/ranx) also have a companion repository of pre-computed runs to facilitated model comparisons called [ranxhub](https://amenra.github.io/ranxhub).\nOn [ranxhub](https://amenra.github.io/ranxhub), you can download and share pre-computed runs for Information Retrieval datasets, such as [MSMARCO Passage Ranking](https://arxiv.org/abs/1611.09268).\n[ranx](https://github.com/AmenRa/ranx) was featured in [ECIR 2022](https://ecir2022.org), [CIKM 2022](https://www.cikm2022.org), and [SIGIR 2023](https://sigir.org/sigir2023). \n \nIf you use [ranx](https://github.com/AmenRa/ranx) to evaluate results or conducting experiments involving fusion for your scientific publication, please consider citing it: [evaluation bibtex](https://dblp.org/rec/conf/ecir/Bassani22.html?view=bibtex), [fusion bibtex](https://dblp.org/rec/conf/cikm/BassaniR22.html?view=bibtex), [ranxhub bibtex](https://dblp.org/rec/conf/sigir/Bassani23.html?view=bibtex).\n\nNB: [ranx](https://github.com/AmenRa/ranx) is not suited for evaluating classifiers. Please, refer to the [FAQ](https://amenra.github.io/ranx/faq) for further details.\n\nFor a quick overview, follow the [Usage](#-usage) section.\n\nFor a in-depth overview, follow the [Examples](#-examples) section.\n\n\n## \u2728 Features\n\n### Metrics\n* [Hits](https://amenra.github.io/ranx/metrics/#hits)\n* [Hit Rate](https://amenra.github.io/ranx/metrics/#hit-rate-success)\n* [Precision](https://amenra.github.io/ranx/metrics/#precision)\n* [Recall](https://amenra.github.io/ranx/metrics/#recall)\n* [F1](https://amenra.github.io/ranx/metrics/#f1)\n* [r-Precision](https://amenra.github.io/ranx/metrics/#r-precision)\n* [Bpref](https://amenra.github.io/ranx/metrics/#bpref)\n* [Rank-biased Precision (RBP)](https://amenra.github.io/ranx/metrics/#rank-biased-precision)\n* [Mean Reciprocal Rank (MRR)](https://amenra.github.io/ranx/metrics/#mean-reciprocal-rank)\n* [Mean Average Precision (MAP)](https://amenra.github.io/ranx/metrics/#mean-average-precision)\n* [Discounted Cumulative Gain (DCG)](https://amenra.github.io/ranx/metrics/#dcg)\n* [Normalized Discounted Cumulative Gain (NDCG)](https://amenra.github.io/ranx/metrics/#ndcg)\n\nThe metrics have been tested against [TREC Eval](https://github.com/usnistgov/trec_eval) for correctness.\n\n### Statistical Tests\n* [Paired Student's t-Test](https://www.itl.nist.gov/div898/software/dataplot/refman1/auxillar/t_test.htm) (default)\n* [Fisher's Randomization Test](https://www.itl.nist.gov/div898/software/dataplot/refman1/auxillar/fishrand.htm)\n* [Tukey's HSD Test](https://www.itl.nist.gov/div898/handbook/prc/section4/prc471.htm)\n\nPlease, refer to [Smucker et al.](https://dl.acm.org/doi/10.1145/1321440.1321528), [Carterette](https://dl.acm.org/doi/10.1145/2094072.2094076), and [Fuhr](http://www.sigir.org/wp-content/uploads/2018/01/p032.pdf) for additional information on statistical tests for Information Retrieval.\n\n### Off-the-shelf Qrels\nYou can load qrels from [ir-datasets](https://ir-datasets.com) as simply as:\n```python\nqrels = Qrels.from_ir_datasets(\"msmarco-document/dev\")\n```\nA full list of the available qrels is provided [here](https://ir-datasets.com).\n\n### Off-the-shelf Runs\nYou can load runs from [ranxhub](https://amenra.github.io/ranxhub/) as simply as:\n```python\nrun = Run.from_ranxhub(\"run-id\")\n```\nA full list of the available runs is provided [here](https://amenra.github.io/ranxhub//browse).\n\n### Fusion Algorithms\n\n| **Name** | **Name** | **Name** | **Name** | **Name** |\n| -------------------------------------------------------- | ---------------------------------------------------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------ |\n| [CombMIN](https://amenra.github.io/ranx/fusion/#combmin) | [CombMNZ](https://amenra.github.io/ranx/fusion/#combmnz) | [RRF](https://amenra.github.io/ranx/fusion/#reciprocal-rank-fusion-rrf) | [MAPFuse](https://amenra.github.io/ranx/fusion/#mapfuse) | [BordaFuse](https://amenra.github.io/ranx/fusion/#bordafuse) |\n| [CombMED](https://amenra.github.io/ranx/fusion/#combmed) | [CombGMNZ](https://amenra.github.io/ranx/fusion/#combgmnz) | [RBC](https://amenra.github.io/ranx/fusion/#rank-biased-centroids-rbc) | [PosFuse](https://amenra.github.io/ranx/fusion/#posfuse) | [Weighted BordaFuse](https://amenra.github.io/ranx/fusion/#weighted-bordafuse) |\n| [CombANZ](https://amenra.github.io/ranx/fusion/#combanz) | [ISR](https://amenra.github.io/ranx/fusion/#isr) | [WMNZ](https://amenra.github.io/ranx/fusion/#wmnz) | [ProbFuse](https://amenra.github.io/ranx/fusion/#probfuse) | [Condorcet](https://amenra.github.io/ranx/fusion/#condorcet) |\n| [CombMAX](https://amenra.github.io/ranx/fusion/#combmax) | [Log_ISR](https://amenra.github.io/ranx/fusion/#log_isr) | [Mixed](https://amenra.github.io/ranx/fusion/#mixed) | [SegFuse](https://amenra.github.io/ranx/fusion/#segfuse) | [Weighted Condorcet](https://amenra.github.io/ranx/fusion/#weighted-condorcet) |\n| [CombSUM](https://amenra.github.io/ranx/fusion/#combsum) | [LogN_ISR](https://amenra.github.io/ranx/fusion/#logn_isr) | [BayesFuse](https://amenra.github.io/ranx/fusion/#bayesfuse) | [SlideFuse](https://amenra.github.io/ranx/fusion/#slidefuse) | [Weighted Sum](https://amenra.github.io/ranx/fusion/#wighted-sum) |\n\nPlease, refer to the [documentation](https://amenra.github.io/ranx/fusion) for further details.\n\n### Normalization Strategies\n\n* [Min-Max Norm](https://amenra.github.io/ranx/normalization/#min-max-norm) \n* [Max Norm](https://amenra.github.io/ranx/normalization/#sum-norm) \n* [Sum Norm](https://amenra.github.io/ranx/normalization/#rank-norm) \n* [ZMUV Norm](https://amenra.github.io/ranx/normalization/#max-norm) \n* [Rank Norm](https://amenra.github.io/ranx/normalization/#zmuv-norm) \n* [Borda Norm](https://amenra.github.io/ranx/normalization/#borda-norm)\n\nPlease, refer to the [documentation](https://amenra.github.io/ranx/fusion) for further details.\n\n\n\n## \ud83d\udd0c Requirements\n```bash\npython>=3.8\n```\nAs of `v.0.3.5`, [ranx](https://github.com/AmenRa/ranx) requires `python>=3.8`.\n\n## \ud83d\udcbe Installation \n\n```bash\npip install ranx\n```\n\n## \ud83d\udca1 Usage\n\n### Create Qrels and Run\n```python\nfrom ranx import Qrels, Run\n\nqrels_dict = { \"q_1\": { \"d_12\": 5, \"d_25\": 3 },\n \"q_2\": { \"d_11\": 6, \"d_22\": 1 } }\n\nrun_dict = { \"q_1\": { \"d_12\": 0.9, \"d_23\": 0.8, \"d_25\": 0.7,\n \"d_36\": 0.6, \"d_32\": 0.5, \"d_35\": 0.4 },\n \"q_2\": { \"d_12\": 0.9, \"d_11\": 0.8, \"d_25\": 0.7,\n \"d_36\": 0.6, \"d_22\": 0.5, \"d_35\": 0.4 } }\n\nqrels = Qrels(qrels_dict)\nrun = Run(run_dict)\n```\n\n### Evaluate\n```python\nfrom ranx import evaluate\n\n# Compute score for a single metric\nevaluate(qrels, run, \"ndcg@5\")\n>>> 0.7861\n\n# Compute scores for multiple metrics at once\nevaluate(qrels, run, [\"map@5\", \"mrr\"])\n>>> {\"map@5\": 0.6416, \"mrr\": 0.75}\n```\n\n### Compare\n```python\nfrom ranx import compare\n\n# Compare different runs and perform Two-sided Paired Student's t-Test\nreport = compare(\n qrels=qrels,\n runs=[run_1, run_2, run_3, run_4, run_5],\n metrics=[\"map@100\", \"mrr@100\", \"ndcg@10\"],\n max_p=0.01 # P-value threshold\n)\n```\nOutput:\n```python\nprint(report)\n```\n```\n# Model MAP@100 MRR@100 NDCG@10\n--- ------- -------- -------- ---------\na model_1 0.320\u1d47 0.320\u1d47 0.368\u1d47\u1d9c\nb model_2 0.233 0.234 0.239\nc model_3 0.308\u1d47 0.309\u1d47 0.330\u1d47\nd model_4 0.366\u1d43\u1d47\u1d9c 0.367\u1d43\u1d47\u1d9c 0.408\u1d43\u1d47\u1d9c\ne model_5 0.405\u1d43\u1d47\u1d9c\u1d48 0.406\u1d43\u1d47\u1d9c\u1d48 0.451\u1d43\u1d47\u1d9c\u1d48\n```\n\n### Fusion\n```python\nfrom ranx import fuse, optimize_fusion\n\nbest_params = optimize_fusion(\n qrels=train_qrels,\n runs=[train_run_1, train_run_2, train_run_3],\n norm=\"min-max\", # The norm. to apply before fusion\n method=\"wsum\", # The fusion algorithm to use (Weighted Sum)\n metric=\"ndcg@100\", # The metric to maximize\n)\n\ncombined_test_run = fuse(\n runs=[test_run_1, test_run_2, test_run_3], \n norm=\"min-max\", \n method=\"wsum\", \n params=best_params,\n)\n```\n\n## \ud83d\udcd6 Examples\n\n| Name | Link |\n| ---------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| Overview | [](https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/1_overview.ipynb) |\n| Qrels and Run | [](https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/2_qrels_and_run.ipynb) |\n| Evaluation | [](https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/3_evaluation.ipynb) |\n| Comparison and Report | [](https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/4_comparison_and_report.ipynb) |\n| Fusion | [](https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/5_fusion.ipynb) |\n| Plot | [](https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/7_plot.ipynb) |\n| Share your runs with [ranxhub](https://amenra.github.io/ranxhub) | [](https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/6_ranxhub.ipynb) |\n\n\n## \ud83d\udcda Documentation\nBrowse the [documentation](https://amenra.github.io/ranx) for more details and examples.\n\n\n## \ud83c\udf93 Citation\nIf you use [ranx](https://github.com/AmenRa/ranx) to evaluate results for your scientific publication, please consider citing our [ECIR 2022](https://ecir2022.org) paper:\n<details>\n <summary>BibTeX</summary>\n \n ```bibtex\n @inproceedings{ranx,\n author = {Elias Bassani},\n title = {ranx: {A} Blazing-Fast Python Library for Ranking Evaluation and Comparison},\n booktitle = {{ECIR} {(2)}},\n series = {Lecture Notes in Computer Science},\n volume = {13186},\n pages = {259--264},\n publisher = {Springer},\n year = {2022},\n doi = {10.1007/978-3-030-99739-7\\_30}\n }\n ```\n</details> \n\nIf you use the fusion functionalities provided by [ranx](https://github.com/AmenRa/ranx) for conducting the experiments of your scientific publication, please consider citing our [CIKM 2022](https://www.cikm2022.org) paper:\n<details>\n <summary>BibTeX</summary>\n \n ```bibtex\n @inproceedings{ranx.fuse,\n author = {Elias Bassani and\n Luca Romelli},\n title = {ranx.fuse: {A} Python Library for Metasearch},\n booktitle = {{CIKM}},\n pages = {4808--4812},\n publisher = {{ACM}},\n year = {2022},\n doi = {10.1145/3511808.3557207}\n }\n ```\n</details>\n\nIf you use pre-computed runs from [ranxhub](https://amenra.github.io/ranxhub) to make comparison for your scientific publication, please consider citing our [SIGIR 2023](https://sigir.org/sigir2023) paper:\n<details>\n <summary>BibTeX</summary>\n \n ```bibtex\n @inproceedings{ranxhub,\n author = {Elias Bassani},\n title = {ranxhub: An Online Repository for Information Retrieval Runs},\n booktitle = {{SIGIR}},\n pages = {3210--3214},\n publisher = {{ACM}},\n year = {2023},\n doi = {10.1145/3539618.3591823}\n }\n ```\n</details> \n\n## \ud83c\udf81 Feature Requests\nWould you like to see other features implemented? Please, open a [feature request](https://github.com/AmenRa/ranx/issues/new?assignees=&labels=enhancement&template=feature_request.md&title=%5BFeature+Request%5D+title).\n\n\n## \ud83e\udd18 Want to contribute?\nWould you like to contribute? Please, drop me an [e-mail](mailto:elias.bssn@gmail.com?subject=[GitHub]%20ranx).\n\n\n## \ud83d\udcc4 License\n[ranx](https://github.com/AmenRa/ranx) is an open-sourced software licensed under the [MIT license](LICENSE).\n",
"bugtrack_url": null,
"license": "",
"summary": "ranx: A Blazing-Fast Python Library for Ranking Evaluation, Comparison, and Fusion",
"version": "0.3.19",
"project_urls": {
"Homepage": "https://github.com/AmenRa/ranx"
},
"split_keywords": [
"trec_eval",
"information retrieval",
"recommender systems",
"evaluation",
"ranking",
"fusion",
"metasearch",
"numba"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "38878159a9a75db6f5774973050c62f1d2e11d3458a02640bdac31cc78f6bb11",
"md5": "7cc6d8d378111cc45f90a185480d690f",
"sha256": "d58de9a6c0b70e3b61a49c6775ba4ceebc01f92740e50212bed2079f673fc2e0"
},
"downloads": -1,
"filename": "ranx-0.3.19-py3-none-any.whl",
"has_sig": false,
"md5_digest": "7cc6d8d378111cc45f90a185480d690f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 99177,
"upload_time": "2023-11-28T08:19:37",
"upload_time_iso_8601": "2023-11-28T08:19:37.246957Z",
"url": "https://files.pythonhosted.org/packages/38/87/8159a9a75db6f5774973050c62f1d2e11d3458a02640bdac31cc78f6bb11/ranx-0.3.19-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a04289bf658aca91dc2987456eb31edc970362ee7870dba1f58ce3d4d6c3ce1b",
"md5": "1ddc8dd4a3df06f4b9bcb6f18c63ebf6",
"sha256": "629fdb4354e4f26923848ad4f702d6f07b06fb9dcba34f6913272117e2a9bfbd"
},
"downloads": -1,
"filename": "ranx-0.3.19.tar.gz",
"has_sig": false,
"md5_digest": "1ddc8dd4a3df06f4b9bcb6f18c63ebf6",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 51614,
"upload_time": "2023-11-28T08:19:40",
"upload_time_iso_8601": "2023-11-28T08:19:40.612418Z",
"url": "https://files.pythonhosted.org/packages/a0/42/89bf658aca91dc2987456eb31edc970362ee7870dba1f58ce3d4d6c3ce1b/ranx-0.3.19.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-28 08:19:40",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "AmenRa",
"github_project": "ranx",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "ranx"
}
JSON
