# scorio
`scorio` implements the Bayes@N framework introduced in [Don't Pass@k: A Bayesian Framework for Large Language Model Evaluation](https://arxiv.org/abs/2504.11651)
[](https://arxiv.org/abs/2504.11651)
[](https://pypi.org/project/scorio/)
[](https://pypi.org/project/scorio/)
[](#license)
---
## Installation
```bash
pip install scorio
```
Requires Python 3.9–3.13 and NumPy.
## Data and shape conventions
- Categories: encode outcomes per trial as integers in `{0, ..., C}`.
- Weights: choose rubric weights `w` of length `C+1` (e.g., `[0, 1]` for binary R).
- Shapes: `R` is `M x N`, `R0` is `M x D` (if provided); both must share the same `M` and category set.
## APIs
- `scorio.eval.bayes(R, w, R0=None) -> (mu: float, sigma: float)`
- `R`: `M x N` int array with entries in `{0, ..., C}`
- `w`: length `C+1` float array of rubric weights
- `R0` (optional): `M x D` int array of prior outcomes (same category set as `R`)
- Returns posterior estimate `mu` of the rubric-weighted performance and its uncertainty `sigma`.
- `scorio.eval.avg(R) -> float`
- Returns the naive mean of elements in `R`. For binary accuracy, encode incorrect=0, correct=1.
## How to use
```python
import numpy as np
from scorio.eval import bayes
# Outcomes R: shape (M, N) with integer categories in {0, ..., C}
R = np.array([
[0, 1, 2, 2, 1], # Item 1, N=5 trials
[1, 1, 0, 2, 2], # Item 2, N=5 trials
])
# Rubric weights w: length C+1. Here: 0=incorrect, 1=partial(0.5), 2=correct(1.0)
w = np.array([0.0, 0.5, 1.0])
# Optional prior outcomes R0: shape (M, D). If omitted, D=0.
R0 = np.array([
[0, 2],
[1, 2],
])
# With prior (D=2 → T=10)
mu, sigma = bayes(R, w, R0)
print(mu, sigma) # expected ~ (0.575, 0.084275)
# Without prior (D=0 → T=8)
mu2, sigma2 = bayes(R, w)
print(mu2, sigma2) # expected ~ (0.5625, 0.091998)
```
## Citing
If you use `scorio` or Bayes@N, please cite:
```
@article{hariri2025dontpassk,
title = {Don't Pass@k: A Bayesian Framework for Large Language Model Evaluation},
author = {Hariri, Mohsen and Samandar, Amirhossein and Hinczewski, Michael and Chaudhary, Vipin},
journal={arXiv preprint arXiv:2504.11651},
year = {2025},
url = {https://mohsenhariri.github.io/bayes-kit/}
}
```
## License
MIT License. See the `LICENSE` file for details.
## Support
- Documentation and updates: https://mohsenhariri.github.io/bayes-kit/
- Issues and feature requests: https://github.com/mohsenhariri/bayes-kit/issues
Raw data
{
"_id": null,
"home_page": null,
"name": "scorio",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.14,>=3.9",
"maintainer_email": null,
"keywords": "bayesian, statistics, ranking, evaluation, machine learning, large language models",
"author": null,
"author_email": "Mohsen Hariri <mohsen.hariri@case.edu>, Amirhossein Samandar <amirhossein.samandar@case.edu>",
"download_url": "https://files.pythonhosted.org/packages/ef/92/557ea6c329d727eceeb9909a7fb0af23e208210986c6aa08feddfed86855/scorio-0.0.1.tar.gz",
"platform": null,
"description": "# scorio\n\n`scorio` implements the Bayes@N framework introduced in [Don't Pass@k: A Bayesian Framework for Large Language Model Evaluation](https://arxiv.org/abs/2504.11651)\n\n[](https://arxiv.org/abs/2504.11651)\n[](https://pypi.org/project/scorio/)\n[](https://pypi.org/project/scorio/)\n[](#license)\n\n---\n\n## Installation\n\n```bash\npip install scorio\n```\n\nRequires Python 3.9\u20133.13 and NumPy.\n\n## Data and shape conventions\n\n- Categories: encode outcomes per trial as integers in `{0, ..., C}`.\n- Weights: choose rubric weights `w` of length `C+1` (e.g., `[0, 1]` for binary R).\n- Shapes: `R` is `M x N`, `R0` is `M x D` (if provided); both must share the same `M` and category set.\n\n## APIs\n\n- `scorio.eval.bayes(R, w, R0=None) -> (mu: float, sigma: float)`\n - `R`: `M x N` int array with entries in `{0, ..., C}`\n - `w`: length `C+1` float array of rubric weights\n - `R0` (optional): `M x D` int array of prior outcomes (same category set as `R`)\n - Returns posterior estimate `mu` of the rubric-weighted performance and its uncertainty `sigma`.\n\n- `scorio.eval.avg(R) -> float`\n - Returns the naive mean of elements in `R`. For binary accuracy, encode incorrect=0, correct=1.\n\n\n## How to use\n\n```python\n\nimport numpy as np\nfrom scorio.eval import bayes\n\n# Outcomes R: shape (M, N) with integer categories in {0, ..., C}\nR = np.array([\n [0, 1, 2, 2, 1], # Item 1, N=5 trials\n [1, 1, 0, 2, 2], # Item 2, N=5 trials\n])\n\n# Rubric weights w: length C+1. Here: 0=incorrect, 1=partial(0.5), 2=correct(1.0)\nw = np.array([0.0, 0.5, 1.0])\n\n# Optional prior outcomes R0: shape (M, D). If omitted, D=0.\nR0 = np.array([\n [0, 2],\n [1, 2],\n])\n\n# With prior (D=2 \u2192 T=10)\nmu, sigma = bayes(R, w, R0)\nprint(mu, sigma) # expected ~ (0.575, 0.084275)\n\n# Without prior (D=0 \u2192 T=8)\nmu2, sigma2 = bayes(R, w)\nprint(mu2, sigma2) # expected ~ (0.5625, 0.091998)\n\n```\n\n\n## Citing\n\nIf you use `scorio` or Bayes@N, please cite:\n\n```\n@article{hariri2025dontpassk,\n title = {Don't Pass@k: A Bayesian Framework for Large Language Model Evaluation},\n author = {Hariri, Mohsen and Samandar, Amirhossein and Hinczewski, Michael and Chaudhary, Vipin},\n journal={arXiv preprint arXiv:2504.11651},\n year = {2025},\n url = {https://mohsenhariri.github.io/bayes-kit/}\n}\n```\n\n\n## License\n\nMIT License. See the `LICENSE` file for details.\n\n\n## Support\n\n- Documentation and updates: https://mohsenhariri.github.io/bayes-kit/\n- Issues and feature requests: https://github.com/mohsenhariri/bayes-kit/issues\n",
"bugtrack_url": null,
"license": null,
"summary": "Bayesian evaluation and ranking toolkit",
"version": "0.0.1",
"project_urls": {
"Documentation": "https://mohsenhariri.github.io/bayes-kit/",
"Homepage": "https://mohsenhariri.github.io/bayes-kit/",
"Issues": "https://github.com/mohsenhariri/bayes-kit/issues",
"Repository": "https://github.com/mohsenhariri/bayes-kit"
},
"split_keywords": [
"bayesian",
" statistics",
" ranking",
" evaluation",
" machine learning",
" large language models"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "19fec0a91d6f2df80183af9cf90a6010c8e573c9e34aec87c0f0fdb3dc4893cf",
"md5": "96e39717804dd9d7da0b60e00eea4f08",
"sha256": "1d6bfafdfe5bc3b242587934ebe3416ee3d34fb8fec80de384c39e3d00484f25"
},
"downloads": -1,
"filename": "scorio-0.0.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "96e39717804dd9d7da0b60e00eea4f08",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.14,>=3.9",
"size": 6388,
"upload_time": "2025-10-06T23:28:38",
"upload_time_iso_8601": "2025-10-06T23:28:38.816677Z",
"url": "https://files.pythonhosted.org/packages/19/fe/c0a91d6f2df80183af9cf90a6010c8e573c9e34aec87c0f0fdb3dc4893cf/scorio-0.0.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "ef92557ea6c329d727eceeb9909a7fb0af23e208210986c6aa08feddfed86855",
"md5": "b46918679746b6d8d51d79e6c6c03bd9",
"sha256": "0ccbe9801321520422a2d88254406b8bbc3fc8c0b1788ec89cf955c735b56a44"
},
"downloads": -1,
"filename": "scorio-0.0.1.tar.gz",
"has_sig": false,
"md5_digest": "b46918679746b6d8d51d79e6c6c03bd9",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.14,>=3.9",
"size": 5916,
"upload_time": "2025-10-06T23:28:40",
"upload_time_iso_8601": "2025-10-06T23:28:40.125273Z",
"url": "https://files.pythonhosted.org/packages/ef/92/557ea6c329d727eceeb9909a7fb0af23e208210986c6aa08feddfed86855/scorio-0.0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-10-06 23:28:40",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mohsenhariri",
"github_project": "bayes-kit",
"github_not_found": true,
"lcname": "scorio"
}
JSON
