| Name | pokie-score JSON |
| Version |
0.1.7.dev0
JSON |
| download |
| home_page | None |
| Summary | Implemenation of the Pokie from Sharief et al. 2025 |
| upload_time | 2025-07-15 00:48:23 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.9 |
| license | MIT License
Copyright (c) [2025] [pokie authors]
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE. |
| keywords |
machine learning
pytorch
statistics
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
tqdm
torch
scipy
numpy
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Pokie: Posterior Accuracy and Model Comparison
Pokie is a Python package for evaluating the calibration and accuracy of posterior distributions through a sample-based, likelihood-free method. It enables Bayesian model comparison in simulation-based settings where the true posterior is unknown and standard metrics fail.
## How Pokie Works
1. Select a Ground Truth
- Pick a truth sample $y^\*_{j} \sim p(y | x^\*, \mathcal{M}^\*)$
2. Draw Posterior Samples
- Draw N samples $\{ y_{i,j} \}_{i=1}^N \sim p(y | x^*, \mathcal{M})$
3. Repeat for Lr Random Regions:
- Sample a random center $c_{j,\ell} \in \mathbb{R}^q$
- Select a random posterior sample to define a radius:
$r_{j,\ell} = \| c_{j,\ell} - y_{i,j} \|$
- Define a region $\mathcal{R}{j,\ell}$ as the hypersphere around $c_{j,\ell}$ with radius $r_{j,\ell}$
4. Count Points
- Count how many posterior samples fall into the region: $n = \sum_i \mathbf{1}[y_{i,j} \in \mathcal{R}_{j,\ell}]$
- Check if the ground-truth sample falls inside: $k = \mathbf{1}[y^*j \in \mathcal{R}_{j,\ell}]$
5. Update Score
```
if k == 1:
score += (n + 1) / (N + 2)
else:
score += (N - n + 1) / (N + 2)
```
6. Final Pokie Score:
$P_{\text{Pokie}}(\mathcal{M}) = \frac{\texttt{score}}{L \cdot L_r}$
## Interpretation
- Well-calibrated posterior → Pokie score → ≈ 2/3
- Misaligned posterior → Pokie score → ≈ 1/2
## Example Usage
```
import torch
from pokie import pokie
# Ground truth parameters (T samples in q-dim)
truth = torch.randn(T, q)
# Posterior samples from M models (M, T, S, q)
posterior = torch.randn(M, T, S, q)
# Run Pokie
score = pokie(truth, posterior, num_runs=100)
print(score) # Pokie scores for each model
```
## Developing
If you're a developer then:
```python
git clone git@github.com:SammyS15/Pokie.git
cd Pokie
git checkout -b my-new-branch
pip install -e .
```
But make an issue first so we can discuss implementation ideas.
Raw data
{
"_id": null,
"home_page": null,
"name": "pokie-score",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "machine learning, pytorch, statistics",
"author": null,
"author_email": "Sammy Sharief <sammybassoon1516@hotmail.com>, Justine Zeghal <zghjustinephd@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/cd/55/8dc03b1899c216b45ef6b584acd00ee692f77feb93652da12f869d5dd250/pokie_score-0.1.7.dev0.tar.gz",
"platform": null,
"description": "# Pokie: Posterior Accuracy and Model Comparison\n\nPokie is a Python package for evaluating the calibration and accuracy of posterior distributions through a sample-based, likelihood-free method. It enables Bayesian model comparison in simulation-based settings where the true posterior is unknown and standard metrics fail.\n\n## How Pokie Works\n\n1.\tSelect a Ground Truth\n- Pick a truth sample $y^\\*_{j} \\sim p(y | x^\\*, \\mathcal{M}^\\*)$\n2.\tDraw Posterior Samples\n- Draw N samples $\\{ y_{i,j} \\}_{i=1}^N \\sim p(y | x^*, \\mathcal{M})$\n3.\tRepeat for Lr Random Regions:\n- Sample a random center $c_{j,\\ell} \\in \\mathbb{R}^q$\n- Select a random posterior sample to define a radius:\n$r_{j,\\ell} = \\| c_{j,\\ell} - y_{i,j} \\|$\n- Define a region $\\mathcal{R}{j,\\ell}$ as the hypersphere around $c_{j,\\ell}$ with radius $r_{j,\\ell}$\n4.\tCount Points\n- Count how many posterior samples fall into the region: $n = \\sum_i \\mathbf{1}[y_{i,j} \\in \\mathcal{R}_{j,\\ell}]$\n- Check if the ground-truth sample falls inside: $k = \\mathbf{1}[y^*j \\in \\mathcal{R}_{j,\\ell}]$\n5.\tUpdate Score\n```\nif k == 1:\n score += (n + 1) / (N + 2)\nelse:\n score += (N - n + 1) / (N + 2)\n```\n6.\tFinal Pokie Score:\n$P_{\\text{Pokie}}(\\mathcal{M}) = \\frac{\\texttt{score}}{L \\cdot L_r}$\n\n\n## Interpretation\n- Well-calibrated posterior \u2192 Pokie score \u2192 \u2248 2/3\n- Misaligned posterior \u2192 Pokie score \u2192 \u2248 1/2\n\n## Example Usage\n```\nimport torch\nfrom pokie import pokie\n\n# Ground truth parameters (T samples in q-dim)\ntruth = torch.randn(T, q)\n\n# Posterior samples from M models (M, T, S, q)\nposterior = torch.randn(M, T, S, q)\n\n# Run Pokie\nscore = pokie(truth, posterior, num_runs=100)\nprint(score) # Pokie scores for each model\n```\n\n## Developing\n\nIf you're a developer then:\n\n```python\ngit clone git@github.com:SammyS15/Pokie.git\ncd Pokie\ngit checkout -b my-new-branch\npip install -e .\n```\n\nBut make an issue first so we can discuss implementation ideas.\n",
"bugtrack_url": null,
"license": "MIT License\n \n Copyright (c) [2025] [pokie authors]\n \n Permission is hereby granted, free of charge, to any person obtaining a copy\n of this software and associated documentation files (the \"Software\"), to deal\n in the Software without restriction, including without limitation the rights\n to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n copies of the Software, and to permit persons to whom the Software is\n furnished to do so, subject to the following conditions:\n \n The above copyright notice and this permission notice shall be included in all\n copies or substantial portions of the Software.\n \n THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n SOFTWARE.",
"summary": "Implemenation of the Pokie from Sharief et al. 2025",
"version": "0.1.7.dev0",
"project_urls": {
"Documentation": "https://github.com/SammyS15/Pokie",
"Homepage": "https://github.com/SammyS15/Pokie",
"Issues": "https://github.com/SammyS15/Pokie/issues",
"Repository": "https://github.com/SammyS15/Pokie"
},
"split_keywords": [
"machine learning",
" pytorch",
" statistics"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "7393cae246f9828d0db8eefa334b95714b9a03b1c0805dc0ce2f091ce0a46945",
"md5": "3c644c2eceef1371806fef86b33945a6",
"sha256": "9f3453f8e80ca5bcb6cd1a8315e58a7a3bf01ce1ba4bcef4e5c7a34dfa5dbf49"
},
"downloads": -1,
"filename": "pokie_score-0.1.7.dev0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3c644c2eceef1371806fef86b33945a6",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 6004,
"upload_time": "2025-07-15T00:48:22",
"upload_time_iso_8601": "2025-07-15T00:48:22.162465Z",
"url": "https://files.pythonhosted.org/packages/73/93/cae246f9828d0db8eefa334b95714b9a03b1c0805dc0ce2f091ce0a46945/pokie_score-0.1.7.dev0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "cd558dc03b1899c216b45ef6b584acd00ee692f77feb93652da12f869d5dd250",
"md5": "3b5dfaf8c96c674590b59851482a825b",
"sha256": "341fdf8eb7d25713d2ef7832a64366cd854819a6ab416a326514ef00165c9510"
},
"downloads": -1,
"filename": "pokie_score-0.1.7.dev0.tar.gz",
"has_sig": false,
"md5_digest": "3b5dfaf8c96c674590b59851482a825b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 8463,
"upload_time": "2025-07-15T00:48:23",
"upload_time_iso_8601": "2025-07-15T00:48:23.166946Z",
"url": "https://files.pythonhosted.org/packages/cd/55/8dc03b1899c216b45ef6b584acd00ee692f77feb93652da12f869d5dd250/pokie_score-0.1.7.dev0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-15 00:48:23",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "SammyS15",
"github_project": "Pokie",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "tqdm",
"specs": []
},
{
"name": "torch",
"specs": []
},
{
"name": "scipy",
"specs": []
},
{
"name": "numpy",
"specs": []
}
],
"lcname": "pokie-score"
}
