| Name | pqm JSON |
| Version |
0.5.1
JSON |
| download |
| home_page | None |
| Summary | Implemenation of the PQMass two sample test from Lemos et al. 2024 |
| upload_time | 2024-08-20 17:25:01 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.9 |
| license | MIT License Copyright (c) [2023] [pqm authors] Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
machine learning
pytorch
statistics
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# PQMass: Probabilistic Assessment of the Quality of Generative Models using Probability Mass Estimation
[](https://github.com/Ciela-Institute/PQM/actions/workflows/ci.yml)
[](https://github.com/psf/black)
[](https://arxiv.org/abs/2402.04355)
Implementation of the PQMass two sample test from Lemos et al. 2024 [here](https://arxiv.org/abs/2402.04355)
## Install
Just do:
```
pip install pqm
```
## Usage
This is the main use case:
```python
from pqm import pqm_pvalue, pqm_chi2
import numpy as np
x_sample = np.random.normal(size = (500, 10))
y_sample = np.random.normal(size = (400, 10))
# To get pvalues from PQMass
pvalues = pqm_pvalue(x_sample, y_sample, num_refs = 100, re_tessellation = 50)
print(np.mean(pvalues), np.std(pvalues))
# To get chi^2 from PQMass
chi2_stat = pqm_chi2(x_sample, y_sample, num_refs = 100, re_tessellation = 50)
print(np.mean(chi2_stat), np.std(chi2_stat))
```
If your two samples are drawn from the same distribution, then the p-value should
be drawn from the random uniform(0,1) distribution. This means that if you get a
very small value (i.e., 1e-6), then you have failed the null hypothesis test, and
the two samples are not drawn from the same distribution.
For the chi^2 metric, given your two sets of samples, if they come from the same
distribution, the histogram of your chi² values should follow the chi² distribution.
The peak of this distribution will be at DoF - 2, and the standard deviation will
be √(2 * DoF). If your histogram shifts to the right of the expected chi² distribution,
it suggests that the samples are out of distribution. Conversely, if the histogram shifts
to the left, it indicates potential duplication or memorization (particularly relevant
for generative models).
## Developing
If you're a developer then:
```
git clone git@github.com:Ciela-Institute/PQM.git
cd PQM
git checkout -b my-new-branch
pip install -e .
```
Raw data
{
"_id": null,
"home_page": null,
"name": "pqm",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "machine learning, pytorch, statistics",
"author": null,
"author_email": "Pablo Lemos <pablo.lemos@mila.quebec>, Connor Stone <connor.stone@umontreal.ca>, Sammy Sharief <sharief2@illinois.edu>",
"download_url": "https://files.pythonhosted.org/packages/3e/b9/e458df15a308d2c2136fc2a2e1367ce2c29c681dac4d12dae98b21ea8da5/pqm-0.5.1.tar.gz",
"platform": null,
"description": "# PQMass: Probabilistic Assessment of the Quality of Generative Models using Probability Mass Estimation\n\n\n[](https://github.com/Ciela-Institute/PQM/actions/workflows/ci.yml)\n[](https://github.com/psf/black)\n\n[](https://arxiv.org/abs/2402.04355)\n\nImplementation of the PQMass two sample test from Lemos et al. 2024 [here](https://arxiv.org/abs/2402.04355)\n\n## Install\n\nJust do:\n\n```\npip install pqm\n```\n\n## Usage\n\nThis is the main use case:\n\n```python\nfrom pqm import pqm_pvalue, pqm_chi2\nimport numpy as np\n\nx_sample = np.random.normal(size = (500, 10))\ny_sample = np.random.normal(size = (400, 10))\n\n# To get pvalues from PQMass\npvalues = pqm_pvalue(x_sample, y_sample, num_refs = 100, re_tessellation = 50)\nprint(np.mean(pvalues), np.std(pvalues))\n\n# To get chi^2 from PQMass\nchi2_stat = pqm_chi2(x_sample, y_sample, num_refs = 100, re_tessellation = 50)\nprint(np.mean(chi2_stat), np.std(chi2_stat))\n```\n\nIf your two samples are drawn from the same distribution, then the p-value should\nbe drawn from the random uniform(0,1) distribution. This means that if you get a\nvery small value (i.e., 1e-6), then you have failed the null hypothesis test, and\nthe two samples are not drawn from the same distribution.\n\nFor the chi^2 metric, given your two sets of samples, if they come from the same\ndistribution, the histogram of your chi\u00b2 values should follow the chi\u00b2 distribution. \nThe peak of this distribution will be at DoF - 2, and the standard deviation will \nbe \u221a(2 * DoF). If your histogram shifts to the right of the expected chi\u00b2 distribution, \nit suggests that the samples are out of distribution. Conversely, if the histogram shifts \nto the left, it indicates potential duplication or memorization (particularly relevant \nfor generative models).\n\n\n## Developing\n\nIf you're a developer then:\n\n```\ngit clone git@github.com:Ciela-Institute/PQM.git\ncd PQM\ngit checkout -b my-new-branch\npip install -e .\n```\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) [2023] [pqm authors] Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.",
"summary": "Implemenation of the PQMass two sample test from Lemos et al. 2024",
"version": "0.5.1",
"project_urls": {
"Documentation": "https://github.com/Ciela-Institute/PQM",
"Homepage": "https://github.com/Ciela-Institute/PQM",
"Issues": "https://github.com/Ciela-Institute/PQM/issues",
"Repository": "https://github.com/Ciela-Institute/PQM"
},
"split_keywords": [
"machine learning",
" pytorch",
" statistics"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "0f1f840aa3bec85a6f4fd6f1d26945a274f42389057db7e5b79182bfd1951576",
"md5": "f59b499e0d2d9525bd2836c50ada96e9",
"sha256": "ccaba248b03d2cb6388a24fe58f680708cb82ac02b98cb0e77e2ea0cf20df7c6"
},
"downloads": -1,
"filename": "pqm-0.5.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f59b499e0d2d9525bd2836c50ada96e9",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 7312,
"upload_time": "2024-08-20T17:24:59",
"upload_time_iso_8601": "2024-08-20T17:24:59.145045Z",
"url": "https://files.pythonhosted.org/packages/0f/1f/840aa3bec85a6f4fd6f1d26945a274f42389057db7e5b79182bfd1951576/pqm-0.5.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3eb9e458df15a308d2c2136fc2a2e1367ce2c29c681dac4d12dae98b21ea8da5",
"md5": "5e0476f7c875cd86303c359e10dbfacc",
"sha256": "81fa6f4a2fcb725d79637a3b233b3c56aa1a73227bdc18aa34507fe53f7ac1d9"
},
"downloads": -1,
"filename": "pqm-0.5.1.tar.gz",
"has_sig": false,
"md5_digest": "5e0476f7c875cd86303c359e10dbfacc",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 289137,
"upload_time": "2024-08-20T17:25:01",
"upload_time_iso_8601": "2024-08-20T17:25:01.565889Z",
"url": "https://files.pythonhosted.org/packages/3e/b9/e458df15a308d2c2136fc2a2e1367ce2c29c681dac4d12dae98b21ea8da5/pqm-0.5.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-20 17:25:01",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Ciela-Institute",
"github_project": "PQM",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "pqm"
}
