# evalify
<p align="center">
<img src="https://user-images.githubusercontent.com/7144929/154332210-fa1fee34-faae-4567-858a-49fa53e99a2b.svg" width="292" height="120" alt="Logo"/>
</p>
<p align="center">
<a href="https://github.com/ma7555/evalify/blob/main/LICENSE">
<img src="https://img.shields.io/github/license/ma7555/evalify"
alt = "License">
</a>
<a href="https://doi.org/10.5281/zenodo.6181723"><img src="https://zenodo.org/badge/DOI/10.5281/zenodo.6181723.svg" alt="DOI"></a>
<a href="https://www.python.org/downloads/">
<img src="https://img.shields.io/badge/python-3.9 | 3.10 | 3.11 | 3.12-blue.svg"
alt = "Python 3.7 | 3.8 | 3.9 | 3">
</a>
<a href="https://pypi.python.org/pypi/evalify">
<img src="https://img.shields.io/pypi/v/evalify.svg"
alt = "Release Status">
</a>
<a href="https://github.com/ma7555/evalify/actions">
<img src="https://github.com/ma7555/evalify/actions/workflows/dev.yml/badge.svg?branch=main" alt="CI Status">
</a>
<a href="https://ma7555.github.io/evalify/">
<img src="https://img.shields.io/website/https/ma7555.github.io/evalify/index.html.svg?label=docs&down_message=unavailable&up_message=available" alt="Documentation Status">
</a>
<a href="https://github.com/astral-sh/ruff">
<img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json" alt="Code style: Ruff">
</a>
<a href="https://codecov.io/gh/ma7555/evalify">
<img src="https://codecov.io/gh/ma7555/evalify/branch/main/graph/badge.svg" />
</a>
<a href="https://pypi.org/project/evalify/"><img alt="PyPI Downloads/Month" src="https://img.shields.io/pypi/dm/evalify">
</a>
</p>
**Evaluate Biometric Authentication Models Literally in Seconds.**
## Installation
#### Stable release:
```bash
pip install evalify
```
#### Bleeding edge:
```bash
pip install git+https://github.com/ma7555/evalify.git
```
## Used for
Evaluating all biometric authentication models, where the model output is a high-level embeddings known as feature vectors for visual or behaviour biometrics or d-vectors for auditory biometrics.
## Usage
```python
import numpy as np
from evalify import Experiment
rng = np.random.default_rng()
nphotos = 500
emb_size = 32
nclasses = 10
X = rng.random((self.nphotos, self.emb_size))
y = rng.integers(self.nclasses, size=self.nphotos)
experiment = Experiment()
experiment.run(X, y)
experiment.get_roc_auc()
print(experiment.roc_auc)
print(experiment.find_threshold_at_fpr(0.01))
```
## How it works
* When you run an experiment, evalify tries all the possible combinations between individuals for authentication based on the `X` and `y` parameters and returns the results including FPR, TPR, FNR, TNR and ROC AUC. `X` is an array of embeddings and `y` is an array of corresponding targets.
* Evalify can find the optimal threshold based on your agreed FPR and desired similarity or distance metric.
## Documentation:
* <https://ma7555.github.io/evalify/>
## Features
* Blazing fast implementation for metrics calculation through optimized einstein sum and vectorized calculations.
* Many operations are dispatched to canonical BLAS, cuBLAS, or other specialized routines.
* Smart sampling options using direct indexing from pre-calculated arrays with total control over sampling strategy and sampling numbers.
* Supports most evaluation metrics:
- `cosine_similarity`
- `pearson_similarity`
- `cosine_distance`
- `euclidean_distance`
- `euclidean_distance_l2`
- `minkowski_distance`
- `manhattan_distance`
- `chebyshev_distance`
* Computation time for 4 metrics 4.2 million samples experiment is **24 seconds vs 51 minutes** if looping using `scipy.spatial.distance` implemntations.
## TODO
* Safer memory allocation. I did not have issues but if you ran out of memory please manually set the `batch_size` argument.
## Contribution
* Contributions are welcomed, and they are greatly appreciated! Every little bit helps, and credit will always be given.
* Please check [CONTRIBUTING.md](https://github.com/ma7555/evalify/blob/main/CONTRIBUTING.md) for guidelines.
## Citation
* If you use this software, please cite it using the metadata from [CITATION.cff](https://github.com/ma7555/evalify/blob/main/CITATION.cff)
Raw data
{
"_id": null,
"home_page": "https://github.com/ma7555/evalify",
"name": "evalify",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.9",
"maintainer_email": null,
"keywords": "biometric verification, biometric authentication, evaluation",
"author": "Mahmoud Bahaa",
"author_email": "evalify@ma7555.anonaddy.com",
"download_url": "https://files.pythonhosted.org/packages/d8/1d/dc1df17f835a116950bc29d94885a7cac859242bd70ab0e27666a07cb4fc/evalify-1.0.0.tar.gz",
"platform": null,
"description": "# evalify\n\n<p align=\"center\">\n\n<img src=\"https://user-images.githubusercontent.com/7144929/154332210-fa1fee34-faae-4567-858a-49fa53e99a2b.svg\" width=\"292\" height=\"120\" alt=\"Logo\"/>\n\n</p>\n\n<p align=\"center\">\n\n<a href=\"https://github.com/ma7555/evalify/blob/main/LICENSE\">\n <img src=\"https://img.shields.io/github/license/ma7555/evalify\"\n alt = \"License\">\n</a>\n<a href=\"https://doi.org/10.5281/zenodo.6181723\"><img src=\"https://zenodo.org/badge/DOI/10.5281/zenodo.6181723.svg\" alt=\"DOI\"></a>\n<a href=\"https://www.python.org/downloads/\">\n <img src=\"https://img.shields.io/badge/python-3.9 | 3.10 | 3.11 | 3.12-blue.svg\"\n alt = \"Python 3.7 | 3.8 | 3.9 | 3\">\n</a>\n<a href=\"https://pypi.python.org/pypi/evalify\">\n <img src=\"https://img.shields.io/pypi/v/evalify.svg\"\n alt = \"Release Status\">\n</a>\n<a href=\"https://github.com/ma7555/evalify/actions\">\n <img src=\"https://github.com/ma7555/evalify/actions/workflows/dev.yml/badge.svg?branch=main\" alt=\"CI Status\">\n</a>\n<a href=\"https://ma7555.github.io/evalify/\">\n <img src=\"https://img.shields.io/website/https/ma7555.github.io/evalify/index.html.svg?label=docs&down_message=unavailable&up_message=available\" alt=\"Documentation Status\">\n</a>\n<a href=\"https://github.com/astral-sh/ruff\">\n <img src=\"https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json\" alt=\"Code style: Ruff\">\n</a>\n\n<a href=\"https://codecov.io/gh/ma7555/evalify\">\n <img src=\"https://codecov.io/gh/ma7555/evalify/branch/main/graph/badge.svg\" />\n</a>\n<a href=\"https://pypi.org/project/evalify/\"><img alt=\"PyPI Downloads/Month\" src=\"https://img.shields.io/pypi/dm/evalify\">\n</a>\n\n</p>\n\n**Evaluate Biometric Authentication Models Literally in\u00a0Seconds.**\n\n## Installation\n#### Stable release:\n```bash\npip install evalify\n```\n#### Bleeding edge:\n```bash\npip install git+https://github.com/ma7555/evalify.git\n```\n## Used for\nEvaluating all biometric authentication models, where the model output is a high-level embeddings known as feature vectors for visual or behaviour biometrics or d-vectors for auditory biometrics.\n\n## Usage\n\n```python\nimport numpy as np\nfrom evalify import Experiment\n\nrng = np.random.default_rng()\nnphotos = 500\nemb_size = 32\nnclasses = 10\nX = rng.random((self.nphotos, self.emb_size))\ny = rng.integers(self.nclasses, size=self.nphotos)\n\nexperiment = Experiment()\nexperiment.run(X, y)\nexperiment.get_roc_auc()\nprint(experiment.roc_auc)\nprint(experiment.find_threshold_at_fpr(0.01))\n```\n## How it works\n* When you run an experiment, evalify tries all the possible combinations between individuals for authentication based on the `X` and `y` parameters and returns the results including FPR, TPR, FNR, TNR and ROC AUC. `X` is an array of embeddings and `y` is an array of corresponding targets.\n* Evalify can find the optimal threshold based on your agreed FPR and desired similarity or distance metric.\n\n## Documentation: \n* <https://ma7555.github.io/evalify/>\n\n\n## Features\n\n* Blazing fast implementation for metrics calculation through optimized einstein sum and vectorized calculations.\n* Many operations are dispatched to canonical BLAS, cuBLAS, or other specialized routines.\n* Smart sampling options using direct indexing from pre-calculated arrays with total control over sampling strategy and sampling numbers.\n* Supports most evaluation metrics:\n - `cosine_similarity`\n - `pearson_similarity`\n - `cosine_distance`\n - `euclidean_distance`\n - `euclidean_distance_l2`\n - `minkowski_distance`\n - `manhattan_distance`\n - `chebyshev_distance`\n* Computation time for 4 metrics 4.2 million samples experiment is **24 seconds vs 51 minutes** if looping using `scipy.spatial.distance` implemntations.\n\n## TODO\n* Safer memory allocation. I did not have issues but if you ran out of memory please manually set the `batch_size` argument.\n\n## Contribution\n* Contributions are welcomed, and they are greatly appreciated! Every little bit helps, and credit will always be given.\n* Please check [CONTRIBUTING.md](https://github.com/ma7555/evalify/blob/main/CONTRIBUTING.md) for guidelines.\n\n## Citation\n* If you use this software, please cite it using the metadata from [CITATION.cff](https://github.com/ma7555/evalify/blob/main/CITATION.cff)\n\n\n",
"bugtrack_url": null,
"license": "BSD-3-Clause",
"summary": "Evaluate your face or voice verification models literally in seconds.",
"version": "1.0.0",
"project_urls": {
"Homepage": "https://github.com/ma7555/evalify"
},
"split_keywords": [
"biometric verification",
" biometric authentication",
" evaluation"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "28bbef0234424a55dab0505018e0ebe904263033234118d72f6e01dc7126ea10",
"md5": "da123037532098bfa992199f5998b700",
"sha256": "6e5e953edf56f44e0ddc09cad2260df8abd273bcd798a32328739a0cd8fa067f"
},
"downloads": -1,
"filename": "evalify-1.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "da123037532098bfa992199f5998b700",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.9",
"size": 12886,
"upload_time": "2024-11-08T23:54:58",
"upload_time_iso_8601": "2024-11-08T23:54:58.829542Z",
"url": "https://files.pythonhosted.org/packages/28/bb/ef0234424a55dab0505018e0ebe904263033234118d72f6e01dc7126ea10/evalify-1.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d81ddc1df17f835a116950bc29d94885a7cac859242bd70ab0e27666a07cb4fc",
"md5": "6f7ec347eb215f0c2b3988b5eba15731",
"sha256": "512162b5e8f58223f9c3e0144f514514db96cf7164724b2eac4e69f743a2c497"
},
"downloads": -1,
"filename": "evalify-1.0.0.tar.gz",
"has_sig": false,
"md5_digest": "6f7ec347eb215f0c2b3988b5eba15731",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.9",
"size": 13252,
"upload_time": "2024-11-08T23:55:00",
"upload_time_iso_8601": "2024-11-08T23:55:00.374013Z",
"url": "https://files.pythonhosted.org/packages/d8/1d/dc1df17f835a116950bc29d94885a7cac859242bd70ab0e27666a07cb4fc/evalify-1.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-08 23:55:00",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ma7555",
"github_project": "evalify",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"tox": true,
"lcname": "evalify"
}
JSON
