# 🧐 Detectors
Package to accelerate research on generalized out-of-distribution (OOD) detection.
Under development. Please report any issues or bugs [here](https://github.com/edadaltocg/detectors/issues).
## Stats
[](https://github.com/edadaltocg/detectors/graphs/commit-activity)
[](https://github.com/edadaltocg/detectors/actions/workflows/build.yml)
[](http://detectors.readthedocs.io/?badge=latest)
[](https://doi.org/10.5281/zenodo.7883080)
## What is it?
This library is aimed at assisting researchers in the field of generalized OOD detection. It is inspired by [HF's Transformers](https://https://github.com/huggingface/transformers) and features implementations of baselines, metrics, and data sets that allow researchers to perform meaningful benchmarking and development of ood detection methods. It features:
- `methods`: more than 20 detection methods implemented.
- `pipelines`: evaluating OOD detectors on popular benchmarks, such as MNIST, CIFAR, and ImageNet benchmarks with random seed support for reproducibility.
- `datasets`: OOD datasets implemented with md5 checksums and without the need to download them manually.
- `models`: model architectures totally integrated with [`timm`](https://github.com/huggingface/pytorch-image-models).
- `eval`: implementation of fast OOD evaluation metrics.
- Several aggregation methods for multi-layer OOD detection.
- Pipelines for open set recognition and covariate drift detection.
## Installation
Please follow the instructions [here](https://pytorch.org/get-started/locally/) to install PyTorch. Installing PyTorch with CUDA support is strongly recommended.
```bash
pip install detectors
```
To install the latest version from the source:
```bash
git clone https://github.com/edadaltocg/detectors.git
cd detectors
pip install --upgrade pip setuptools wheel
pip install -e .
```
Also, you have easy access to the Python scripts from the examples:
```bash
cd examples
```
## Examples
The following examples show how to use the library and how it can be integrated into your research. For more examples, please check the [documentation](https://detectors.readthedocs.io/en/latest/use_cases/).
### Running a benchmark
The following example shows how to run a benchmark.
```python
import detectors
import torch
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = detectors.create_model("resnet18_cifar10", pretrained=True)
model = model.to(device)
test_transform = detectors.create_transform(model)
pipeline = detectors.create_pipeline("ood_benchmark_cifar10", transform=test_transform)
method = detectors.create_detector("msp", model=model)
pipeline_results = pipeline.run(method)
print(pipeline.report(pipeline_results["results"]))
```
We recommend running benchmarks on machines equipped with large RAM and GPUs with 16GB of memory or larger to leverage large batch sizes and faster inference.
### Creating a detector
The following example shows how to create a detector. The only requirement is that the method takes an input `x` and returns a score.
```python
import torch
import detectors
@detectors.register_detector("awesome_detector")
def awesome_detector(x: torch.Tensor, model, **kwargs):
# Do something awesome with the model and the input
return scores
# Instantiating the detector
method = detectors.create_detector("awesome_detector", model=model)
```
Alternatively, you can use the `Detector` class to create a detector that requires some initialization or state to be fitted before being called (e.g., Mahalanobis detector):
```python
import torch
import detectors
@detectors.register_detector("awesome_detector")
class AwesomeDetector(detectors.Detector):
def __init__(self, model, **kwargs):
self.model = model
def __call__(self, x: torch.Tensor, **kwargs):
# Do something awesome with the model and the input
return scores
# Instantiating the detector
method = detectors.create_detector("awesome_detector", model=model)
```
Check the [documentation](https://detectors.readthedocs.io/en/latest/use_cases/) for more information.
### Listing available resources
The following example shows how to list all available resources in the library.
```python
import detectors
# list all available models (same as timm.list_models)
print(detectors.list_models())
# list all available models with a specific pattern
print(detectors.list_models("*cifar*"))
# list all available datasets
print(detectors.list_datasets())
# list all available detectors
print(detectors.list_detectors())
# list all available pipelines
print(detectors.list_pipelines())
```
## FAQ over specific documents
**Methods**
- [Documentation](https://detectors.readthedocs.io/en/latest/use_cases/)
**Pipelines**
- [Documentation](https://detectors.readthedocs.io/en/latest/use_cases/)
**Pypi**
- [Website](https://pypi.org/project/detectors)
## Contributing
As an open-source project in a rapidly developing field, we are open to contributions, whether in the form of a new feature, improved infra, or better documentation.
See the [contributing guidelines](https://github.com/edadaltocg/detectors/blob/master/CONTRIBUTING.md) for instructions on how to make your first contribution to `detectors`.
### Thanks to all our contributors
<a href="https://github.com/edadaltocg/detectors/graphs/contributors">
<img src="https://contributors-img.web.app/image?repo=edadaltocg/detectors" />
</a>
## Contact
Concerning this package, its use, and bugs, use the [issue page](https://github.com/edadaltocg/detectors/issues) of the [ruptures repository](https://github.com/edadaltocg/detectors). For other inquiries, you can contact me [here](https://edadaltocg.github.io/contact/).
## Important links
- [Documentation](http://detectors.readthedocs.io/)
- [Pypi package index](https://pypi.python.org/pypi/detectors)
- [Github repository](https://github.com/edadaltocg/detectors)
## Limitations
- This library is only compatible with PyTorch models.
- This library has implemented only computer vision pipelines and datasets.
## Citing detectors
The detection of Out-of-Distribution (OOD) has created a new way of securing machine intelligence, but despite its many successes, it can be difficult to understand due to the various methods available and their intricate implementations. The fast pace of research and the wide range of OOD methods makes it challenging to navigate the field, which can be a problem for those who have recently joined the field or want to deploy OOD detection. The library we have created aims to lower these barriers by providing a resource for researchers of any background to understand the methods available, how they work, and how to be successful with OOD detection.
If you find this repository useful, please consider giving it a star 🌟 and citing it as below:
```bibtex
@software{detectors2023,
author = {Eduardo Dadalto},
title = {Detectors: a Python Library for Generalized Out-Of-Distribution Detection},
url = {https://github.com/edadaltocg/detectors},
doi = {https://doi.org/10.5281/zenodo.7883596},
month = {5},
year = {2023}
}
```
Raw data
{
"_id": null,
"home_page": "https://github.com/edadaltocg/detectors",
"name": "detectors",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8.0",
"maintainer_email": "",
"keywords": "vision deep learning pytorch OOD",
"author": "Eduardo Dadalto",
"author_email": "edadaltocg@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/c6/fb/65c911684e7731c2901faeb6c3b6fb4df9c781ed6d26a38a9a312b821f0c/detectors-0.1.11.tar.gz",
"platform": null,
"description": "# \ud83e\uddd0 Detectors\n\nPackage to accelerate research on generalized out-of-distribution (OOD) detection.\n\nUnder development. Please report any issues or bugs [here](https://github.com/edadaltocg/detectors/issues).\n\n## Stats\n\n[](https://github.com/edadaltocg/detectors/graphs/commit-activity)\n[](https://github.com/edadaltocg/detectors/actions/workflows/build.yml)\n[](http://detectors.readthedocs.io/?badge=latest)\n[](https://doi.org/10.5281/zenodo.7883080)\n\n## What is it?\n\nThis library is aimed at assisting researchers in the field of generalized OOD detection. It is inspired by [HF's Transformers](https://https://github.com/huggingface/transformers) and features implementations of baselines, metrics, and data sets that allow researchers to perform meaningful benchmarking and development of ood detection methods. It features:\n\n- `methods`: more than 20 detection methods implemented.\n- `pipelines`: evaluating OOD detectors on popular benchmarks, such as MNIST, CIFAR, and ImageNet benchmarks with random seed support for reproducibility.\n- `datasets`: OOD datasets implemented with md5 checksums and without the need to download them manually.\n- `models`: model architectures totally integrated with [`timm`](https://github.com/huggingface/pytorch-image-models).\n- `eval`: implementation of fast OOD evaluation metrics.\n- Several aggregation methods for multi-layer OOD detection.\n- Pipelines for open set recognition and covariate drift detection.\n\n## Installation\n\nPlease follow the instructions [here](https://pytorch.org/get-started/locally/) to install PyTorch. Installing PyTorch with CUDA support is strongly recommended.\n\n```bash\npip install detectors\n```\n\nTo install the latest version from the source:\n\n```bash\ngit clone https://github.com/edadaltocg/detectors.git\ncd detectors\npip install --upgrade pip setuptools wheel\npip install -e .\n```\n\nAlso, you have easy access to the Python scripts from the examples:\n\n```bash\ncd examples\n```\n\n## Examples\n\nThe following examples show how to use the library and how it can be integrated into your research. For more examples, please check the [documentation](https://detectors.readthedocs.io/en/latest/use_cases/).\n\n### Running a benchmark\n\nThe following example shows how to run a benchmark.\n\n```python\nimport detectors\nimport torch\n\n\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nmodel = detectors.create_model(\"resnet18_cifar10\", pretrained=True)\nmodel = model.to(device)\ntest_transform = detectors.create_transform(model)\n\npipeline = detectors.create_pipeline(\"ood_benchmark_cifar10\", transform=test_transform)\nmethod = detectors.create_detector(\"msp\", model=model)\n\npipeline_results = pipeline.run(method)\nprint(pipeline.report(pipeline_results[\"results\"]))\n```\n\nWe recommend running benchmarks on machines equipped with large RAM and GPUs with 16GB of memory or larger to leverage large batch sizes and faster inference.\n\n### Creating a detector\n\nThe following example shows how to create a detector. The only requirement is that the method takes an input `x` and returns a score.\n\n```python\nimport torch\nimport detectors\n\n\n@detectors.register_detector(\"awesome_detector\")\ndef awesome_detector(x: torch.Tensor, model, **kwargs):\n # Do something awesome with the model and the input\n return scores\n\n# Instantiating the detector\nmethod = detectors.create_detector(\"awesome_detector\", model=model)\n```\n\nAlternatively, you can use the `Detector` class to create a detector that requires some initialization or state to be fitted before being called (e.g., Mahalanobis detector):\n\n```python\nimport torch\nimport detectors\n\n\n@detectors.register_detector(\"awesome_detector\")\nclass AwesomeDetector(detectors.Detector):\n def __init__(self, model, **kwargs):\n self.model = model\n\n def __call__(self, x: torch.Tensor, **kwargs):\n # Do something awesome with the model and the input\n return scores\n\n# Instantiating the detector\nmethod = detectors.create_detector(\"awesome_detector\", model=model)\n```\n\nCheck the [documentation](https://detectors.readthedocs.io/en/latest/use_cases/) for more information.\n\n### Listing available resources\n\nThe following example shows how to list all available resources in the library.\n\n```python\nimport detectors\n\n\n# list all available models (same as timm.list_models)\nprint(detectors.list_models())\n# list all available models with a specific pattern\nprint(detectors.list_models(\"*cifar*\"))\n# list all available datasets\nprint(detectors.list_datasets())\n# list all available detectors\nprint(detectors.list_detectors())\n# list all available pipelines\nprint(detectors.list_pipelines())\n```\n\n## FAQ over specific documents\n\n**Methods**\n\n- [Documentation](https://detectors.readthedocs.io/en/latest/use_cases/)\n\n**Pipelines**\n\n- [Documentation](https://detectors.readthedocs.io/en/latest/use_cases/)\n\n**Pypi**\n\n- [Website](https://pypi.org/project/detectors)\n\n## Contributing\n\nAs an open-source project in a rapidly developing field, we are open to contributions, whether in the form of a new feature, improved infra, or better documentation.\n\nSee the [contributing guidelines](https://github.com/edadaltocg/detectors/blob/master/CONTRIBUTING.md) for instructions on how to make your first contribution to `detectors`.\n\n### Thanks to all our contributors\n\n<a href=\"https://github.com/edadaltocg/detectors/graphs/contributors\">\n <img src=\"https://contributors-img.web.app/image?repo=edadaltocg/detectors\" />\n</a>\n\n## Contact\n\nConcerning this package, its use, and bugs, use the [issue page](https://github.com/edadaltocg/detectors/issues) of the [ruptures repository](https://github.com/edadaltocg/detectors). For other inquiries, you can contact me [here](https://edadaltocg.github.io/contact/).\n\n## Important links\n\n- [Documentation](http://detectors.readthedocs.io/)\n- [Pypi package index](https://pypi.python.org/pypi/detectors)\n- [Github repository](https://github.com/edadaltocg/detectors)\n\n## Limitations\n\n- This library is only compatible with PyTorch models.\n- This library has implemented only computer vision pipelines and datasets.\n\n## Citing detectors\n\nThe detection of Out-of-Distribution (OOD) has created a new way of securing machine intelligence, but despite its many successes, it can be difficult to understand due to the various methods available and their intricate implementations. The fast pace of research and the wide range of OOD methods makes it challenging to navigate the field, which can be a problem for those who have recently joined the field or want to deploy OOD detection. The library we have created aims to lower these barriers by providing a resource for researchers of any background to understand the methods available, how they work, and how to be successful with OOD detection.\n\nIf you find this repository useful, please consider giving it a star \ud83c\udf1f and citing it as below:\n\n```bibtex\n@software{detectors2023,\nauthor = {Eduardo Dadalto},\ntitle = {Detectors: a Python Library for Generalized Out-Of-Distribution Detection},\nurl = {https://github.com/edadaltocg/detectors},\ndoi = {https://doi.org/10.5281/zenodo.7883596},\nmonth = {5},\nyear = {2023}\n}\n```\n",
"bugtrack_url": null,
"license": "APACHE 2.0",
"summary": "Detectors: a python package to benchmark generalized out-of-distribution detection methods.",
"version": "0.1.11",
"project_urls": {
"Homepage": "https://github.com/edadaltocg/detectors"
},
"split_keywords": [
"vision",
"deep",
"learning",
"pytorch",
"ood"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "b57571031d03fd35fbd79c2ceee72c03950263e883a3d45a62cea137b394c24e",
"md5": "d56957e348c867c73879f256e85240d0",
"sha256": "705e80082d9197ec895019c7eececdc508ba857a1e468dde2e6f9fced149c8de"
},
"downloads": -1,
"filename": "detectors-0.1.11-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d56957e348c867c73879f256e85240d0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8.0",
"size": 616793,
"upload_time": "2024-01-02T02:08:04",
"upload_time_iso_8601": "2024-01-02T02:08:04.615375Z",
"url": "https://files.pythonhosted.org/packages/b5/75/71031d03fd35fbd79c2ceee72c03950263e883a3d45a62cea137b394c24e/detectors-0.1.11-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c6fb65c911684e7731c2901faeb6c3b6fb4df9c781ed6d26a38a9a312b821f0c",
"md5": "afb84e01094528a268e306b26112b160",
"sha256": "4c363c15bc374d495c3937f8e1057d34679d2c045e3fae5177e3ae655045a14e"
},
"downloads": -1,
"filename": "detectors-0.1.11.tar.gz",
"has_sig": false,
"md5_digest": "afb84e01094528a268e306b26112b160",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8.0",
"size": 574519,
"upload_time": "2024-01-02T02:08:06",
"upload_time_iso_8601": "2024-01-02T02:08:06.528437Z",
"url": "https://files.pythonhosted.org/packages/c6/fb/65c911684e7731c2901faeb6c3b6fb4df9c781ed6d26a38a9a312b821f0c/detectors-0.1.11.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-02 02:08:06",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "edadaltocg",
"github_project": "detectors",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "detectors"
}
JSON
