calibrator

Name	calibrator JSON
Version	0.0.13 JSON
	download
home_page	None
Summary	A package for calibration on deep learning models for classification tasks.
upload_time	2024-11-05 02:48:39
maintainer	None
docs_url	None
author	None
requires_python	>=3.8
license	None
keywords
VCS
bugtrack_url
requirements	No requirements were recorded.
Travis-CI	No Travis.
coveralls test coverage	No coveralls.

            # Calibration
a python package for calibrating the deep learning models

## Install
```sh
pip install calibrator
```


## Use case
```python
from calibrator import LocalCalibrator
import torch

val_logits = torch.randn(1000, 10)
val_labels = torch.randint(0, 10, (1000,))
test_logits = torch.randn(1000, 10)

calibrator = LocalCalibrator()
eps_opt = calibrator.fit(val_logits, val_labels)
calibrated_probability = calibrator.calibrate(test_logits)
```

## Benchmarking
| Datasets | Method | Method Description | ECE | ACE | MCE | CECE | PIECE |
| ------------------ | ----------- | ---------- | ---------- |  -------- | ---------- |  -------- | ---------- |
| CIFAR-10 |   |   |   |   |   |   |
| CIFAR-100 |   |   |   |   |   |   |
| ImageNet-1K |   |   |   |   |   |   |

## Package Information
### Post-Hoc Calibration methods
| Calibration methods | Description | Paper | Source Code | Status |
| ------------------ | ----------- | ---------- | ---------- | -------- |
| Temperature Scaling (TS) | ICML 2017 | [paper](https://arxiv.org/abs/1706.04599) | [code](https://github.com/gpleiss/temperature_scaling/blob/master/temperature_scaling.py) | ✅ Implemented |
| Parameterized Temperature Scaling (PTS) | ECCV 2022 | [paper](https://arxiv.org/abs/2102.12182) | [ code](https://github.com/tochris/pts-uncertainty) | 🔜 Pending |
| Ensemble Temperature Scaling (ETS) | ICML 2020 | [paper](https://proceedings.mlr.press/v119/zhang20k.html) | [code](https://github.com/yashchandra/ensemble-calibration) | 🔜 Pending |
| Class-based Temperature Scaling (CTS) | EUSIPCO 2021 | [paper](https://ieeexplore.ieee.org/document/9616219) | unavailable | 🔜 Pending |
| Group Calibration with Temperature Scaling (GCTS)| NeurIPS 2023 | [paper](https://arxiv.org/abs/2306.04985) | [code](https://github.com/ThyrixYang/group_calibration) | 🔜 Pending |
| Proximity-informed Calibration (PROCAL) | NeurIPS 2023 | [paper](https://arxiv.org/abs/2306.04590) | [code](https://github.com/MiaoXiong2320/ProximityBias-Calibration) | 🔜 Pending |
| Isotonic Regression |   |   |   |   |
| Histogram Binning |   |   |   |   |
| Platt Scaling |   |   |   |   |
| Bayesian Binning into Quantiles (BBQ) | AAAI 2015 | [paper](https://ojs.aaai.org/index.php/AAAI/article/view/9602) |  | 🔜 Pending |
| BetaCal |   |   |   |   |
| Scaling-Binning Calibrator | NeuIPS 2019 | [paper](https://arxiv.org/pdf/1909.10155) | [code](https://pypi.org/project/uncertainty-calibration/) | 🔜 Pending |
| Dirichlet calibration |  NeurIPS 2019 |  [paper](https://arxiv.org/abs/1910.12656) | [code](https://github.com/dirichletcal/experiments_neurips) | 🔜 Pending |


### Train-time Calibration Methods
| Calibration Losses | Description | Paper | Source Code | Status |
| ------------------ | ----------- | ---------- | ---------- | -------- |
| Focal Loss |   |   |   |   |
| Dual Focal Loss |   |   |   |   |
| Adaptive Focal Loss |   |   |   |   |



### Metrics
| Metrics | Description | Paper | Source Code | Status |
| ------------------ | ----------- | ---------- | ---------- | -------- |
| Expected Calibration Error (ECE) | AAAI 2015 | [paper](https://ojs.aaai.org/index.php/AAAI/article/view/9602) | [code](https://github.com/torrvision/focal_calibration/blob/main/Metrics/metrics.py) | ✅ Implemented |
| Maximum Calibration Error (MCE) | AAAI 2015 | [paper](https://ojs.aaai.org/index.php/AAAI/article/view/9602) | [code](https://github.com/torrvision/focal_calibration/blob/main/Metrics/metrics.py) | 🔜 Pending |
| Adaptive Calibration Error (ACE) | CVPRW 2019 | [paper](https://openaccess.thecvf.com/content_CVPRW_2019/papers/Uncertainty%20and%20Robustness%20in%20Deep%20Visual%20Learning/Nixon_Measuring_Calibration_in_Deep_Learning_CVPRW_2019_paper.pdf) | [code](https://github.com/torrvision/focal_calibration/blob/main/Metrics/metrics.py) | ✅ Implemented |
| Classwise Expected Calibration Error (CECE) | NeurIPS 2019 | [paper](https://arxiv.org/abs/1910.12656) | [code](https://github.com/torrvision/focal_calibration/blob/main/Metrics/metrics.py) | ✅ Implemented |
| Negative Log Likelihood (NLL) |   |   |   |   |
| Accuracy |   |   |   |   |
| Proximity-informed Expected Calibration Error (PIECE) | NeurIPS 2023 | [paper](https://arxiv.org/abs/2306.04590) | [code](https://github.com/MiaoXiong2320/ProximityBias-Calibration) | 🔜 Pending |

### Pre-trained Model Weights
| Datasets | Description | Paper | Source Code | Status |
| ------------------ | ----------- | ---------- | ---------- | -------- |
| CIFAR-10 |   |   |   |   |
| CIFAR-100 |   |   |   |   |
| ImageNet |   |   |   |   |
| ImageNet-100 |   |   |   |   |
| ImageNet-1000 |   |   |   |   |

### Pre-calculated Logits
| Datasets | Description | Paper | Source Code | Status |
| ------------------ | ----------- | ---------- | ---------- | -------- |
| CIFAR-10 |   |   |   |   |
| CIFAR-100 |   |   |   |   |
| ImageNet |   |   |   |   |
| ImageNet-100 |   |   |   |   |
| ImageNet-1000 |   |   |   |   |


            

Raw data

            {
    "_id": null,
    "home_page": null,
    "name": "calibrator",
    "maintainer": null,
    "docs_url": null,
    "requires_python": ">=3.8",
    "maintainer_email": null,
    "keywords": null,
    "author": null,
    "author_email": "Linwei Tao <dylan.tao94@gmail.com>",
    "download_url": "https://files.pythonhosted.org/packages/af/1f/417d52cfab8d0958173983955af9596892debc028e3fd2ea30dc99039115/calibrator-0.0.13.tar.gz",
    "platform": null,
    "description": "# Calibration\na python package for calibrating the deep learning models\n\n## Install\n```sh\npip install calibrator\n```\n\n\n## Use case\n```python\nfrom calibrator import LocalCalibrator\nimport torch\n\nval_logits = torch.randn(1000, 10)\nval_labels = torch.randint(0, 10, (1000,))\ntest_logits = torch.randn(1000, 10)\n\ncalibrator = LocalCalibrator()\neps_opt = calibrator.fit(val_logits, val_labels)\ncalibrated_probability = calibrator.calibrate(test_logits)\n```\n\n## Benchmarking\n| Datasets | Method | Method Description | ECE | ACE | MCE | CECE | PIECE |\n| ------------------ | ----------- | ---------- | ---------- |  -------- | ---------- |  -------- | ---------- |\n| CIFAR-10 |   |   |   |   |   |   |\n| CIFAR-100 |   |   |   |   |   |   |\n| ImageNet-1K |   |   |   |   |   |   |\n\n## Package Information\n### Post-Hoc Calibration methods\n| Calibration methods | Description | Paper | Source Code | Status |\n| ------------------ | ----------- | ---------- | ---------- | -------- |\n| Temperature Scaling (TS) | ICML 2017 | [paper](https://arxiv.org/abs/1706.04599) | [code](https://github.com/gpleiss/temperature_scaling/blob/master/temperature_scaling.py) | \u2705 Implemented |\n| Parameterized Temperature Scaling (PTS) | ECCV 2022 | [paper](https://arxiv.org/abs/2102.12182) | [ code](https://github.com/tochris/pts-uncertainty) | \ud83d\udd1c Pending |\n| Ensemble Temperature Scaling (ETS) | ICML 2020 | [paper](https://proceedings.mlr.press/v119/zhang20k.html) | [code](https://github.com/yashchandra/ensemble-calibration) | \ud83d\udd1c Pending |\n| Class-based Temperature Scaling (CTS) | EUSIPCO 2021 | [paper](https://ieeexplore.ieee.org/document/9616219) | unavailable | \ud83d\udd1c Pending |\n| Group Calibration with Temperature Scaling (GCTS)| NeurIPS 2023 | [paper](https://arxiv.org/abs/2306.04985) | [code](https://github.com/ThyrixYang/group_calibration) | \ud83d\udd1c Pending |\n| Proximity-informed Calibration (PROCAL) | NeurIPS 2023 | [paper](https://arxiv.org/abs/2306.04590) | [code](https://github.com/MiaoXiong2320/ProximityBias-Calibration) | \ud83d\udd1c Pending |\n| Isotonic Regression |   |   |   |   |\n| Histogram Binning |   |   |   |   |\n| Platt Scaling |   |   |   |   |\n| Bayesian Binning into Quantiles (BBQ) | AAAI 2015 | [paper](https://ojs.aaai.org/index.php/AAAI/article/view/9602) |  | \ud83d\udd1c Pending |\n| BetaCal |   |   |   |   |\n| Scaling-Binning Calibrator | NeuIPS 2019 | [paper](https://arxiv.org/pdf/1909.10155) | [code](https://pypi.org/project/uncertainty-calibration/) | \ud83d\udd1c Pending |\n| Dirichlet calibration |  NeurIPS 2019 |  [paper](https://arxiv.org/abs/1910.12656) | [code](https://github.com/dirichletcal/experiments_neurips) | \ud83d\udd1c Pending |\n\n\n### Train-time Calibration Methods\n| Calibration Losses | Description | Paper | Source Code | Status |\n| ------------------ | ----------- | ---------- | ---------- | -------- |\n| Focal Loss |   |   |   |   |\n| Dual Focal Loss |   |   |   |   |\n| Adaptive Focal Loss |   |   |   |   |\n\n\n\n### Metrics\n| Metrics | Description | Paper | Source Code | Status |\n| ------------------ | ----------- | ---------- | ---------- | -------- |\n| Expected Calibration Error (ECE) | AAAI 2015 | [paper](https://ojs.aaai.org/index.php/AAAI/article/view/9602) | [code](https://github.com/torrvision/focal_calibration/blob/main/Metrics/metrics.py) | \u2705 Implemented |\n| Maximum Calibration Error (MCE) | AAAI 2015 | [paper](https://ojs.aaai.org/index.php/AAAI/article/view/9602) | [code](https://github.com/torrvision/focal_calibration/blob/main/Metrics/metrics.py) | \ud83d\udd1c Pending |\n| Adaptive Calibration Error (ACE) | CVPRW 2019 | [paper](https://openaccess.thecvf.com/content_CVPRW_2019/papers/Uncertainty%20and%20Robustness%20in%20Deep%20Visual%20Learning/Nixon_Measuring_Calibration_in_Deep_Learning_CVPRW_2019_paper.pdf) | [code](https://github.com/torrvision/focal_calibration/blob/main/Metrics/metrics.py) | \u2705 Implemented |\n| Classwise Expected Calibration Error (CECE) | NeurIPS 2019 | [paper](https://arxiv.org/abs/1910.12656) | [code](https://github.com/torrvision/focal_calibration/blob/main/Metrics/metrics.py) | \u2705 Implemented |\n| Negative Log Likelihood (NLL) |   |   |   |   |\n| Accuracy |   |   |   |   |\n| Proximity-informed Expected Calibration Error (PIECE) | NeurIPS 2023 | [paper](https://arxiv.org/abs/2306.04590) | [code](https://github.com/MiaoXiong2320/ProximityBias-Calibration) | \ud83d\udd1c Pending |\n\n### Pre-trained Model Weights\n| Datasets | Description | Paper | Source Code | Status |\n| ------------------ | ----------- | ---------- | ---------- | -------- |\n| CIFAR-10 |   |   |   |   |\n| CIFAR-100 |   |   |   |   |\n| ImageNet |   |   |   |   |\n| ImageNet-100 |   |   |   |   |\n| ImageNet-1000 |   |   |   |   |\n\n### Pre-calculated Logits\n| Datasets | Description | Paper | Source Code | Status |\n| ------------------ | ----------- | ---------- | ---------- | -------- |\n| CIFAR-10 |   |   |   |   |\n| CIFAR-100 |   |   |   |   |\n| ImageNet |   |   |   |   |\n| ImageNet-100 |   |   |   |   |\n| ImageNet-1000 |   |   |   |   |\n\n",
    "bugtrack_url": null,
    "license": null,
    "summary": "A package for calibration on deep learning models for classification tasks.",
    "version": "0.0.13",
    "project_urls": {
        "Homepage": "https://github.com/pypa/sampleproject",
        "Issues": "https://github.com/pypa/sampleproject/issues"
    },
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "7cd31f4aca9d693e28801b197815cea628cbcb261777e867047c1454a089a154",
                "md5": "a0c5ea76fc47c1a0a72c1366c82361b1",
                "sha256": "f1d8f3da1ffce3503b42afa970c76a5d0b9cbbf9b407ee386e3aca40039ca5b9"
            },
            "downloads": -1,
            "filename": "calibrator-0.0.13-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "a0c5ea76fc47c1a0a72c1366c82361b1",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.8",
            "size": 11787,
            "upload_time": "2024-11-05T02:48:38",
            "upload_time_iso_8601": "2024-11-05T02:48:38.228329Z",
            "url": "https://files.pythonhosted.org/packages/7c/d3/1f4aca9d693e28801b197815cea628cbcb261777e867047c1454a089a154/calibrator-0.0.13-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "af1f417d52cfab8d0958173983955af9596892debc028e3fd2ea30dc99039115",
                "md5": "c41b50a9292b69c9657d13836a33c8dc",
                "sha256": "efd85ad0a2336abeca9870200e9925b53c3c85de1555df84c47745fed8578636"
            },
            "downloads": -1,
            "filename": "calibrator-0.0.13.tar.gz",
            "has_sig": false,
            "md5_digest": "c41b50a9292b69c9657d13836a33c8dc",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.8",
            "size": 10118,
            "upload_time": "2024-11-05T02:48:39",
            "upload_time_iso_8601": "2024-11-05T02:48:39.207625Z",
            "url": "https://files.pythonhosted.org/packages/af/1f/417d52cfab8d0958173983955af9596892debc028e3fd2ea30dc99039115/calibrator-0.0.13.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-11-05 02:48:39",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "pypa",
    "github_project": "sampleproject",
    "github_fetch_exception": true,
    "lcname": "calibrator"
}