# miseval: a metric library for Medical Image Segmentation EVALuation
[](https://www.python.org/)
[](https://github.com/frankkramer-lab/miseval)
[](https://pypi.org/project/miseval/)
[](https://pypistats.org/packages/miseval)
[](https://www.gnu.org/licenses/gpl-3.0.en.html)
The open-source and free to use Python package miseval was developed to establish a standardized medical image segmentation evaluation procedure. We hope that our this will help improve evaluation quality, reproducibility, and comparability in future studies in the field of medical image segmentation.
#### Guideline on Evaluation Metrics for Medical Image Segmentation
1. Use DSC as main metric for validation and performance interpretation.
2. Use AHD for interpretation on point position sensitivity (contour) if needed.
3. Avoid any interpretations based on high pixel accuracy scores.
4. Provide next to DSC also IoU, Sensitivity, and Specificity for method comparability.
5. Provide sample visualizations, comparing the annotated and predicted segmentation, for visual evaluation as well as to avoid statistical bias.
6. Avoid cherry-picking high-scoring samples.
7. Provide histograms or box plots showing the scoring distribution across the dataset.
8. For multi-class problems, provide metric computations for each class individually.
9. Avoid confirmation bias through macro-averaging classes which is pushing scores via background class inclusion.
10. Provide access to evaluation scripts and results with journal data services or third-party services like GitHub and Zenodo for easier reproducibility.
## Implemented Metrics
| Metric | Index in miseval | Function in miseval |
| ----------- | ----------- | ----------- |
| Dice Similarity Index | "DSC", "Dice", "DiceSimilarityCoefficient" | miseval.calc_DSC() |
| Intersection-Over-Union | "IoU", "Jaccard", "IntersectionOverUnion" | miseval.calc_IoU() |
| Sensitivity | "SENS", "Sensitivity", "Recall", "TPR", "TruePositiveRate" | miseval.calc_Sensitivity() |
| Specificity | "SPEC", "Specificity", "TNR", "TrueNegativeRate" | miseval.calc_Specificity() |
| Precision | "PREC", "Precision" | miseval.calc_Precision() |
| Accuracy | "ACC", "Accuracy", "RI", "RandIndex" | miseval.calc_Accuracy() |
| Balanced Accuracy | "BACC", "BalancedAccuracy" | miseval.calc_BalancedAccuracy() |
| Adjusted Rand Index | "ARI", "AdjustedRandIndex" | miseval.calc_AdjustedRandIndex() |
| AUC | "AUC", "AUC_trapezoid" | miseval.calc_AUC() |
| Cohen's Kappa | "KAP", "Kappa", "CohensKappa" | miseval.calc_Kappa() |
| Hausdorff Distance | "HD", "HausdorffDistance" | miseval.calc_SimpleHausdorffDistance() |
| Average Hausdorff Distance | "AHD", "AverageHausdorffDistance" | miseval.calc_AverageHausdorffDistance() |
| Volumetric Similarity | "VS", "VolumetricSimilarity" | miseval.calc_VolumetricSimilarity() |
| Matthews Correlation Coefficient | "MCC", "MatthewsCorrelationCoefficient" | miseval.calc_MCC() |
| Normalized Matthews Correlation Coefficient | "nMCC", "MCC_normalized" | miseval.calc_MCC_Normalized() |
| Absolute Matthews Correlation Coefficient | "aMCC", "MCC_absolute" | miseval.calc_MCC_Absolute() |
| Boundary Distance | "BD", "Distance", " BoundaryDistance" | miseval.calc_Boundary_Distance() |
| Hinge Loss | "Hinge", "HingeLoss" | miseval.calc_Hinge() |
| Cross-Entropy | "CE", "CrossEntropy" | miseval.calc_CrossEntropy() |
| True Positive | "TP", "TruePositive" | miseval.calc_TruePositive() |
| False Positive | "FP", "FalsePositive" | miseval.calc_FalsePositive() |
| True Negative | "TN", "TrueNegative" | miseval.calc_TrueNegative() |
| False Negative | "FN", "FalseNegative" | miseval.calc_FalseNegative() |
#### Options for Boundary Distance computation
```
List of available distances:
Bhattacharyya distance bhattacharyya
Bhattacharyya coefficient bhattacharyya_coefficient
Canberra distance canberra
Chebyshev distance chebyshev
Chi Square distance chi_square
Cosine Distance cosine
Euclidean distance euclidean
Hamming distance hamming
Jensen-Shannon divergence jensen_shannon
Kullback-Leibler divergence kullback_leibler
Mean absolute error mae
Taxicab geometry manhattan, cityblock, total_variation
Minkowski distance minkowsky
Mean squared error mse
Pearson's distance pearson
Squared deviations from the mean squared_variation
Distance Pooling (how to combine computed distances to a single value):
Distance Sum sum
Distance Averaging mean
Minimum Distance amin
Maximum Distance amax
```
## How to Use
#### Example
```python
# load libraries
import numpy as np
from miseval import evaluate
# Get some ground truth / annotated segmentations
np.random.seed(1)
real_bi = np.random.randint(2, size=(64,64)) # binary (2 classes)
real_mc = np.random.randint(5, size=(64,64)) # multi-class (5 classes)
# Get some predicted segmentations
np.random.seed(2)
pred_bi = np.random.randint(2, size=(64,64)) # binary (2 classes)
pred_mc = np.random.randint(5, size=(64,64)) # multi-class (5 classes)
# Run binary evaluation
dice = evaluate(real_bi, pred_bi, metric="DSC")
# returns single np.float64 e.g. 0.75
# Run multi-class evaluation
dice_list = evaluate(real_mc, pred_mc, metric="DSC", multi_class=True,
n_classes=5)
# returns array of np.float64 e.g. [0.9, 0.2, 0.6, 0.0, 0.4]
# for each class, one score
```
#### Core function: Evaluate()
Every metric in miseval can be called via our core function `evaluate()`.
The miseval eavluate function can be run with different metrics as backbone.
You can pass the following options to the metric parameter:
- String naming one of the metric labels, for example "DSC"
- Directly passing a metric function, for example calc_DSC_Sets (from dice.py)
- Passing a custom metric function
List of metrics : See `miseval/__init__.py` under section "Access Functions to Metric Functions"
The classes in a segmentation mask must be ongoing starting from 0 (integers from 0 to n_classes-1).
A segmentation mask is allowed to have either no channel axis or just 1 (e.g. 512x512x1),
which contains the annotation.
```python
"""
Arguments:
truth (NumPy Matrix): Ground Truth segmentation mask.
pred (NumPy Matrix): Prediction segmentation mask.
metric (String or Function): Metric function. Either a function directly or encoded as
String from miseval or a custom function.
multi_class (Boolean): Boolean parameter, if segmentation is a binary or multi-class
problem. By default False -> Binary mode.
n_classes (Integer): Number of classes. By default 2 -> Binary
kwargs (arguments): Additional arguments for passing down to metric functions.
Output:
score (Float) or scores (List of Float)
The multi_class parameter defines the output of this function.
If n_classes > 2, multi_class is automatically True.
If multi_class == False & n_classes == 2, only a single score (float) is returned.
If multi_class == True, multiple scores as a list are returned (for each class one score).
"""
def evaluate(truth, pred, metric, multi_class=False, n_classes=2, **kwargs)
```
## Installation
- **Install miseval from PyPI (recommended):**
```sh
pip install miseval
```
- **Alternatively: install miseval from the GitHub source:**
First, clone miseval using git:
```sh
git clone https://github.com/frankkramer-lab/miseval
```
Then, go into the miseval folder and run the install command:
```sh
cd miseval
python setup.py install
```
## Author
Dominik Müller\
Email: dominik.mueller@informatik.uni-augsburg.de\
IT-Infrastructure for Translational Medical Research\
University Augsburg\
Bavaria, Germany
## How to cite / More information
Dominik Müller, Dennis Hartmann, Philip Meyer, Florian Auer, Iñaki Soto-Rey, Frank Kramer. (2022)
MISeval: a Metric Library for Medical Image Segmentation Evaluation.
PubMed: https://pubmed.ncbi.nlm.nih.gov/35612011/
DOI: https://doi.org/10.3233/shti220391
arXiv e-print: https://arxiv.org/abs/2201.09395
```
@Article{misevalMUELLER2022,
title={MISeval: a Metric Library for Medical Image Segmentation Evaluation},
author={Dominik Müller, Dennis Hartmann, Philip Meyer, Florian Auer, Iñaki Soto-Rey, Frank Kramer},
year={2022},
journal={Studies in health technology and informatics},
volume={294},
number={},
pages={33-37},
doi={10.3233/shti220391},
eprint={2201.09395},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
```
Thank you for citing our work.
## License
This project is licensed under the GNU GENERAL PUBLIC LICENSE Version 3.\
See the LICENSE.md file for license rights and limitations.
Raw data
{
"_id": null,
"home_page": "https://github.com/frankkramer-lab/miseval",
"name": "miseval",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": null,
"author": "Dominik M\u00fcller",
"author_email": "dominik.mueller@informatik.uni-augsburg.de",
"download_url": "https://files.pythonhosted.org/packages/5a/d6/010110f6b6be6158a34908c4ece1650aef38899d72c44d82d91a6430d5fa/miseval-1.3.0.tar.gz",
"platform": null,
"description": "# miseval: a metric library for Medical Image Segmentation EVALuation\n\n[](https://www.python.org/)\n[](https://github.com/frankkramer-lab/miseval)\n[](https://pypi.org/project/miseval/)\n[](https://pypistats.org/packages/miseval)\n[](https://www.gnu.org/licenses/gpl-3.0.en.html)\n\nThe open-source and free to use Python package miseval was developed to establish a standardized medical image segmentation evaluation procedure. We hope that our this will help improve evaluation quality, reproducibility, and comparability in future studies in the field of medical image segmentation.\n\n#### Guideline on Evaluation Metrics for \tMedical Image Segmentation\n\n1. Use DSC as main metric for validation and performance interpretation.\n2. Use AHD for interpretation on point position sensitivity (contour) if needed.\n3. Avoid any interpretations based on high pixel accuracy scores.\n4. Provide next to DSC also IoU, Sensitivity, and Specificity for method comparability.\n5. Provide sample visualizations, comparing the annotated and predicted segmentation, for visual evaluation as well as to avoid statistical bias.\n6. Avoid cherry-picking high-scoring samples.\n7. Provide histograms or box plots showing the scoring distribution across the dataset.\n8. For multi-class problems, provide metric computations for each class individually.\n9. Avoid confirmation bias through macro-averaging classes which is pushing scores via background class inclusion.\n10. Provide access to evaluation scripts and results with journal data services or third-party services like GitHub and Zenodo for easier reproducibility.\n\n## Implemented Metrics\n\n| Metric | Index in miseval | Function in miseval |\n| ----------- | ----------- | ----------- |\n| Dice Similarity Index | \"DSC\", \"Dice\", \"DiceSimilarityCoefficient\" | miseval.calc_DSC() |\n| Intersection-Over-Union | \"IoU\", \"Jaccard\", \"IntersectionOverUnion\" | miseval.calc_IoU() |\n| Sensitivity | \"SENS\", \"Sensitivity\", \"Recall\", \"TPR\", \"TruePositiveRate\" | miseval.calc_Sensitivity() |\n| Specificity | \"SPEC\", \"Specificity\", \"TNR\", \"TrueNegativeRate\" | miseval.calc_Specificity() |\n| Precision | \"PREC\", \"Precision\" | miseval.calc_Precision() |\n| Accuracy | \"ACC\", \"Accuracy\", \"RI\", \"RandIndex\" | miseval.calc_Accuracy() |\n| Balanced Accuracy | \"BACC\", \"BalancedAccuracy\" | miseval.calc_BalancedAccuracy() |\n| Adjusted Rand Index | \"ARI\", \"AdjustedRandIndex\" | miseval.calc_AdjustedRandIndex() |\n| AUC | \"AUC\", \"AUC_trapezoid\" | miseval.calc_AUC() |\n| Cohen's Kappa | \"KAP\", \"Kappa\", \"CohensKappa\" | miseval.calc_Kappa() |\n| Hausdorff Distance | \"HD\", \"HausdorffDistance\" | miseval.calc_SimpleHausdorffDistance() |\n| Average Hausdorff Distance | \"AHD\", \"AverageHausdorffDistance\" | miseval.calc_AverageHausdorffDistance() |\n| Volumetric Similarity | \"VS\", \"VolumetricSimilarity\" | miseval.calc_VolumetricSimilarity() |\n| Matthews Correlation Coefficient | \"MCC\", \"MatthewsCorrelationCoefficient\" | miseval.calc_MCC() |\n| Normalized Matthews Correlation Coefficient | \"nMCC\", \"MCC_normalized\" | miseval.calc_MCC_Normalized() |\n| Absolute Matthews Correlation Coefficient | \"aMCC\", \"MCC_absolute\" | miseval.calc_MCC_Absolute() |\n| Boundary Distance | \"BD\", \"Distance\", \" BoundaryDistance\" | miseval.calc_Boundary_Distance() |\n| Hinge Loss | \"Hinge\", \"HingeLoss\" | miseval.calc_Hinge() |\n| Cross-Entropy | \"CE\", \"CrossEntropy\" | miseval.calc_CrossEntropy() |\n| True Positive | \"TP\", \"TruePositive\" | miseval.calc_TruePositive() |\n| False Positive | \"FP\", \"FalsePositive\" | miseval.calc_FalsePositive() |\n| True Negative | \"TN\", \"TrueNegative\" | miseval.calc_TrueNegative() |\n| False Negative | \"FN\", \"FalseNegative\" | miseval.calc_FalseNegative() |\n\n#### Options for Boundary Distance computation\n\n```\nList of available distances:\n Bhattacharyya distance bhattacharyya\n Bhattacharyya coefficient bhattacharyya_coefficient\n Canberra distance canberra\n Chebyshev distance chebyshev\n Chi Square distance chi_square\n Cosine Distance cosine\n Euclidean distance euclidean\n Hamming distance hamming\n Jensen-Shannon divergence jensen_shannon\n Kullback-Leibler divergence kullback_leibler\n Mean absolute error mae\n Taxicab geometry manhattan, cityblock, total_variation\n Minkowski distance minkowsky\n Mean squared error mse\n Pearson's distance pearson\n Squared deviations from the mean squared_variation\n\nDistance Pooling (how to combine computed distances to a single value):\n Distance Sum sum\n Distance Averaging mean\n Minimum Distance amin\n Maximum Distance amax\n```\n\n## How to Use\n\n#### Example\n\n```python\n# load libraries\nimport numpy as np\nfrom miseval import evaluate\n\n# Get some ground truth / annotated segmentations\nnp.random.seed(1)\nreal_bi = np.random.randint(2, size=(64,64)) # binary (2 classes)\nreal_mc = np.random.randint(5, size=(64,64)) # multi-class (5 classes)\n# Get some predicted segmentations\nnp.random.seed(2)\npred_bi = np.random.randint(2, size=(64,64)) # binary (2 classes)\npred_mc = np.random.randint(5, size=(64,64)) # multi-class (5 classes)\n\n# Run binary evaluation\ndice = evaluate(real_bi, pred_bi, metric=\"DSC\") \n # returns single np.float64 e.g. 0.75\n\n# Run multi-class evaluation\ndice_list = evaluate(real_mc, pred_mc, metric=\"DSC\", multi_class=True,\n n_classes=5) \n # returns array of np.float64 e.g. [0.9, 0.2, 0.6, 0.0, 0.4]\n # for each class, one score\n```\n\n#### Core function: Evaluate()\n\nEvery metric in miseval can be called via our core function `evaluate()`.\n\nThe miseval eavluate function can be run with different metrics as backbone. \nYou can pass the following options to the metric parameter:\n- String naming one of the metric labels, for example \"DSC\"\n- Directly passing a metric function, for example calc_DSC_Sets (from dice.py)\n- Passing a custom metric function\n\nList of metrics : See `miseval/__init__.py` under section \"Access Functions to Metric Functions\"\n\nThe classes in a segmentation mask must be ongoing starting from 0 (integers from 0 to n_classes-1).\n\nA segmentation mask is allowed to have either no channel axis or just 1 (e.g. 512x512x1),\nwhich contains the annotation. \n\n```python\n\"\"\"\nArguments:\n truth (NumPy Matrix): Ground Truth segmentation mask.\n pred (NumPy Matrix): Prediction segmentation mask.\n metric (String or Function): Metric function. Either a function directly or encoded as\n String from miseval or a custom function.\n multi_class (Boolean): Boolean parameter, if segmentation is a binary or multi-class\n problem. By default False -> Binary mode.\n n_classes (Integer): Number of classes. By default 2 -> Binary\n kwargs (arguments): Additional arguments for passing down to metric functions.\n\nOutput:\n score (Float) or scores (List of Float)\n\n The multi_class parameter defines the output of this function.\n If n_classes > 2, multi_class is automatically True.\n If multi_class == False & n_classes == 2, only a single score (float) is returned.\n If multi_class == True, multiple scores as a list are returned (for each class one score).\n\"\"\"\ndef evaluate(truth, pred, metric, multi_class=False, n_classes=2, **kwargs)\n```\n\n## Installation\n\n\n- **Install miseval from PyPI (recommended):**\n\n```sh\npip install miseval\n```\n\n- **Alternatively: install miseval from the GitHub source:**\n\nFirst, clone miseval using git:\n\n```sh\ngit clone https://github.com/frankkramer-lab/miseval\n```\n\nThen, go into the miseval folder and run the install command:\n\n```sh\ncd miseval\npython setup.py install\n```\n\n## Author\n\nDominik M\u00fcller\\\nEmail: dominik.mueller@informatik.uni-augsburg.de\\\nIT-Infrastructure for Translational Medical Research\\\nUniversity Augsburg\\\nBavaria, Germany\n\n## How to cite / More information\n\nDominik M\u00fcller, Dennis Hartmann, Philip Meyer, Florian Auer, I\u00f1aki Soto-Rey, Frank Kramer. (2022) \nMISeval: a Metric Library for Medical Image Segmentation Evaluation. \nPubMed: https://pubmed.ncbi.nlm.nih.gov/35612011/ \nDOI: https://doi.org/10.3233/shti220391 \narXiv e-print: https://arxiv.org/abs/2201.09395 \n\n```\n@Article{misevalMUELLER2022,\n title={MISeval: a Metric Library for Medical Image Segmentation Evaluation},\n author={Dominik M\u00fcller, Dennis Hartmann, Philip Meyer, Florian Auer, I\u00f1aki Soto-Rey, Frank Kramer},\n year={2022},\n journal={Studies in health technology and informatics},\n volume={294},\n number={},\n pages={33-37},\n doi={10.3233/shti220391},\n eprint={2201.09395},\n archivePrefix={arXiv},\n primaryClass={cs.CV}\n}\n```\n\nThank you for citing our work.\n\n## License\n\nThis project is licensed under the GNU GENERAL PUBLIC LICENSE Version 3.\\\nSee the LICENSE.md file for license rights and limitations.\n",
"bugtrack_url": null,
"license": "GPLv3",
"summary": "A Metric Library for Medical Image Segmentation Evaluation",
"version": "1.3.0",
"project_urls": {
"Homepage": "https://github.com/frankkramer-lab/miseval"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "3cdcce8ade3c2d8a810fb77820af1b3f0d18e3ef28e087c51d28f4f7ecf4d5d7",
"md5": "d18db706ab2bf065ce9e4857f3594222",
"sha256": "6bd953e2ba8a375094efa4153d74b0af2458e0345ded9744bbdaa75be79e2e69"
},
"downloads": -1,
"filename": "miseval-1.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d18db706ab2bf065ce9e4857f3594222",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 60962,
"upload_time": "2024-10-10T19:08:14",
"upload_time_iso_8601": "2024-10-10T19:08:14.413603Z",
"url": "https://files.pythonhosted.org/packages/3c/dc/ce8ade3c2d8a810fb77820af1b3f0d18e3ef28e087c51d28f4f7ecf4d5d7/miseval-1.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "5ad6010110f6b6be6158a34908c4ece1650aef38899d72c44d82d91a6430d5fa",
"md5": "b52cf34406a09e4589249e08719e4b06",
"sha256": "12417417e054a676a78f0733d280b0b3d4f070e4267d435f9d45160bcbf07e73"
},
"downloads": -1,
"filename": "miseval-1.3.0.tar.gz",
"has_sig": false,
"md5_digest": "b52cf34406a09e4589249e08719e4b06",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 34566,
"upload_time": "2024-10-10T19:05:12",
"upload_time_iso_8601": "2024-10-10T19:05:12.342480Z",
"url": "https://files.pythonhosted.org/packages/5a/d6/010110f6b6be6158a34908c4ece1650aef38899d72c44d82d91a6430d5fa/miseval-1.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-10 19:05:12",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "frankkramer-lab",
"github_project": "miseval",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "miseval"
}
JSON
