[//]: # ()
<a href="https://github.com/caumente/AUDIT" title="Go to GitHub repo"><img src="https://img.shields.io/static/v1?label=caumente&message=AUDIT&color=e78ac3&logo=github" alt="caumente - AUDIT"></a>
<a href="https://github.com/caumente/AUDIT"><img src="https://img.shields.io/github/stars/caumente/AUDIT?style=social" alt="stars - AUDIT"></a>
<a href="https://github.com/caumente/AUDIT"><img src="https://img.shields.io/github/forks/caumente/AUDIT?style=social" alt="forks - AUDIT"></a>
<a href="https://github.com/caumente/audit/releases/"><img src="https://img.shields.io/github/release/caumente/audit?include_prereleases=&sort=semver&color=e78ac3" alt="GitHub release"></a>
<a href="#license"><img src="https://img.shields.io/badge/License-Apache_2.0-e78ac3" alt="License"></a>
<a href="https://github.com/caumente/audit/issues"><img src="https://img.shields.io/github/issues/caumente/audit" alt="issues - AUDIT"></a>
## Summary
AUDIT, Analysis & evalUation Dashboard of artIficial inTelligence, is a tool designed to analyze,
visualize, and detect biases in brain MRI data and models. It provides tools for loading and processing MRI data,
extracting relevant features, and visualizing model performance and biases in predictions. AUDIT presents the
following features:
- **Data management**: Easily work and preprocess MRIs from various sources.
- **Feature extraction**: Extract relevant features from the images and their segmentations for analysis.
- **Model robustness**: Assess model generalization by evaluating its performance across several experiments
and conditions.
- **Bias detection**: Identify potential biases either in model predictions and performance or on your data.
- **Longitudinal analysis**: Track the model performance over different time points.
- **High compatibility**: Provides connection with tools like ITK-SNAP and other external tools.
Details of our work are provided in [*our paper*](...........), **AUDIT**. We hope that
users will use *AUDIT* to gain novel insights into brain tumor segmentation field.
## Usage
- **Home Page**: The main landing page of the tool.
- **Univariate Analysis**: Exploration of individual variables to understand their distributions and discover
outliers in it.
- **Multivariate Analysis**: Examination of multiple variables simultaneously to explore relationships and
hidden patterns.
- **Segmentation Error Matrix**: A pseudo-confusion matrix displaying the errors associated with the
segmentation tasks.
- **Model Performance Analysis**: Evaluation of the effectiveness and accuracy of a single model.
- **Pairwise Model Performance Comparison**: Perform pair-wise comparisons between models to find statistical
significant differences.
- **Multi-model Performance Comparison**: Comparative analysis of performance metrics across multiple models.
- **Longitudinal Measurements**: Analysis of data collected over time to observe trends and changes on model
accuracy.
- **Subjects Exploration**: Detailed examination of individual subjects within the dataset.
## Web AUDIT
Last released version of **AUDIT** is hosted at https://auditapp.streamlitapp.com for an online overview of its functionalities.
## Getting Started
For a more detailed exploration of AUDIT, please check our [*official documentation*](https://github.com/caumente/AUDIT)
### 1 Installation
Create an isolated Anaconda environment:
```bash
conda create -n audit_env python=3.10
conda activate audit_env
```
Clone the repository:
```bash
git clone https://github.com/caumente/AUDIT.git
cd AUDIT
```
Install the required packages:
```bash
pip install -r requirements.txt
```
### 2. Configuration
Edit the config files in `./src/audit/configs/` directory to set up the paths for data loading and other configurations:
<details>
<summary><strong>2.1. Feature extractor config</strong></summary>
```yaml
# Paths to all the datasets
data_paths:
dataset_1: '/home/user/AUDIT/datasets/dataset_1/dataset_1_images'
dataset_N: '/home/user/AUDIT/datasets/dataset_N/dataset_N_images'
# Sequences available
sequences:
- '_t1'
- '_t2'
- '_t1ce'
- '_flair'
# Mapping of labels to their numeric values
labels:
BKG: 0
EDE: 3
ENH: 1
NEC: 2
# List of features to extract
features:
statistical: true
texture: false
spatial: false
tumor: false
# Longitudinal study settings
#longitudinal:
# dataset_N:
# pattern: "_" # Pattern used for splitting filename
# longitudinal_id: 1 # Index position for the subject ID after splitting the filename
# time_point: 2 # Index position for the time point after splitting the filename
# Path where extracted features will be saved
output_path: '/home/usr/AUDIT/outputs/features'
logs_path: '/home/usr/AUDIT/logs/features'
```
</details>
<details>
<summary><strong>2.2. Metric extractor config</strong></summary>
```yaml
# Path to the raw dataset
data_path: '/home/carlos/AUDIT/datasets/dataset_1/dataset_1_images'
# Paths to model predictions
model_predictions_paths:
model_1: '/home/user/AUDIT/datasets/dataset_1/dataset_1_seg/model_1'
model_M: '/home/user/AUDIT/datasets/dataset_1/dataset_1_seg/model_M'
# Mapping of labels to their numeric values
labels:
BKG: 0
EDE: 3
ENH: 1
NEC: 2
# List of metrics to compute
metrics:
dice: true
jacc: false
accu: false
prec: false
sens: false
spec: false
haus: false
# Library used for computing all the metrics
package: custom
calculate_stats: false
# Path where output metrics will be saved
output_path: '/home/user/AUDIT/outputs/metrics'
filename: 'LUMIERE'
logs_path: '/home/user/AUDIT/logs/metric'
```
</details>
<details>
<summary><strong>2.3. APP config</strong></summary>
```yaml
# Sequences available. First of them will be used to compute properties like spacing
sequences:
- '_t1'
- '_t2'
- '_t1ce'
- '_flair'
# Mapping of labels to their numeric values
labels:
BKG: 0
EDE: 3
ENH: 1
NEC: 2
# Root path for datasets, features extracted, and metrics extracted
datasets_path: '/home/user/AUDIT/datasets'
features_path: '/home/user/AUDIT/outputs/features'
metrics_path: '/home/user/AUDIT/outputs/metrics'
# Paths for raw datasets
raw_datasets:
dataset_1: "${datasets_path}/dataset_1/dataset_1_images"
dataset_N: "${datasets_path}/dataset_N/dataset_N_images"
# Paths for feature extraction CSV files
features:
dataset_1: "${features_path}/extracted_information_dataset_1.csv"
dataset_N: "${features_path}/extracted_information_dataset_N.csv"
# Paths for metric extraction CSV files
metrics:
dataset_1: "${metrics_path}/extracted_information_dataset_1.csv"
dataset_N: "${metrics_path}/extracted_information_dataset_N.csv"
# Paths for models predictions
predictions:
dataset_1:
model_1: "${datasets_path}/dataset_1/dataset_1_seg/model_1"
model_M: "${datasets_path}/dataset_1/dataset_1_seg/model_M"
dataset_N:
model_1: "${datasets_path}/dataset_N/dataset_N_seg/model_1"
model_M: "${datasets_path}/dataset_N/dataset_N_seg/model_M"
```
</details>
### 3. Run AUDIT backend
Use the following commands to run the *Feature extractor* and *Metric extractor* scripts:
```bash
python src/audit/feature_extractor.py
```
```bash
python src/audit/metric_extractor.py
```
A _logs_ folder will be created after running each of the scripts to keep track of the execution. All the output files
will be stored in the folder defined in the corresponding config file (by default in the _output_ folder).
### 4. Run AUDIT app
Use the following streamlit command to run the APP and start the data exploration:
```bash
python python src/audit/app/launcher.py
```
### 5. Additional configurations
#### 5.1. ITK-Snap
AUDIT is prepared for opening cases with ITK-Snap while exploring the data in the different dashboards. However, the
ITK-Snap tool must have been installed and preconfigured before. Here we provide a simple necessary configuration to
use it in each operative system:
<details>
<summary><strong>5.1.1. On Mac OS</strong></summary>
```bash
```
</details>
<details>
<summary><strong>5.1.2. On Linux OS</strong></summary>
```bash
```
</details>
## Authors
Please feel free to contact us with any issues, comments, or questions.
#### Carlos Aumente
- Email: <UO297103@uniovi.es>
- GitHub: https://github.com/caumente
#### Mauricio Reyes
#### Michael Muller
#### Jorge Díez
#### Beatriz Remeseiro
## License
Apache License 2.0
Raw data
{
"_id": null,
"home_page": "https://github.com/caumente/AUDIT",
"name": "auditapp",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.10",
"maintainer_email": null,
"keywords": "medical image analysis, deep learning, mri, model evaluation, dashboard",
"author": "Carlos Aumente",
"author_email": "carlosaumente@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/be/1f/9dcf5589c7eaf2e2f85735aa49e5fa280c8867521a7c0248752aec079241/auditapp-0.0.10.tar.gz",
"platform": null,
"description": "\n[//]: # ()\n\n\n\n<a href=\"https://github.com/caumente/AUDIT\" title=\"Go to GitHub repo\"><img src=\"https://img.shields.io/static/v1?label=caumente&message=AUDIT&color=e78ac3&logo=github\" alt=\"caumente - AUDIT\"></a>\n<a href=\"https://github.com/caumente/AUDIT\"><img src=\"https://img.shields.io/github/stars/caumente/AUDIT?style=social\" alt=\"stars - AUDIT\"></a>\n<a href=\"https://github.com/caumente/AUDIT\"><img src=\"https://img.shields.io/github/forks/caumente/AUDIT?style=social\" alt=\"forks - AUDIT\"></a>\n\n\n<a href=\"https://github.com/caumente/audit/releases/\"><img src=\"https://img.shields.io/github/release/caumente/audit?include_prereleases=&sort=semver&color=e78ac3\" alt=\"GitHub release\"></a>\n<a href=\"#license\"><img src=\"https://img.shields.io/badge/License-Apache_2.0-e78ac3\" alt=\"License\"></a>\n<a href=\"https://github.com/caumente/audit/issues\"><img src=\"https://img.shields.io/github/issues/caumente/audit\" alt=\"issues - AUDIT\"></a>\n\n\n## Summary\n\nAUDIT, Analysis & evalUation Dashboard of artIficial inTelligence, is a tool designed to analyze,\nvisualize, and detect biases in brain MRI data and models. It provides tools for loading and processing MRI data,\nextracting relevant features, and visualizing model performance and biases in predictions. AUDIT presents the \nfollowing features:\n\n\n- **Data management**: Easily work and preprocess MRIs from various sources.\n- **Feature extraction**: Extract relevant features from the images and their segmentations for analysis.\n- **Model robustness**: Assess model generalization by evaluating its performance across several experiments\n and conditions.\n- **Bias detection**: Identify potential biases either in model predictions and performance or on your data.\n- **Longitudinal analysis**: Track the model performance over different time points.\n- **High compatibility**: Provides connection with tools like ITK-SNAP and other external tools.\n\nDetails of our work are provided in [*our paper*](...........), **AUDIT**. We hope that \nusers will use *AUDIT* to gain novel insights into brain tumor segmentation field. \n\n\n## Usage\n- **Home Page**: The main landing page of the tool.\n- **Univariate Analysis**: Exploration of individual variables to understand their distributions and discover\n outliers in it.\n- **Multivariate Analysis**: Examination of multiple variables simultaneously to explore relationships and\n hidden patterns.\n- **Segmentation Error Matrix**: A pseudo-confusion matrix displaying the errors associated with the\n segmentation tasks.\n- **Model Performance Analysis**: Evaluation of the effectiveness and accuracy of a single model.\n- **Pairwise Model Performance Comparison**: Perform pair-wise comparisons between models to find statistical\n significant differences.\n- **Multi-model Performance Comparison**: Comparative analysis of performance metrics across multiple models.\n- **Longitudinal Measurements**: Analysis of data collected over time to observe trends and changes on model\n accuracy.\n- **Subjects Exploration**: Detailed examination of individual subjects within the dataset.\n\n## Web AUDIT\n\nLast released version of **AUDIT** is hosted at https://auditapp.streamlitapp.com for an online overview of its functionalities.\n\n\n## Getting Started\n\nFor a more detailed exploration of AUDIT, please check our [*official documentation*](https://github.com/caumente/AUDIT)\n\n### 1 Installation \n\nCreate an isolated Anaconda environment:\n\n```bash\nconda create -n audit_env python=3.10\nconda activate audit_env\n```\n\nClone the repository:\n ```bash\n git clone https://github.com/caumente/AUDIT.git\n cd AUDIT\n ```\n\nInstall the required packages:\n ```bash\n pip install -r requirements.txt\n ```\n\n### 2. Configuration\n\nEdit the config files in `./src/audit/configs/` directory to set up the paths for data loading and other configurations:\n\n\n<details>\n <summary><strong>2.1. Feature extractor config</strong></summary>\n\n```yaml\n# Paths to all the datasets\ndata_paths:\n dataset_1: '/home/user/AUDIT/datasets/dataset_1/dataset_1_images'\n dataset_N: '/home/user/AUDIT/datasets/dataset_N/dataset_N_images'\n\n# Sequences available\nsequences:\n - '_t1'\n - '_t2'\n - '_t1ce'\n - '_flair'\n\n# Mapping of labels to their numeric values\nlabels:\n BKG: 0\n EDE: 3\n ENH: 1\n NEC: 2\n\n# List of features to extract\nfeatures:\n statistical: true\n texture: false\n spatial: false\n tumor: false\n\n# Longitudinal study settings\n#longitudinal:\n# dataset_N:\n# pattern: \"_\" # Pattern used for splitting filename\n# longitudinal_id: 1 # Index position for the subject ID after splitting the filename\n# time_point: 2 # Index position for the time point after splitting the filename\n\n\n# Path where extracted features will be saved\noutput_path: '/home/usr/AUDIT/outputs/features'\nlogs_path: '/home/usr/AUDIT/logs/features'\n```\n</details>\n\n\n<details>\n <summary><strong>2.2. Metric extractor config</strong></summary>\n\n```yaml\n# Path to the raw dataset\ndata_path: '/home/carlos/AUDIT/datasets/dataset_1/dataset_1_images'\n\n# Paths to model predictions\nmodel_predictions_paths:\n model_1: '/home/user/AUDIT/datasets/dataset_1/dataset_1_seg/model_1'\n model_M: '/home/user/AUDIT/datasets/dataset_1/dataset_1_seg/model_M'\n\n# Mapping of labels to their numeric values\nlabels:\n BKG: 0\n EDE: 3\n ENH: 1\n NEC: 2\n\n# List of metrics to compute\nmetrics:\n dice: true\n jacc: false\n accu: false\n prec: false\n sens: false\n spec: false\n haus: false\n\n# Library used for computing all the metrics\npackage: custom\ncalculate_stats: false\n\n# Path where output metrics will be saved\noutput_path: '/home/user/AUDIT/outputs/metrics'\nfilename: 'LUMIERE'\nlogs_path: '/home/user/AUDIT/logs/metric'\n```\n</details>\n\n\n<details>\n <summary><strong>2.3. APP config</strong></summary>\n\n```yaml\n# Sequences available. First of them will be used to compute properties like spacing\nsequences:\n - '_t1'\n - '_t2'\n - '_t1ce'\n - '_flair'\n\n# Mapping of labels to their numeric values\nlabels:\n BKG: 0\n EDE: 3\n ENH: 1\n NEC: 2\n\n# Root path for datasets, features extracted, and metrics extracted\ndatasets_path: '/home/user/AUDIT/datasets'\nfeatures_path: '/home/user/AUDIT/outputs/features'\nmetrics_path: '/home/user/AUDIT/outputs/metrics'\n\n# Paths for raw datasets\nraw_datasets:\n dataset_1: \"${datasets_path}/dataset_1/dataset_1_images\"\n dataset_N: \"${datasets_path}/dataset_N/dataset_N_images\"\n\n# Paths for feature extraction CSV files\nfeatures:\n dataset_1: \"${features_path}/extracted_information_dataset_1.csv\"\n dataset_N: \"${features_path}/extracted_information_dataset_N.csv\"\n\n# Paths for metric extraction CSV files\nmetrics:\n dataset_1: \"${metrics_path}/extracted_information_dataset_1.csv\"\n dataset_N: \"${metrics_path}/extracted_information_dataset_N.csv\"\n\n# Paths for models predictions\npredictions:\n dataset_1:\n model_1: \"${datasets_path}/dataset_1/dataset_1_seg/model_1\"\n model_M: \"${datasets_path}/dataset_1/dataset_1_seg/model_M\"\n dataset_N:\n model_1: \"${datasets_path}/dataset_N/dataset_N_seg/model_1\"\n model_M: \"${datasets_path}/dataset_N/dataset_N_seg/model_M\"\n```\n</details>\n\n\n### 3. Run AUDIT backend\n\nUse the following commands to run the *Feature extractor* and *Metric extractor* scripts:\n\n```bash\npython src/audit/feature_extractor.py\n```\n\n```bash\npython src/audit/metric_extractor.py\n```\n\nA _logs_ folder will be created after running each of the scripts to keep track of the execution. All the output files \nwill be stored in the folder defined in the corresponding config file (by default in the _output_ folder).\n\n### 4. Run AUDIT app\n\nUse the following streamlit command to run the APP and start the data exploration:\n\n```bash\npython python src/audit/app/launcher.py\n```\n\n### 5. Additional configurations\n\n#### 5.1. ITK-Snap\n\nAUDIT is prepared for opening cases with ITK-Snap while exploring the data in the different dashboards. However, the \nITK-Snap tool must have been installed and preconfigured before. Here we provide a simple necessary configuration to \nuse it in each operative system:\n\n<details>\n <summary><strong>5.1.1. On Mac OS</strong></summary>\n\n```bash\n```\n</details>\n\n\n<details>\n <summary><strong>5.1.2. On Linux OS</strong></summary>\n\n```bash\n```\n</details>\n\n\n\n## Authors\n\nPlease feel free to contact us with any issues, comments, or questions.\n\n#### Carlos Aumente \n\n- Email: <UO297103@uniovi.es>\n- GitHub: https://github.com/caumente\n\n#### Mauricio Reyes \n#### Michael Muller \n#### Jorge D\u00edez \n#### Beatriz Remeseiro \n\n## License\nApache License 2.0\n\n\n\n\n\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "AUDIT, Analysis & evalUation Dashboard of artIficial inTelligence",
"version": "0.0.10",
"project_urls": {
"Documentation": "https://caumente.github.io/AUDIT/",
"Homepage": "https://github.com/caumente/AUDIT",
"Repository": "https://github.com/caumente/AUDIT"
},
"split_keywords": [
"medical image analysis",
" deep learning",
" mri",
" model evaluation",
" dashboard"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8baf546a2533c22beec77e682898587693288580e903b22d3977c97cecd68385",
"md5": "8e97d79ce5ccfedd5ba58f66cc408e09",
"sha256": "5e4b2d2180332f2c6a605427aa614d83e4593677d6d952b69c6009828f27cd6c"
},
"downloads": -1,
"filename": "auditapp-0.0.10-py3-none-any.whl",
"has_sig": false,
"md5_digest": "8e97d79ce5ccfedd5ba58f66cc408e09",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.10",
"size": 505609,
"upload_time": "2025-01-18T10:52:51",
"upload_time_iso_8601": "2025-01-18T10:52:51.552652Z",
"url": "https://files.pythonhosted.org/packages/8b/af/546a2533c22beec77e682898587693288580e903b22d3977c97cecd68385/auditapp-0.0.10-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "be1f9dcf5589c7eaf2e2f85735aa49e5fa280c8867521a7c0248752aec079241",
"md5": "73351336d39bf2499d9e16c692b95aed",
"sha256": "202fec78c9e6102f5cb9ec7c8e6958f37461fa4579e9e478366ea6f4e021db81"
},
"downloads": -1,
"filename": "auditapp-0.0.10.tar.gz",
"has_sig": false,
"md5_digest": "73351336d39bf2499d9e16c692b95aed",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.10",
"size": 490224,
"upload_time": "2025-01-18T10:52:55",
"upload_time_iso_8601": "2025-01-18T10:52:55.192536Z",
"url": "https://files.pythonhosted.org/packages/be/1f/9dcf5589c7eaf2e2f85735aa49e5fa280c8867521a7c0248752aec079241/auditapp-0.0.10.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-18 10:52:55",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "caumente",
"github_project": "AUDIT",
"github_not_found": true,
"lcname": "auditapp"
}
JSON
