SUOD: Accelerating Large-scare Unsupervised Heterogeneous Outlier Detection
===========================================================================
**Deployment & Documentation & Stats**
.. image:: https://img.shields.io/pypi/v/suod.svg?color=brightgreen
:target: https://pypi.org/project/suod/
:alt: PyPI version
.. image:: https://readthedocs.org/projects/suod/badge/?version=latest
:target: https://suod.readthedocs.io/en/latest/?badge=latest
:alt: Documentation Status
.. image:: https://img.shields.io/github/stars/yzhao062/suod.svg
:target: https://github.com/yzhao062/suod/stargazers
:alt: GitHub stars
.. image:: https://img.shields.io/github/forks/yzhao062/suod.svg?color=blue
:target: https://github.com/yzhao062/suod/network
:alt: GitHub forks
.. image:: https://pepy.tech/badge/suod
:target: https://pepy.tech/project/suod
:alt: Downloads
.. image:: https://pepy.tech/badge/suod/month
:target: https://pepy.tech/project/suod
:alt: Downloads
----
**Build Status & Coverage & Maintainability & License**
.. image:: https://github.com/yzhao062/suod/actions/workflows/testing.yml/badge.svg
:target: https://github.com/yzhao062/suod/actions/workflows/testing.yml
:alt: testing
.. image:: https://coveralls.io/repos/github/yzhao062/SUOD/badge.svg
:target: https://coveralls.io/github/yzhao062/SUOD
:alt: Coverage Status
.. image:: https://img.shields.io/github/license/yzhao062/suod.svg
:target: https://github.com/yzhao062/suod/blob/master/LICENSE
:alt: License
----
**News**: SUOD is now integrated into `PyOD <https://github.com/yzhao062/pyod>`_.
It can be easily invoked in PyOD by following the `SUOD example <https://github.com/yzhao062/pyod/blob/master/examples/suod_example.py>`_.
In a nutshell, we could easily initialize a few outlier detectors and then use SUOD for collective training and prediction!
.. code-block:: python
from pyod.models.suod import SUOD
# initialized a group of outlier detectors for acceleration
detector_list = [LOF(n_neighbors=15), LOF(n_neighbors=20),
LOF(n_neighbors=25), LOF(n_neighbors=35),
COPOD(), IForest(n_estimators=100),
IForest(n_estimators=200)]
# decide the number of parallel process, and the combination method
# then clf can be used as any outlier detection model
clf = SUOD(base_estimators=detector_list, n_jobs=2, combination='average',
verbose=False)
----
**Background**: Outlier detection (OD) is a key data mining task for identifying abnormal objects from general samples with numerous high-stake applications including fraud detection and intrusion detection.
Due to the lack of ground truth labels, practitioners often have to build a large number of unsupervised models that are heterogeneous (i.e., different algorithms and hyperparameters) for further combination and analysis with ensemble learning, rather than relying on a single model.
However, **this yields severe scalability issues on high-dimensional, large datasets**.
**SUOD** (**S**\calable **U**\nsupervised **O**\utlier **D**\etection) is an **acceleration framework for large-scale unsupervised heterogeneous outlier detector training and prediction**.
It focuses on three complementary aspects to accelerate (dimensionality reduction for high-dimensional data, model approximation for complex models, and execution efficiency improvement for taskload imbalance within distributed systems), while controlling detection performance degradation.
Since its inception in Sep 2019, SUOD has been successfully used in various academic researches and industry applications with more than 700,000 downloads,
including PyOD [#Zhao2019PyOD]_ and `IQVIA <https://www.iqvia.com/>`_ medical claim analysis.
.. image:: https://raw.githubusercontent.com/yzhao062/SUOD/master/figs/system_overview.png
:target: https://raw.githubusercontent.com/yzhao062/SUOD/master/figs/system_overview.png
:alt: SUOD System
SUOD is featured for:
* **Unified APIs, detailed documentation, and examples** for the easy use.
* **Optimized performance with JIT and parallelization** when possible, using `numba <https://github.com/numba/numba>`_ and `joblib <https://github.com/joblib/joblib>`_.
* **Fully compatible with the models in PyOD**.
* **Customizable modules and flexible design**: each module may be turned on/off or totally replaced by custom functions.
Roadmap:
* Provide more choices of distributed schedulers (adapted for SUOD), e.g., batch sampling, Sparrow (SOSP'13), Pigeon (SoCC'19) etc.
* Enable the flexibility of selecting data projection methods.
----
**API Demo**\ :
.. code-block:: python
from suod.models.base import SUOD
# initialize a set of base outlier detectors to train and predict on
base_estimators = [
LOF(n_neighbors=5, contamination=contamination),
LOF(n_neighbors=15, contamination=contamination),
LOF(n_neighbors=25, contamination=contamination),
HBOS(contamination=contamination),
PCA(contamination=contamination),
OCSVM(contamination=contamination),
KNN(n_neighbors=5, contamination=contamination),
KNN(n_neighbors=15, contamination=contamination),
KNN(n_neighbors=25, contamination=contamination)]
# initialize a SUOD model with all features turned on
model = SUOD(base_estimators=base_estimators, n_jobs=6, # number of workers
rp_flag_global=True, # global flag for random projection
bps_flag=True, # global flag for balanced parallel scheduling
approx_flag_global=False, # global flag for model approximation
contamination=contamination)
model.fit(X_train) # fit all models with X
model.approximate(X_train) # conduct model approximation if it is enabled
predicted_labels = model.predict(X_test) # predict labels
predicted_scores = model.decision_function(X_test) # predict scores
predicted_probs = model.predict_proba(X_test) # predict outlying probability
----
`The corresponding paper <https://www.andrew.cmu.edu/user/yuezhao2/papers/21-mlsys-suod.pdf>`_ is published in Conference on Machine Learning Systems (MLSys).
See https://mlsys.org/ for more information.
If you use SUOD in a scientific publication, we would appreciate citations to the following paper::
@inproceedings{zhao2021suod,
title={SUOD: Accelerating Large-scale Unsupervised Heterogeneous Outlier Detection},
author={Zhao, Yue and Hu, Xiyang and Cheng, Cheng and Wang, Cong and Wan, Changlin and Wang, Wen and Yang, Jianing and Bai, Haoping and Li, Zheng and Xiao, Cao and others},
journal={Proceedings of Machine Learning and Systems},
year={2021}
}
::
Zhao, Y., Hu, X., Cheng, C., Wang, C., Wan, C., Wang, W., Yang, J., Bai, H., Li, Z., Xiao, C. and Wang, Y., 2021. SUOD: Accelerating Large-scale Unsupervised Heterogeneous Outlier Detection. Proceedings of Machine Learning and Systems (MLSys).
**Table of Contents**\ :
* `Installation <#installation>`_
* `API Cheatsheet & Reference <#api-cheatsheet--reference>`_
* `Examples <#examples>`_
* `Model Save & Load <#model-save--load>`_
------------
Installation
^^^^^^^^^^^^
It is recommended to use **pip** for installation. Please make sure
**the latest version** is installed, as suod is updated frequently:
.. code-block:: bash
pip install suod # normal install
pip install --upgrade suod # or update if needed
pip install --pre suod # or include pre-release version for new features
Alternatively, you could clone and run setup.py file:
.. code-block:: bash
git clone https://github.com/yzhao062/suod.git
cd suod
pip install .
**Required Dependencies**\ :
* Python 3.8+
* joblib
* numpy>=1.13
* pandas (**optional for building the cost forecast model**)
* pyod
* scipy>=0.19.1
* scikit_learn>=1.0
------------
API Cheatsheet & Reference
^^^^^^^^^^^^^^^^^^^^^^^^^^
Full API Reference: (https://suod.readthedocs.io/en/latest/api.html).
* **fit(X, y)**\ : Fit estimator. y is optional for unsupervised methods.
* **approximate(X)**\ : Use supervised models to approximate unsupervised base detectors. Fit should be invoked first.
* **predict(X)**\ : Predict on a particular sample once the estimator is fitted.
* **predict_proba(X)**\ : Predict the probability of a sample belonging to each class once the estimator is fitted.
Examples
^^^^^^^^
All three modules can be **executed separately** and the demo codes are in /examples/module_examples/{M1_RP, M2_BPS, and M3_PSA}.
For instance, you could navigate to /M1_RP/demo_random_projection.py. Demo codes all start with "demo_*.py".
**The examples for the full framework** can be found under /examples folder; run "demo_base.py" for
a simplified example. Run "demo_full.py" for a full example.
It is noted the best performance may be achieved with multiple cores available.
------------
Model Save & Load
^^^^^^^^^^^^^^^^^
SUOD takes a similar approach of sklearn regarding model persistence.
See `model persistence <https://scikit-learn.org/stable/modules/model_persistence.html>`_ for clarification.
In short, we recommend to use joblib or pickle for saving and loading SUOD models.
See `"examples/demo_model_save_load.py" <https://github.com/yzhao062/suod/blob/master/examples/demo_model_save_load.py>`_ for an example.
In short, it is simple as below:
.. code-block:: python
from joblib import dump, load
# save the fitted model
dump(model, 'model.joblib')
# load the model
model = load('model.joblib')
**More to come...**
Last updated on Jan 14th, 2021.
Feel free to star and watch for the future update :)
----
References
----------
.. [#Johnson1984Extensions] Johnson, W.B. and Lindenstrauss, J., 1984. Extensions of Lipschitz mappings into a Hilbert space. *Contemporary mathematics*, 26(189-206), p.1.
.. [#Zhao2019PyOD] Zhao, Y., Nasrullah, Z. and Li, Z., 2019. PyOD: A Python Toolbox for Scalable Outlier Detection. *Journal of Machine Learning Research*, 20, pp.1-7.
Raw data
{
"_id": null,
"home_page": "https://github.com/yzhao062/suod",
"name": "suod",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "ensemble learning,anomaly detection,outlier ensembles,data mining,machine learning,python",
"author": "Yue Zhao",
"author_email": "yzhao062@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/24/74/9fdfdb0b91100290aad865b88f8e0ba91a75d7c8343a432cdccb3bc00718/suod-0.1.3.tar.gz",
"platform": null,
"description": "SUOD: Accelerating Large-scare Unsupervised Heterogeneous Outlier Detection\r\n===========================================================================\r\n\r\n**Deployment & Documentation & Stats**\r\n\r\n.. image:: https://img.shields.io/pypi/v/suod.svg?color=brightgreen\r\n :target: https://pypi.org/project/suod/\r\n :alt: PyPI version\r\n\r\n\r\n.. image:: https://readthedocs.org/projects/suod/badge/?version=latest\r\n :target: https://suod.readthedocs.io/en/latest/?badge=latest\r\n :alt: Documentation Status\r\n\r\n\r\n.. image:: https://img.shields.io/github/stars/yzhao062/suod.svg\r\n :target: https://github.com/yzhao062/suod/stargazers\r\n :alt: GitHub stars\r\n\r\n\r\n.. image:: https://img.shields.io/github/forks/yzhao062/suod.svg?color=blue\r\n :target: https://github.com/yzhao062/suod/network\r\n :alt: GitHub forks\r\n\r\n\r\n.. image:: https://pepy.tech/badge/suod\r\n :target: https://pepy.tech/project/suod\r\n :alt: Downloads\r\n\r\n\r\n.. image:: https://pepy.tech/badge/suod/month\r\n :target: https://pepy.tech/project/suod\r\n :alt: Downloads\r\n\r\n\r\n----\r\n\r\n\r\n**Build Status & Coverage & Maintainability & License**\r\n\r\n\r\n.. image:: https://github.com/yzhao062/suod/actions/workflows/testing.yml/badge.svg\r\n :target: https://github.com/yzhao062/suod/actions/workflows/testing.yml\r\n :alt: testing\r\n\r\n.. image:: https://coveralls.io/repos/github/yzhao062/SUOD/badge.svg\r\n :target: https://coveralls.io/github/yzhao062/SUOD\r\n :alt: Coverage Status\r\n\r\n.. image:: https://img.shields.io/github/license/yzhao062/suod.svg\r\n :target: https://github.com/yzhao062/suod/blob/master/LICENSE\r\n :alt: License\r\n\r\n\r\n----\r\n\r\n**News**: SUOD is now integrated into `PyOD <https://github.com/yzhao062/pyod>`_.\r\nIt can be easily invoked in PyOD by following the `SUOD example <https://github.com/yzhao062/pyod/blob/master/examples/suod_example.py>`_.\r\nIn a nutshell, we could easily initialize a few outlier detectors and then use SUOD for collective training and prediction!\r\n\r\n.. code-block:: python\r\n\r\n from pyod.models.suod import SUOD\r\n\r\n # initialized a group of outlier detectors for acceleration\r\n detector_list = [LOF(n_neighbors=15), LOF(n_neighbors=20),\r\n LOF(n_neighbors=25), LOF(n_neighbors=35),\r\n COPOD(), IForest(n_estimators=100),\r\n IForest(n_estimators=200)]\r\n\r\n # decide the number of parallel process, and the combination method\r\n # then clf can be used as any outlier detection model\r\n clf = SUOD(base_estimators=detector_list, n_jobs=2, combination='average',\r\n verbose=False)\r\n\r\n\r\n----\r\n\r\n\r\n**Background**: Outlier detection (OD) is a key data mining task for identifying abnormal objects from general samples with numerous high-stake applications including fraud detection and intrusion detection.\r\nDue to the lack of ground truth labels, practitioners often have to build a large number of unsupervised models that are heterogeneous (i.e., different algorithms and hyperparameters) for further combination and analysis with ensemble learning, rather than relying on a single model.\r\nHowever, **this yields severe scalability issues on high-dimensional, large datasets**.\r\n\r\n**SUOD** (**S**\\calable **U**\\nsupervised **O**\\utlier **D**\\etection) is an **acceleration framework for large-scale unsupervised heterogeneous outlier detector training and prediction**.\r\nIt focuses on three complementary aspects to accelerate (dimensionality reduction for high-dimensional data, model approximation for complex models, and execution efficiency improvement for taskload imbalance within distributed systems), while controlling detection performance degradation.\r\n\r\nSince its inception in Sep 2019, SUOD has been successfully used in various academic researches and industry applications with more than 700,000 downloads,\r\nincluding PyOD [#Zhao2019PyOD]_ and `IQVIA <https://www.iqvia.com/>`_ medical claim analysis.\r\n\r\n\r\n.. image:: https://raw.githubusercontent.com/yzhao062/SUOD/master/figs/system_overview.png\r\n :target: https://raw.githubusercontent.com/yzhao062/SUOD/master/figs/system_overview.png\r\n :alt: SUOD System\r\n\r\nSUOD is featured for:\r\n\r\n* **Unified APIs, detailed documentation, and examples** for the easy use.\r\n* **Optimized performance with JIT and parallelization** when possible, using `numba <https://github.com/numba/numba>`_ and `joblib <https://github.com/joblib/joblib>`_.\r\n* **Fully compatible with the models in PyOD**.\r\n* **Customizable modules and flexible design**: each module may be turned on/off or totally replaced by custom functions.\r\n\r\nRoadmap:\r\n\r\n* Provide more choices of distributed schedulers (adapted for SUOD), e.g., batch sampling, Sparrow (SOSP'13), Pigeon (SoCC'19) etc.\r\n* Enable the flexibility of selecting data projection methods.\r\n\r\n----\r\n\r\n**API Demo**\\ :\r\n\r\n\r\n.. code-block:: python\r\n\r\n\r\n from suod.models.base import SUOD\r\n\r\n # initialize a set of base outlier detectors to train and predict on\r\n base_estimators = [\r\n LOF(n_neighbors=5, contamination=contamination),\r\n LOF(n_neighbors=15, contamination=contamination),\r\n LOF(n_neighbors=25, contamination=contamination),\r\n HBOS(contamination=contamination),\r\n PCA(contamination=contamination),\r\n OCSVM(contamination=contamination),\r\n KNN(n_neighbors=5, contamination=contamination),\r\n KNN(n_neighbors=15, contamination=contamination),\r\n KNN(n_neighbors=25, contamination=contamination)]\r\n\r\n # initialize a SUOD model with all features turned on\r\n model = SUOD(base_estimators=base_estimators, n_jobs=6, # number of workers\r\n rp_flag_global=True, # global flag for random projection\r\n bps_flag=True, # global flag for balanced parallel scheduling\r\n approx_flag_global=False, # global flag for model approximation\r\n contamination=contamination)\r\n\r\n model.fit(X_train) # fit all models with X\r\n model.approximate(X_train) # conduct model approximation if it is enabled\r\n predicted_labels = model.predict(X_test) # predict labels\r\n predicted_scores = model.decision_function(X_test) # predict scores\r\n predicted_probs = model.predict_proba(X_test) # predict outlying probability\r\n\r\n----\r\n\r\n`The corresponding paper <https://www.andrew.cmu.edu/user/yuezhao2/papers/21-mlsys-suod.pdf>`_ is published in Conference on Machine Learning Systems (MLSys).\r\nSee https://mlsys.org/ for more information.\r\n\r\n\r\nIf you use SUOD in a scientific publication, we would appreciate citations to the following paper::\r\n\r\n\r\n @inproceedings{zhao2021suod,\r\n title={SUOD: Accelerating Large-scale Unsupervised Heterogeneous Outlier Detection},\r\n author={Zhao, Yue and Hu, Xiyang and Cheng, Cheng and Wang, Cong and Wan, Changlin and Wang, Wen and Yang, Jianing and Bai, Haoping and Li, Zheng and Xiao, Cao and others},\r\n journal={Proceedings of Machine Learning and Systems},\r\n year={2021}\r\n }\r\n\r\n::\r\n\r\n Zhao, Y., Hu, X., Cheng, C., Wang, C., Wan, C., Wang, W., Yang, J., Bai, H., Li, Z., Xiao, C. and Wang, Y., 2021. SUOD: Accelerating Large-scale Unsupervised Heterogeneous Outlier Detection. Proceedings of Machine Learning and Systems (MLSys).\r\n\r\n\r\n**Table of Contents**\\ :\r\n\r\n\r\n* `Installation <#installation>`_\r\n* `API Cheatsheet & Reference <#api-cheatsheet--reference>`_\r\n* `Examples <#examples>`_\r\n* `Model Save & Load <#model-save--load>`_\r\n\r\n\r\n------------\r\n\r\n\r\nInstallation\r\n^^^^^^^^^^^^\r\n\r\nIt is recommended to use **pip** for installation. Please make sure\r\n**the latest version** is installed, as suod is updated frequently:\r\n\r\n.. code-block:: bash\r\n\r\n pip install suod # normal install\r\n pip install --upgrade suod # or update if needed\r\n pip install --pre suod # or include pre-release version for new features\r\n\r\nAlternatively, you could clone and run setup.py file:\r\n\r\n.. code-block:: bash\r\n\r\n git clone https://github.com/yzhao062/suod.git\r\n cd suod\r\n pip install .\r\n\r\n\r\n**Required Dependencies**\\ :\r\n\r\n\r\n* Python 3.8+\r\n* joblib\r\n* numpy>=1.13\r\n* pandas (**optional for building the cost forecast model**)\r\n* pyod\r\n* scipy>=0.19.1\r\n* scikit_learn>=1.0\r\n\r\n------------\r\n\r\n\r\nAPI Cheatsheet & Reference\r\n^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n\r\nFull API Reference: (https://suod.readthedocs.io/en/latest/api.html).\r\n\r\n* **fit(X, y)**\\ : Fit estimator. y is optional for unsupervised methods.\r\n* **approximate(X)**\\ : Use supervised models to approximate unsupervised base detectors. Fit should be invoked first.\r\n* **predict(X)**\\ : Predict on a particular sample once the estimator is fitted.\r\n* **predict_proba(X)**\\ : Predict the probability of a sample belonging to each class once the estimator is fitted.\r\n\r\n\r\nExamples\r\n^^^^^^^^\r\n\r\nAll three modules can be **executed separately** and the demo codes are in /examples/module_examples/{M1_RP, M2_BPS, and M3_PSA}.\r\nFor instance, you could navigate to /M1_RP/demo_random_projection.py. Demo codes all start with \"demo_*.py\".\r\n\r\n**The examples for the full framework** can be found under /examples folder; run \"demo_base.py\" for\r\na simplified example. Run \"demo_full.py\" for a full example.\r\n\r\nIt is noted the best performance may be achieved with multiple cores available.\r\n\r\n------------\r\n\r\n\r\nModel Save & Load\r\n^^^^^^^^^^^^^^^^^\r\n\r\nSUOD takes a similar approach of sklearn regarding model persistence.\r\nSee `model persistence <https://scikit-learn.org/stable/modules/model_persistence.html>`_ for clarification.\r\n\r\nIn short, we recommend to use joblib or pickle for saving and loading SUOD models.\r\nSee `\"examples/demo_model_save_load.py\" <https://github.com/yzhao062/suod/blob/master/examples/demo_model_save_load.py>`_ for an example.\r\nIn short, it is simple as below:\r\n\r\n.. code-block:: python\r\n\r\n from joblib import dump, load\r\n\r\n # save the fitted model\r\n dump(model, 'model.joblib')\r\n # load the model\r\n model = load('model.joblib')\r\n\r\n\r\n\r\n**More to come...**\r\nLast updated on Jan 14th, 2021.\r\n\r\nFeel free to star and watch for the future update :)\r\n\r\n----\r\n\r\nReferences\r\n----------\r\n\r\n.. [#Johnson1984Extensions] Johnson, W.B. and Lindenstrauss, J., 1984. Extensions of Lipschitz mappings into a Hilbert space. *Contemporary mathematics*, 26(189-206), p.1.\r\n\r\n.. [#Zhao2019PyOD] Zhao, Y., Nasrullah, Z. and Li, Z., 2019. PyOD: A Python Toolbox for Scalable Outlier Detection. *Journal of Machine Learning Research*, 20, pp.1-7.\r\n\r\n",
"bugtrack_url": null,
"license": "",
"summary": "A Scalable Framework for Unsupervised Outlier Detection (Anomaly Detection)",
"version": "0.1.3",
"project_urls": {
"Download": "https://github.com/yzhao062/suod/archive/master.zip",
"Homepage": "https://github.com/yzhao062/suod"
},
"split_keywords": [
"ensemble learning",
"anomaly detection",
"outlier ensembles",
"data mining",
"machine learning",
"python"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "24749fdfdb0b91100290aad865b88f8e0ba91a75d7c8343a432cdccb3bc00718",
"md5": "b503c16d7ebb2d12a4b48c4532d615b5",
"sha256": "d8ef15f00ba463f6a4cfc35889d237932512f534b1bea863ed9a42aa58e7379d"
},
"downloads": -1,
"filename": "suod-0.1.3.tar.gz",
"has_sig": false,
"md5_digest": "b503c16d7ebb2d12a4b48c4532d615b5",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 172055,
"upload_time": "2024-02-08T01:53:44",
"upload_time_iso_8601": "2024-02-08T01:53:44.257711Z",
"url": "https://files.pythonhosted.org/packages/24/74/9fdfdb0b91100290aad865b88f8e0ba91a75d7c8343a432cdccb3bc00718/suod-0.1.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-08 01:53:44",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "yzhao062",
"github_project": "suod",
"travis_ci": true,
"coveralls": false,
"github_actions": true,
"circle": true,
"appveyor": true,
"requirements": [],
"lcname": "suod"
}
JSON
