<!-- Logo & Title -->
<p align="center">
<img width=450 src="https://github.com/zillow/luminaire/blob/master/docs/assets/luminaire_logo_blue.svg" alt="Luminaire"></p>
<h3 align="center">A hands-off Anomaly Detection Library</h3>
<!-- Badges Start-->
[](https://badge.fury.io/py/luminaire)
[](https://pypi.org/project/luminaire/)
[](https://github.com/zillow/luminaire/blob/master/LICENSE.txt)
[](https://github.com/zillow/luminaire/actions?query=workflow%3A%22Luminaire+CI%22)
[](https://github.com/zillow/luminaire/actions?query=workflow%3A%22Luminaire+CD%22)
[](https://github.com/zillow/luminaire/actions?query=workflow%3A%22Luminaire+Docs%22)
<!-- Badges End -->
---
**Table of contents**
- [What is Luminaire](#what-is-luminaire)
- [Quick Start](#quick-start)
- [Time Series Outlier Detection Workflow](#time-series-outlier-detection-workflow)
- [Anomaly Detection for High Frequency Time Series](#anomaly-detection-for-high-frequency-time-series)
- [Examples](#examples)
- [Contributing](#contributing)
- [Citing](#citing)
- [Other Useful Resources](#other-useful-resources)
- [Blogs](#blogs)
- [Development Team](#development-team)
## What is Luminaire
[Luminaire](https://zillow.github.io/luminaire/) is a python package that provides ML-driven solutions for monitoring time series data. Luminaire provides several anomaly detection and forecasting capabilities that incorporate correlational and seasonal patterns as well as uncontrollable variations in the data over time.
## Quick Start
Install Luminaire from [PyPI](https://pypi.org/project/luminaire/) using ``pip``
```bash
pip install luminaire
```
Import ``luminaire`` module in python
```python
import luminaire
```
See [Examples](#Examples) to get started. Also, refer to the [Luminaire documentation](https://zillow.github.io/luminaire) for detailed description of methods and usage.
## Time Series Outlier Detection Workflow
Luminaire outlier detection workflow can be divided into 3 major components:
### Data Preprocessing and Profiling Component
This component can be called to prepare a time series prior to training an anomaly detection model on it. This step applies a number of methods that make anomaly detection more accurate and reliable, including missing data imputation, identifying and removing recent outliers from training data, necessary mathematical transformations, and data truncation based on recent change points. It also generates profiling information (historical change points, trend changes, etc.) that are considered in the training process.
Profiling information for time series data can be used to monitor data drift and irregular long-term swings.
### Modeling Component
This component performs time series model training based on the user-specified configuration OR optimized configuration (see Luminaire hyperparameter optimization). Luminaire model training is integrated with different structural time series models as well as filtering based models. See [Luminaire outlier detection](https://zillow.github.io/luminaire/tutorial/outlier_batch.html) for more information.
The Luminaire modeling step can be called after the data preprocessing and profiling step to perform necessary data preparation before training.
### Configuration Optimization Component
Luminaire's integration with configuration optimization enables a hands-off anomaly detection process where
the user needs to provide very minimal configuration for monitoring any type of time series data. This step can be combined with the preprocessing and modeling for any auto-configured anomaly detection use case. See [fully automatic outlier detection](https://zillow.github.io/luminaire/tutorial/optimization.html#fully-automatic-outlier-detection) for a detailed walkthrough.
## Anomaly Detection for High Frequency Time Series
Luminaire can also monitor a set of data points over windows of time instead of tracking individual data points. This approach is well-suited for streaming use cases where sustained fluctuations are of greater concern than individual fluctuations. See [anomaly detection for streaming data](https://zillow.github.io/luminaire/tutorial/streaming.html) for detailed information.
## Examples
### Batch Time Series Monitoring
```python
import pandas as pd
from luminaire.optimization.hyperparameter_optimization import HyperparameterOptimization
from luminaire.exploration.data_exploration import DataExploration
data = pd.read_csv('Path to input time series data')
# Input data should have a time column set as the index column of the dataframe and a value column named as 'raw'
# Optimization
hopt_obj = HyperparameterOptimization(freq='D')
opt_config = hopt_obj.run(data=data)
# Profiling
de_obj = DataExploration(freq='D', **opt_config)
training_data, pre_prc = de_obj.profile(data)
# Identify Model
model_class_name = opt_config['LuminaireModel']
module = __import__('luminaire.model', fromlist=[''])
model_class = getattr(module, model_class_name)
# Training
model_object = model_class(hyper_params=opt_config, freq='D')
success, model_date, trained_model = model_object.train(data=training_data, **pre_prc)
# Scoring
trained_model.score(100, '2021-01-01')
```
### Streaming Time Series Monitoring
```python
import pandas as pd
from luminaire.model.window_density import WindowDensityHyperParams, WindowDensityModel
from luminaire.exploration.data_exploration import DataExploration
data = pd.read_csv('Path to input time series data')
# Input data should have a time column set as the index column of the dataframe and a value column named as 'raw'
# Configuration Specs and Profiling
config = WindowDensityHyperParams().params
de_obj = DataExploration(**config)
data, pre_prc = de_obj.stream_profile(df=data)
config.update(pre_prc)
# Training
wdm_obj = WindowDensityModel(hyper_params=config)
success, training_end, model = wdm_obj.train(data=data)
# Scoring
score, scored_window = model.score(scoring_data) # scoring_data is data over a time-window instead of a datapoint
```
## Contributing
Want to help improve Luminaire? Check out our [contributing documentation](CONTRIBUTING.rst).
## Citing
Please cite the following article if Luminaire is used for any research purpose or scientific publication:
*Chakraborty, S., Shah, S., Soltani, K., Swigart, A., Yang, L., & Buckingham, K. (2020, December). Building an
Automated and Self-Aware Anomaly Detection System. In 2020 IEEE International Conference on Big Data (Big Data)
(pp. 1465-1475). IEEE.* ([arxiv link](https://arxiv.org/abs/2011.05047))
## Other Useful Resources
- *Chakraborty, S., Shah, S., Soltani, K., & Swigart, A. (2019, December). Root Cause Detection Among Anomalous Time
Series Using Temporal State Alignment. In 2019 18th IEEE International Conference On Machine Learning And Applications
(ICMLA) (pp. 523-528). IEEE.* ([arxiv link](https://arxiv.org/abs/2001.01056))
## Blogs
- Zillow Tech Hub: [*Automatic and Self-aware Anomaly Detection at Zillow Using Luminaire*](https://medium.com/zillow-tech-hub/automatic-and-self-aware-anomaly-detection-at-zillow-using-luminaire-7addfdae4ca9)
## Development Team
Luminaire is developed and maintained by [Sayan Chakraborty](https://github.com/sayanchk), [Smit Shah](https://github.com/shahsmit14), [Kiumars Soltani](https://github.com/kiumarss), [Luyao Yang]( https://github.com/snazzyfox), [Anna Swigart](https://github.com/annaswigart), [Kyle Buckingham](https://github.com/kylebuckingham) and many other contributors from the Zillow Group A.I. team.
Raw data
{
"_id": null,
"home_page": "https://zillow.github.io/luminaire",
"name": "luminaire",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "anomaly-detection outlier-detection forecasting time-series automl",
"author": "Zillow Group A.I. team",
"author_email": "luminaire-dev-oss@zillowgroup.com",
"download_url": "https://files.pythonhosted.org/packages/e7/3a/816aaf27f1ab83c53672148f8b61d298e7c4bfa731de2841b89603eda1cc/luminaire-0.4.3.tar.gz",
"platform": null,
"description": "<!-- Logo & Title -->\n<p align=\"center\">\n<img width=450 src=\"https://github.com/zillow/luminaire/blob/master/docs/assets/luminaire_logo_blue.svg\" alt=\"Luminaire\"></p>\n<h3 align=\"center\">A hands-off Anomaly Detection Library</h3>\n\n<!-- Badges Start-->\n[](https://badge.fury.io/py/luminaire)\n[](https://pypi.org/project/luminaire/)\n[](https://github.com/zillow/luminaire/blob/master/LICENSE.txt)\n[](https://github.com/zillow/luminaire/actions?query=workflow%3A%22Luminaire+CI%22)\n[](https://github.com/zillow/luminaire/actions?query=workflow%3A%22Luminaire+CD%22)\n[](https://github.com/zillow/luminaire/actions?query=workflow%3A%22Luminaire+Docs%22)\n<!-- Badges End -->\n\n---\n**Table of contents**\n\n- [What is Luminaire](#what-is-luminaire)\n- [Quick Start](#quick-start)\n- [Time Series Outlier Detection Workflow](#time-series-outlier-detection-workflow)\n- [Anomaly Detection for High Frequency Time Series](#anomaly-detection-for-high-frequency-time-series)\n- [Examples](#examples)\n- [Contributing](#contributing)\n- [Citing](#citing)\n- [Other Useful Resources](#other-useful-resources)\n- [Blogs](#blogs)\n- [Development Team](#development-team)\n\n\n## What is Luminaire\n\n[Luminaire](https://zillow.github.io/luminaire/) is a python package that provides ML-driven solutions for monitoring time series data. Luminaire provides several anomaly detection and forecasting capabilities that incorporate correlational and seasonal patterns as well as uncontrollable variations in the data over time.\n\n## Quick Start\n\nInstall Luminaire from [PyPI](https://pypi.org/project/luminaire/) using ``pip``\n\n```bash\npip install luminaire\n```\n\nImport ``luminaire`` module in python \n```python\nimport luminaire\n```\n\nSee [Examples](#Examples) to get started. Also, refer to the [Luminaire documentation](https://zillow.github.io/luminaire) for detailed description of methods and usage.\n\n\n## Time Series Outlier Detection Workflow\n\n\nLuminaire outlier detection workflow can be divided into 3 major components:\n\n### Data Preprocessing and Profiling Component\n\nThis component can be called to prepare a time series prior to training an anomaly detection model on it. This step applies a number of methods that make anomaly detection more accurate and reliable, including missing data imputation, identifying and removing recent outliers from training data, necessary mathematical transformations, and data truncation based on recent change points. It also generates profiling information (historical change points, trend changes, etc.) that are considered in the training process. \n\nProfiling information for time series data can be used to monitor data drift and irregular long-term swings.\n\n### Modeling Component\n\nThis component performs time series model training based on the user-specified configuration OR optimized configuration (see Luminaire hyperparameter optimization). Luminaire model training is integrated with different structural time series models as well as filtering based models. See [Luminaire outlier detection](https://zillow.github.io/luminaire/tutorial/outlier_batch.html) for more information.\n \nThe Luminaire modeling step can be called after the data preprocessing and profiling step to perform necessary data preparation before training.\n \n### Configuration Optimization Component\n \nLuminaire's integration with configuration optimization enables a hands-off anomaly detection process where\nthe user needs to provide very minimal configuration for monitoring any type of time series data. This step can be combined with the preprocessing and modeling for any auto-configured anomaly detection use case. See [fully automatic outlier detection](https://zillow.github.io/luminaire/tutorial/optimization.html#fully-automatic-outlier-detection) for a detailed walkthrough. \n\n## Anomaly Detection for High Frequency Time Series\n\nLuminaire can also monitor a set of data points over windows of time instead of tracking individual data points. This approach is well-suited for streaming use cases where sustained fluctuations are of greater concern than individual fluctuations. See [anomaly detection for streaming data](https://zillow.github.io/luminaire/tutorial/streaming.html) for detailed information.\n\n## Examples\n\n### Batch Time Series Monitoring\n```python\nimport pandas as pd\nfrom luminaire.optimization.hyperparameter_optimization import HyperparameterOptimization\nfrom luminaire.exploration.data_exploration import DataExploration\n\ndata = pd.read_csv('Path to input time series data')\n# Input data should have a time column set as the index column of the dataframe and a value column named as 'raw'\n\n# Optimization\nhopt_obj = HyperparameterOptimization(freq='D')\nopt_config = hopt_obj.run(data=data)\n\n# Profiling\nde_obj = DataExploration(freq='D', **opt_config)\ntraining_data, pre_prc = de_obj.profile(data)\n\n# Identify Model\nmodel_class_name = opt_config['LuminaireModel']\nmodule = __import__('luminaire.model', fromlist=[''])\nmodel_class = getattr(module, model_class_name)\n\n# Training\nmodel_object = model_class(hyper_params=opt_config, freq='D')\nsuccess, model_date, trained_model = model_object.train(data=training_data, **pre_prc)\n\n# Scoring\ntrained_model.score(100, '2021-01-01')\n```\n\n### Streaming Time Series Monitoring\n```python\nimport pandas as pd\nfrom luminaire.model.window_density import WindowDensityHyperParams, WindowDensityModel\nfrom luminaire.exploration.data_exploration import DataExploration\n\ndata = pd.read_csv('Path to input time series data')\n# Input data should have a time column set as the index column of the dataframe and a value column named as 'raw'\n\n# Configuration Specs and Profiling\nconfig = WindowDensityHyperParams().params\nde_obj = DataExploration(**config)\ndata, pre_prc = de_obj.stream_profile(df=data)\nconfig.update(pre_prc)\n\n# Training\nwdm_obj = WindowDensityModel(hyper_params=config)\nsuccess, training_end, model = wdm_obj.train(data=data)\n\n# Scoring\nscore, scored_window = model.score(scoring_data) # scoring_data is data over a time-window instead of a datapoint\n```\n\n## Contributing\n\nWant to help improve Luminaire? Check out our [contributing documentation](CONTRIBUTING.rst).\n\n## Citing\n\nPlease cite the following article if Luminaire is used for any research purpose or scientific publication:\n\n*Chakraborty, S., Shah, S., Soltani, K., Swigart, A., Yang, L., & Buckingham, K. (2020, December). Building an \nAutomated and Self-Aware Anomaly Detection System. In 2020 IEEE International Conference on Big Data (Big Data) \n(pp. 1465-1475). IEEE.* ([arxiv link](https://arxiv.org/abs/2011.05047))\n\n## Other Useful Resources\n\n- *Chakraborty, S., Shah, S., Soltani, K., & Swigart, A. (2019, December). Root Cause Detection Among Anomalous Time \nSeries Using Temporal State Alignment. In 2019 18th IEEE International Conference On Machine Learning And Applications \n(ICMLA) (pp. 523-528). IEEE.* ([arxiv link](https://arxiv.org/abs/2001.01056))\n\n\n## Blogs\n\n- Zillow Tech Hub: [*Automatic and Self-aware Anomaly Detection at Zillow Using Luminaire*](https://medium.com/zillow-tech-hub/automatic-and-self-aware-anomaly-detection-at-zillow-using-luminaire-7addfdae4ca9)\n\n\n## Development Team\n\nLuminaire is developed and maintained by [Sayan Chakraborty](https://github.com/sayanchk), [Smit Shah](https://github.com/shahsmit14), [Kiumars Soltani](https://github.com/kiumarss), [Luyao Yang]( https://github.com/snazzyfox), [Anna Swigart](https://github.com/annaswigart), [Kyle Buckingham](https://github.com/kylebuckingham) and many other contributors from the Zillow Group A.I. team.\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "Luminaire is a python package that provides ML driven solutions for monitoring time series data",
"version": "0.4.3",
"project_urls": {
"Homepage": "https://zillow.github.io/luminaire",
"Source": "https://github.com/zillow/luminaire",
"Tracker": "https://github.com/zillow/luminaire/issues"
},
"split_keywords": [
"anomaly-detection",
"outlier-detection",
"forecasting",
"time-series",
"automl"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "86802bce1ea66a2687b78fca103671175b5ca04410fb2eb864c2f1c8d017f313",
"md5": "6f3d7e79760b030c2b7340f3c809c1bc",
"sha256": "f1caf580a060549a298e4828cd01a2e69928820ec7f0611619d4a64325a4e42a"
},
"downloads": -1,
"filename": "luminaire-0.4.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "6f3d7e79760b030c2b7340f3c809c1bc",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 59383,
"upload_time": "2024-01-31T19:16:11",
"upload_time_iso_8601": "2024-01-31T19:16:11.252974Z",
"url": "https://files.pythonhosted.org/packages/86/80/2bce1ea66a2687b78fca103671175b5ca04410fb2eb864c2f1c8d017f313/luminaire-0.4.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "e73a816aaf27f1ab83c53672148f8b61d298e7c4bfa731de2841b89603eda1cc",
"md5": "a77a73f91cddbb8cbb410b8904359037",
"sha256": "253b91851ca6f1685c8974d94a260727b53981e2dab883bdfe99d4b43a10dae5"
},
"downloads": -1,
"filename": "luminaire-0.4.3.tar.gz",
"has_sig": false,
"md5_digest": "a77a73f91cddbb8cbb410b8904359037",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 55834,
"upload_time": "2024-01-31T19:16:12",
"upload_time_iso_8601": "2024-01-31T19:16:12.477554Z",
"url": "https://files.pythonhosted.org/packages/e7/3a/816aaf27f1ab83c53672148f8b61d298e7c4bfa731de2841b89603eda1cc/luminaire-0.4.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-31 19:16:12",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "zillow",
"github_project": "luminaire",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "bayescd",
"specs": [
[
">=",
"0.4"
]
]
},
{
"name": "changepy",
"specs": [
[
">=",
"0.3.1"
]
]
},
{
"name": "hyperopt",
"specs": [
[
">=",
"0.1.2"
]
]
},
{
"name": "numpy",
"specs": [
[
"<=",
"1.22.4"
],
[
">=",
"1.17.5"
]
]
},
{
"name": "pandas",
"specs": [
[
"<=",
"2.0.3"
],
[
">=",
"0.25.3"
]
]
},
{
"name": "pykalman",
"specs": [
[
">=",
"0.9.5"
]
]
},
{
"name": "scipy",
"specs": [
[
">=",
"1.6.0"
]
]
},
{
"name": "statsmodels",
"specs": [
[
">=",
"0.13.0"
],
[
"<=",
"0.13.5"
]
]
},
{
"name": "scikit-learn",
"specs": [
[
">=",
"0.24.2"
]
]
},
{
"name": "decorator",
"specs": [
[
">=",
"5.1.0"
]
]
}
],
"lcname": "luminaire"
}
JSON
