| Name | salesforce-merlion JSON |
| Version |
2.0.1
JSON |
| download |
| home_page | https://github.com/salesforce/Merlion |
| Summary | Merlion: A Machine Learning Framework for Time Series Intelligence |
| upload_time | 2023-02-07 07:48:51 |
| maintainer | |
| docs_url | None |
| author | Aadyot Bhatnagar, Paul Kassianik, Chenghao Liu, Tian Lan, Wenzhuo Yang, Rowan Cassius, Doyen Sahoo, Devansh Arpit, Sri Subramanian, Gerald Woo, Amrita Saha, Arun Kumar Jagota, Gokulakrishnan Gopalakrishnan, Manpreet Singh, K C Krithika, Sukumar Maddineni, Daeki Cho, Bo Zong, Yingbo Zhou, Caiming Xiong, Silvio Savarese, Steven Hoi, Huan Wang |
| requires_python | >=3.7.0 |
| license | 3-Clause BSD |
| keywords |
time series
forecasting
anomaly detection
machine learning
automl
ensemble learning
benchmarking
python
scientific toolkit
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<div align="center">
<img alt="Logo" src="https://github.com/salesforce/Merlion/raw/main/merlion_logo.svg" width="80%"/>
</div>
<div align="center">
<a href="https://github.com/salesforce/Merlion/actions">
<img alt="Tests" src="https://github.com/salesforce/Merlion/actions/workflows/tests.yml/badge.svg?branch=main"/>
</a>
<a href="https://github.com/salesforce/Merlion/actions">
<img alt="Coverage" src="https://github.com/salesforce/Merlion/raw/badges/coverage.svg"/>
</a>
<a href="https://pypi.python.org/pypi/salesforce-merlion">
<img alt="PyPI Version" src="https://img.shields.io/pypi/v/salesforce-merlion.svg"/>
</a>
<a href="https://opensource.salesforce.com/Merlion/index.html">
<img alt="docs" src="https://github.com/salesforce/Merlion/actions/workflows/docs.yml/badge.svg"/>
</a>
</div>
# Merlion: A Machine Learning Library for Time Series
## Table of Contents
1. [Introduction](#introduction)
1. [Comparison with Related Libraries](#comparison-with-related-libraries)
1. [Installation](#installation)
1. [Documentation](#documentation)
1. [Getting Started](#getting-started)
1. [Anomaly Detection](#anomaly-detection)
1. [Forecasting](#forecasting)
1. [Evaluation and Benchmarking](#evaluation-and-benchmarking)
1. [Technical Report and Citing Merlion](#technical-report-and-citing-merlion)
## Introduction
Merlion is a Python library for time series intelligence. It provides an end-to-end machine learning framework that
includes loading and transforming data, building and training models, post-processing model outputs, and evaluating
model performance. It supports various time series learning tasks, including forecasting, anomaly detection,
and change point detection for both univariate and multivariate time series. This library aims to provide engineers and
researchers a one-stop solution to rapidly develop models for their specific time series needs, and benchmark them
across multiple time series datasets.
Merlion's key features are
- Standardized and easily extensible data loading & benchmarking for a wide range of forecasting and anomaly
detection datasets. This includes transparent support for custom datasets.
- A library of diverse models for anomaly detection, forecasting, and change point detection, all
unified under a shared interface. Models include classic statistical methods, tree ensembles, and deep
learning approaches. Advanced users may fully configure each model as desired.
- Abstract `DefaultDetector` and `DefaultForecaster` models that are efficient, robustly achieve good performance,
and provide a starting point for new users.
- AutoML for automated hyperaparameter tuning and model selection.
- Unified API for using a wide range of models to forecast with
[exogenous regressors](https://opensource.salesforce.com/Merlion/tutorials/forecast/3_ForecastExogenous.html).
- Practical, industry-inspired post-processing rules for anomaly detectors that make anomaly scores more interpretable,
while also reducing the number of false positives.
- Easy-to-use ensembles that combine the outputs of multiple models to achieve more robust performance.
- Flexible evaluation pipelines that simulate the live deployment & re-training of a model in production,
and evaluate performance on both forecasting and anomaly detection.
- Native support for visualizing model predictions, including with a clickable visual UI.
- Distributed computation [backend](https://opensource.salesforce.com/Merlion/merlion.spark.html) using PySpark,
which can be used to serve time series applications at industrial scale.
## Comparison with Related Libraries
The table below provides a visual overview of how Merlion's key features compare to other libraries for time series
anomaly detection and/or forecasting.
| | Merlion | Prophet | Alibi Detect | Kats | darts | statsmodels | nixtla | GluonTS | RRCF | STUMPY | Greykite |pmdarima
:--- | :---: | :---:| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :----: | :---:
| Univariate Forecasting | ✅ | ✅| | ✅ | ✅ | ✅ | ✅ | ✅ | | |✅| ✅
| Multivariate Forecasting | ✅ | | | ✅ | ✅ | ✅| ✅ | ✅ | | | | |
| Univariate Anomaly Detection | ✅ | ✅ | ✅ | ✅ | ✅ | | | | ✅ | ✅ | ✅ | ✅ |
| Multivariate Anomaly Detection | ✅ | | ✅ | ✅ | ✅ | | | | ✅ | ✅ | | | |
| Pre Processing | ✅ | | ✅ | ✅ | ✅ | | ✅ | ✅ | | | ✅ | ✅
| Post Processing | ✅ | | ✅ | | | | | | | | | |
| AutoML | ✅ | | | ✅ | | | | | | | | ✅ | | ✅
| Ensembles | ✅ | | | ✅ | ✅ | | | | | ✅ | | | |
| Benchmarking | ✅ | | | | ✅ | ✅ | ✅ | | | | ✅ |
| Visualization | ✅ | ✅ | | ✅ | ✅ | | | | | | ✅ |
The following features are new in Merlion 2.0:
| | Merlion | Prophet | Alibi Detect | Kats | darts | statsmodels | nixtla | GluonTS | RRCF | STUMPY | Greykite |pmdarima
:--- | :---: | :---:| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :----: | :---:
| Exogenous Regressors | ✅ | ✅ | | |✅ | ✅ | | | | | ✅ | ✅
| Change Point Detection | ✅ | ✅ | ✅ | ✅ | | | | | | | ✅ |
| Clickable Visual UI | ✅ | | | | | | | | | | |
| Distributed Backend | ✅ | | | | | | ✅ | | | | |
## Installation
Merlion consists of two sub-repos: `merlion` implements the library's core time series intelligence features,
and `ts_datasets` provides standardized data loaders for multiple time series datasets. These loaders load
time series as ``pandas.DataFrame`` s with accompanying metadata.
You can install `merlion` from PyPI by calling ``pip install salesforce-merlion``. You may install from source by
cloning this repoand calling ``pip install Merlion/``, or ``pip install -e Merlion/`` to install in editable mode.
You may install additional dependencies via ``pip install salesforce-merlion[all]``, or by calling
``pip install "Merlion/[all]"`` if installing from source.
Individually, the optional dependencies include ``dashboard`` for a GUI dashboard,
``spark`` for a distributed computation backend with PySpark, and ``deep-learning`` for all deep learning models.
To install the data loading package `ts_datasets`, clone this repo and call ``pip install -e Merlion/ts_datasets/``.
This package must be installed in editable mode (i.e. with the ``-e`` flag) if you don't want to manually specify the
root directory of every dataset when initializing its data loader.
Note the following external dependencies:
1. Some of our forecasting models depend on OpenMP. If using ``conda``, please ``conda install -c conda-forge lightgbm``
before installing our package. This will ensure that OpenMP is configured to work with the ``lightgbm`` package
(one of our dependencies) in your ``conda`` environment. If using Mac, please install [Homebrew](https://brew.sh/)
and call ``brew install libomp`` so that the OpenMP libary is available for the model.
2. Some of our anomaly detection models depend on the Java Development Kit (JDK). For Ubuntu, call
``sudo apt-get install openjdk-11-jdk``. For Mac OS, install [Homebrew](<https://brew.sh/>) and call
``brew tap adoptopenjdk/openjdk && brew install --cask adoptopenjdk11``. Also ensure that ``java`` can be found
on your ``PATH``, and that the ``JAVA_HOME`` environment variable is set.
## Documentation
For example code and an introduction to Merlion, see the Jupyter notebooks in
[`examples`](https://github.com/salesforce/Merlion/tree/main/examples), and the guided walkthrough
[here](https://opensource.salesforce.com/Merlion/tutorials.html). You may find detailed API documentation (including the
example code) [here](https://opensource.salesforce.com/Merlion/index.html). The
[technical report](https://arxiv.org/abs/2109.09265) outlines Merlion's overall architecture
and presents experimental results on time series anomaly detection & forecasting for both univariate and multivariate
time series.
## Getting Started
The easiest way to get started is to use the GUI web-based
[dashboard](https://opensource.salesforce.com/Merlion/merlion.dashboard.html).
This dashboard provides a great way to quickly experiment with many models on your own custom datasets.
To use it, install Merlion with the optional ``dashboard`` dependency (i.e.
``pip install salesforce-merlion[dashboard]``), and call ``python -m merlion.dashboard`` from the command line.
You can view the dashboard at http://localhost:8050.
Below, we show some screenshots of the dashboard for both anomaly detection and forecasting.
To help you get started with using Merlion in your own code, we provide below some minimal examples using Merlion
default models for both anomaly detection and forecasting.
### Anomaly Detection
Here, we show the code to replicate the results from the anomaly detection dashboard above.
We begin by importing Merlion’s `TimeSeries` class and the data loader for the Numenta Anomaly Benchmark `NAB`.
We can then divide a specific time series from this dataset into training and testing splits.
```python
from merlion.utils import TimeSeries
from ts_datasets.anomaly import NAB
# Data loader returns pandas DataFrames, which we convert to Merlion TimeSeries
time_series, metadata = NAB(subset="realKnownCause")[3]
train_data = TimeSeries.from_pd(time_series[metadata.trainval])
test_data = TimeSeries.from_pd(time_series[~metadata.trainval])
test_labels = TimeSeries.from_pd(metadata.anomaly[~metadata.trainval])
```
We can then initialize and train Merlion’s `DefaultDetector`, which is an anomaly detection model that
balances performance with efficiency. We also obtain its predictions on the test split.
```python
from merlion.models.defaults import DefaultDetectorConfig, DefaultDetector
model = DefaultDetector(DefaultDetectorConfig())
model.train(train_data=train_data)
test_pred = model.get_anomaly_label(time_series=test_data)
```
Next, we visualize the model's predictions.
```python
from merlion.plot import plot_anoms
import matplotlib.pyplot as plt
fig, ax = model.plot_anomaly(time_series=test_data)
plot_anoms(ax=ax, anomaly_labels=test_labels)
plt.show()
```
Finally, we can quantitatively evaluate the model. The precision and recall come from the fact that the model
fired 3 alarms, with 2 true positives, 1 false negative, and 1 false positive. We also evaluate the mean time
the model took to detect each anomaly that it correctly detected.
```python
from merlion.evaluate.anomaly import TSADMetric
p = TSADMetric.Precision.value(ground_truth=test_labels, predict=test_pred)
r = TSADMetric.Recall.value(ground_truth=test_labels, predict=test_pred)
f1 = TSADMetric.F1.value(ground_truth=test_labels, predict=test_pred)
mttd = TSADMetric.MeanTimeToDetect.value(ground_truth=test_labels, predict=test_pred)
print(f"Precision: {p:.4f}, Recall: {r:.4f}, F1: {f1:.4f}\n"
f"Mean Time To Detect: {mttd}")
```
```
Precision: 0.6667, Recall: 0.6667, F1: 0.6667
Mean Time To Detect: 1 days 10:22:30
```
### Forecasting
Here, we show the code to replicate the results from the forecasting dashboard above.
We begin by importing Merlion’s `TimeSeries` class and the data loader for the `M4` dataset. We can then divide a
specific time series from this dataset into training and testing splits.
```python
from merlion.utils import TimeSeries
from ts_datasets.forecast import M4
# Data loader returns pandas DataFrames, which we convert to Merlion TimeSeries
time_series, metadata = M4(subset="Hourly")[0]
train_data = TimeSeries.from_pd(time_series[metadata.trainval])
test_data = TimeSeries.from_pd(time_series[~metadata.trainval])
```
We can then initialize and train Merlion’s `DefaultForecaster`, which is an forecasting model that balances
performance with efficiency. We also obtain its predictions on the test split.
```python
from merlion.models.defaults import DefaultForecasterConfig, DefaultForecaster
model = DefaultForecaster(DefaultForecasterConfig())
model.train(train_data=train_data)
test_pred, test_err = model.forecast(time_stamps=test_data.time_stamps)
```
Next, we visualize the model’s predictions.
```python
import matplotlib.pyplot as plt
fig, ax = model.plot_forecast(time_series=test_data, plot_forecast_uncertainty=True)
plt.show()
```
Finally, we quantitatively evaluate the model. sMAPE measures the error of the prediction on a scale of 0 to 100
(lower is better), while MSIS evaluates the quality of the 95% confidence band on a scale of 0 to 100 (lower is better).
```python
# Evaluate the model's predictions quantitatively
from scipy.stats import norm
from merlion.evaluate.forecast import ForecastMetric
# Compute the sMAPE of the predictions (0 to 100, smaller is better)
smape = ForecastMetric.sMAPE.value(ground_truth=test_data, predict=test_pred)
# Compute the MSIS of the model's 95% confidence interval (0 to 100, smaller is better)
lb = TimeSeries.from_pd(test_pred.to_pd() + norm.ppf(0.025) * test_err.to_pd().values)
ub = TimeSeries.from_pd(test_pred.to_pd() + norm.ppf(0.975) * test_err.to_pd().values)
msis = ForecastMetric.MSIS.value(ground_truth=test_data, predict=test_pred,
insample=train_data, lb=lb, ub=ub)
print(f"sMAPE: {smape:.4f}, MSIS: {msis:.4f}")
```
```
sMAPE: 4.1944, MSIS: 18.9331
```
## Evaluation and Benchmarking
One of Merlion's key features is an evaluation pipeline that simulates the live deployment
of a model on historical data. This enables you to compare models on the datasets relevant
to them, under the conditions that they may encounter in a production environment. Our
evaluation pipeline proceeds as follows:
1. Train an initial model on recent historical training data (designated as the training split of the time series)
1. At a regular interval (e.g. once per day), retrain the entire model on the most recent data. This can be either the
entire history of the time series, or a more limited window (e.g. 4 weeks).
1. Obtain the model's predictions (anomaly scores or forecasts) for the time series values that occur between
re-trainings. You may customize whether this should be done in batch (predicting all values at once),
streaming (updating the model's internal state after each data point without fully re-training it),
or some intermediate cadence.
1. Compare the model's predictions against the ground truth (labeled anomalies for anomaly detection, or the actual
time series values for forecasting), and report quantitative evaluation metrics.
We provide scripts that allow you to use this pipeline to evaluate arbitrary models on arbitrary datasets.
For example, invoking
```shell script
python benchmark_anomaly.py --dataset NAB_realAWSCloudwatch --model IsolationForest --retrain_freq 1d
```
will evaluate the anomaly detection performance of the `IsolationForest` (retrained once a day) on the
"realAWSCloudwatch" subset of the NAB dataset. Similarly, invoking
```shell script
python benchmark_forecast.py --dataset M4_Hourly --model ETS
```
will evaluate the batch forecasting performance (i.e. no retraining) of `ETS` on the "Hourly" subset of the M4 dataset.
You can find the results produced by running these scripts in the Experiments section of the
[technical report](https://arxiv.org/abs/2109.09265).
## Technical Report and Citing Merlion
You can find more details in our technical report: https://arxiv.org/abs/2109.09265
If you're using Merlion in your research or applications, please cite using this BibTeX:
```
@article{bhatnagar2021merlion,
title={Merlion: A Machine Learning Library for Time Series},
author={Aadyot Bhatnagar and Paul Kassianik and Chenghao Liu and Tian Lan and Wenzhuo Yang
and Rowan Cassius and Doyen Sahoo and Devansh Arpit and Sri Subramanian and Gerald Woo
and Amrita Saha and Arun Kumar Jagota and Gokulakrishnan Gopalakrishnan and Manpreet Singh
and K C Krithika and Sukumar Maddineni and Daeki Cho and Bo Zong and Yingbo Zhou
and Caiming Xiong and Silvio Savarese and Steven Hoi and Huan Wang},
year={2021},
eprint={2109.09265},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
```
Raw data
{
"_id": null,
"home_page": "https://github.com/salesforce/Merlion",
"name": "salesforce-merlion",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7.0",
"maintainer_email": "",
"keywords": "time series,forecasting,anomaly detection,machine learning,autoML,ensemble learning,benchmarking,Python,scientific toolkit",
"author": "Aadyot Bhatnagar, Paul Kassianik, Chenghao Liu, Tian Lan, Wenzhuo Yang, Rowan Cassius, Doyen Sahoo, Devansh Arpit, Sri Subramanian, Gerald Woo, Amrita Saha, Arun Kumar Jagota, Gokulakrishnan Gopalakrishnan, Manpreet Singh, K C Krithika, Sukumar Maddineni, Daeki Cho, Bo Zong, Yingbo Zhou, Caiming Xiong, Silvio Savarese, Steven Hoi, Huan Wang",
"author_email": "abhatnagar@salesforce.com",
"download_url": "https://files.pythonhosted.org/packages/fb/94/53db468198065d576277d276e086414ce9084216b87c0613f2d45219d433/salesforce-merlion-2.0.1.tar.gz",
"platform": null,
"description": "<div align=\"center\">\n<img alt=\"Logo\" src=\"https://github.com/salesforce/Merlion/raw/main/merlion_logo.svg\" width=\"80%\"/>\n</div>\n\n<div align=\"center\">\n <a href=\"https://github.com/salesforce/Merlion/actions\">\n <img alt=\"Tests\" src=\"https://github.com/salesforce/Merlion/actions/workflows/tests.yml/badge.svg?branch=main\"/>\n </a>\n <a href=\"https://github.com/salesforce/Merlion/actions\">\n <img alt=\"Coverage\" src=\"https://github.com/salesforce/Merlion/raw/badges/coverage.svg\"/>\n </a>\n <a href=\"https://pypi.python.org/pypi/salesforce-merlion\">\n <img alt=\"PyPI Version\" src=\"https://img.shields.io/pypi/v/salesforce-merlion.svg\"/>\n </a>\n <a href=\"https://opensource.salesforce.com/Merlion/index.html\">\n <img alt=\"docs\" src=\"https://github.com/salesforce/Merlion/actions/workflows/docs.yml/badge.svg\"/>\n </a>\n</div>\n\n# Merlion: A Machine Learning Library for Time Series\n\n## Table of Contents\n1. [Introduction](#introduction)\n1. [Comparison with Related Libraries](#comparison-with-related-libraries)\n1. [Installation](#installation)\n1. [Documentation](#documentation)\n1. [Getting Started](#getting-started)\n 1. [Anomaly Detection](#anomaly-detection)\n 1. [Forecasting](#forecasting)\n1. [Evaluation and Benchmarking](#evaluation-and-benchmarking)\n1. [Technical Report and Citing Merlion](#technical-report-and-citing-merlion)\n\n## Introduction\nMerlion is a Python library for time series intelligence. It provides an end-to-end machine learning framework that\nincludes loading and transforming data, building and training models, post-processing model outputs, and evaluating\nmodel performance. It supports various time series learning tasks, including forecasting, anomaly detection,\nand change point detection for both univariate and multivariate time series. This library aims to provide engineers and\nresearchers a one-stop solution to rapidly develop models for their specific time series needs, and benchmark them\nacross multiple time series datasets.\n\nMerlion's key features are\n- Standardized and easily extensible data loading & benchmarking for a wide range of forecasting and anomaly\n detection datasets. This includes transparent support for custom datasets.\n- A library of diverse models for anomaly detection, forecasting, and change point detection, all\n unified under a shared interface. Models include classic statistical methods, tree ensembles, and deep\n learning approaches. Advanced users may fully configure each model as desired.\n- Abstract `DefaultDetector` and `DefaultForecaster` models that are efficient, robustly achieve good performance,\n and provide a starting point for new users.\n- AutoML for automated hyperaparameter tuning and model selection.\n- Unified API for using a wide range of models to forecast with\n [exogenous regressors](https://opensource.salesforce.com/Merlion/tutorials/forecast/3_ForecastExogenous.html).\n- Practical, industry-inspired post-processing rules for anomaly detectors that make anomaly scores more interpretable,\n while also reducing the number of false positives.\n- Easy-to-use ensembles that combine the outputs of multiple models to achieve more robust performance. \n- Flexible evaluation pipelines that simulate the live deployment & re-training of a model in production,\n and evaluate performance on both forecasting and anomaly detection.\n- Native support for visualizing model predictions, including with a clickable visual UI.\n- Distributed computation [backend](https://opensource.salesforce.com/Merlion/merlion.spark.html) using PySpark,\n which can be used to serve time series applications at industrial scale.\n\n\n## Comparison with Related Libraries\n\nThe table below provides a visual overview of how Merlion's key features compare to other libraries for time series\nanomaly detection and/or forecasting.\n\n| | Merlion | Prophet | Alibi Detect | Kats | darts | statsmodels | nixtla | GluonTS | RRCF | STUMPY | Greykite |pmdarima \n:--- | :---: | :---:| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :----: | :---:\n| Univariate Forecasting | \u2705 | \u2705| | \u2705 | \u2705 | \u2705 | \u2705 | \u2705 | | |\u2705| \u2705 \n| Multivariate Forecasting | \u2705 | | | \u2705 | \u2705 | \u2705| \u2705 | \u2705 | | | | |\n| Univariate Anomaly Detection | \u2705 | \u2705 | \u2705 | \u2705 | \u2705 | | | | \u2705 | \u2705 | \u2705 | \u2705 | \n| Multivariate Anomaly Detection | \u2705 | | \u2705 | \u2705 | \u2705 | | | | \u2705 | \u2705 | | | |\n| Pre Processing | \u2705 | | \u2705 | \u2705 | \u2705 | | \u2705 | \u2705 | | | \u2705 | \u2705\n| Post Processing | \u2705 | | \u2705 | | | | | | | | | |\n| AutoML | \u2705 | | | \u2705 | | | | | | | | \u2705 | | \u2705 \n| Ensembles | \u2705 | | | \u2705 | \u2705 | | | | | \u2705 | | | | \n| Benchmarking | \u2705 | | | | \u2705 | \u2705 | \u2705 | | | | \u2705 | \n| Visualization | \u2705 | \u2705 | | \u2705 | \u2705 | | | | | | \u2705 | \n\nThe following features are new in Merlion 2.0:\n\n| | Merlion | Prophet | Alibi Detect | Kats | darts | statsmodels | nixtla | GluonTS | RRCF | STUMPY | Greykite |pmdarima \n:--- | :---: | :---:| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :----: | :---:\n| Exogenous Regressors | \u2705 | \u2705 | | |\u2705 | \u2705 | | | | | \u2705 | \u2705\n| Change Point Detection | \u2705 | \u2705 | \u2705 | \u2705 | | | | | | | \u2705 |\n| Clickable Visual UI | \u2705 | | | | | | | | | | |\n| Distributed Backend | \u2705 | | | | | | \u2705 | | | | | \n\n## Installation\n\nMerlion consists of two sub-repos: `merlion` implements the library's core time series intelligence features,\nand `ts_datasets` provides standardized data loaders for multiple time series datasets. These loaders load\ntime series as ``pandas.DataFrame``\u00a0s with accompanying metadata.\n\nYou can install `merlion` from PyPI by calling ``pip install salesforce-merlion``. You may install from source by\ncloning this repoand calling ``pip install Merlion/``, or ``pip install -e Merlion/`` to install in editable mode.\nYou may install additional dependencies via ``pip install salesforce-merlion[all]``, or by calling\n``pip install \"Merlion/[all]\"`` if installing from source. \nIndividually, the optional dependencies include ``dashboard`` for a GUI dashboard,\n``spark`` for a distributed computation backend with PySpark, and ``deep-learning`` for all deep learning models.\n\nTo install the data loading package `ts_datasets`, clone this repo and call ``pip install -e Merlion/ts_datasets/``.\nThis package must be installed in editable mode (i.e. with the ``-e`` flag) if you don't want to manually specify the\nroot directory of every dataset when initializing its data loader.\n\nNote the following external dependencies:\n\n1. Some of our forecasting models depend on OpenMP. If using ``conda``, please ``conda install -c conda-forge lightgbm``\n before installing our package. This will ensure that OpenMP is configured to work with the ``lightgbm`` package\n (one of our dependencies) in your ``conda`` environment. If using Mac, please install [Homebrew](https://brew.sh/)\n and call ``brew install libomp`` so that the OpenMP libary is available for the model.\n\n2. Some of our anomaly detection models depend on the Java Development Kit (JDK). For Ubuntu, call\n ``sudo apt-get install openjdk-11-jdk``. For Mac OS, install [Homebrew](<https://brew.sh/>) and call\n ``brew tap adoptopenjdk/openjdk && brew install --cask adoptopenjdk11``. Also ensure that ``java`` can be found\n on your ``PATH``, and that the ``JAVA_HOME`` environment variable is set.\n\n## Documentation\n\nFor example code and an introduction to Merlion, see the Jupyter notebooks in\n[`examples`](https://github.com/salesforce/Merlion/tree/main/examples), and the guided walkthrough\n[here](https://opensource.salesforce.com/Merlion/tutorials.html). You may find detailed API documentation (including the\nexample code) [here](https://opensource.salesforce.com/Merlion/index.html). The\n[technical report](https://arxiv.org/abs/2109.09265) outlines Merlion's overall architecture\nand presents experimental results on time series anomaly detection & forecasting for both univariate and multivariate\ntime series.\n\n## Getting Started\nThe easiest way to get started is to use the GUI web-based\n[dashboard](https://opensource.salesforce.com/Merlion/merlion.dashboard.html).\nThis dashboard provides a great way to quickly experiment with many models on your own custom datasets.\nTo use it, install Merlion with the optional ``dashboard`` dependency (i.e.\n``pip install salesforce-merlion[dashboard]``), and call ``python -m merlion.dashboard`` from the command line.\nYou can view the dashboard at http://localhost:8050.\nBelow, we show some screenshots of the dashboard for both anomaly detection and forecasting.\n\n\n\n\n\nTo help you get started with using Merlion in your own code, we provide below some minimal examples using Merlion\ndefault models for both anomaly detection and forecasting.\n\n### Anomaly Detection\nHere, we show the code to replicate the results from the anomaly detection dashboard above.\nWe begin by importing Merlion\u2019s `TimeSeries` class and the data loader for the Numenta Anomaly Benchmark `NAB`.\nWe can then divide a specific time series from this dataset into training and testing splits.\n\n```python\nfrom merlion.utils import TimeSeries\nfrom ts_datasets.anomaly import NAB\n\n# Data loader returns pandas DataFrames, which we convert to Merlion TimeSeries\ntime_series, metadata = NAB(subset=\"realKnownCause\")[3]\ntrain_data = TimeSeries.from_pd(time_series[metadata.trainval])\ntest_data = TimeSeries.from_pd(time_series[~metadata.trainval])\ntest_labels = TimeSeries.from_pd(metadata.anomaly[~metadata.trainval])\n```\n\nWe can then initialize and train Merlion\u2019s `DefaultDetector`, which is an anomaly detection model that\nbalances performance with efficiency. We also obtain its predictions on the test split.\n\n```python\nfrom merlion.models.defaults import DefaultDetectorConfig, DefaultDetector\nmodel = DefaultDetector(DefaultDetectorConfig())\nmodel.train(train_data=train_data)\ntest_pred = model.get_anomaly_label(time_series=test_data)\n```\n\nNext, we visualize the model's predictions.\n\n```python\nfrom merlion.plot import plot_anoms\nimport matplotlib.pyplot as plt\nfig, ax = model.plot_anomaly(time_series=test_data)\nplot_anoms(ax=ax, anomaly_labels=test_labels)\nplt.show()\n```\n\n\nFinally, we can quantitatively evaluate the model. The precision and recall come from the fact that the model\nfired 3 alarms, with 2 true positives, 1 false negative, and 1 false positive. We also evaluate the mean time\nthe model took to detect each anomaly that it correctly detected.\n\n```python\nfrom merlion.evaluate.anomaly import TSADMetric\np = TSADMetric.Precision.value(ground_truth=test_labels, predict=test_pred)\nr = TSADMetric.Recall.value(ground_truth=test_labels, predict=test_pred)\nf1 = TSADMetric.F1.value(ground_truth=test_labels, predict=test_pred)\nmttd = TSADMetric.MeanTimeToDetect.value(ground_truth=test_labels, predict=test_pred)\nprint(f\"Precision: {p:.4f}, Recall: {r:.4f}, F1: {f1:.4f}\\n\"\n f\"Mean Time To Detect: {mttd}\")\n```\n```\nPrecision: 0.6667, Recall: 0.6667, F1: 0.6667\nMean Time To Detect: 1 days 10:22:30\n```\n### Forecasting\nHere, we show the code to replicate the results from the forecasting dashboard above.\nWe begin by importing Merlion\u2019s `TimeSeries` class and the data loader for the `M4` dataset. We can then divide a\nspecific time series from this dataset into training and testing splits.\n\n```python\nfrom merlion.utils import TimeSeries\nfrom ts_datasets.forecast import M4\n\n# Data loader returns pandas DataFrames, which we convert to Merlion TimeSeries\ntime_series, metadata = M4(subset=\"Hourly\")[0]\ntrain_data = TimeSeries.from_pd(time_series[metadata.trainval])\ntest_data = TimeSeries.from_pd(time_series[~metadata.trainval])\n```\n\nWe can then initialize and train Merlion\u2019s `DefaultForecaster`, which is an forecasting model that balances\nperformance with efficiency. We also obtain its predictions on the test split.\n\n```python\nfrom merlion.models.defaults import DefaultForecasterConfig, DefaultForecaster\nmodel = DefaultForecaster(DefaultForecasterConfig())\nmodel.train(train_data=train_data)\ntest_pred, test_err = model.forecast(time_stamps=test_data.time_stamps)\n```\n\nNext, we visualize the model\u2019s predictions.\n\n```python\nimport matplotlib.pyplot as plt\nfig, ax = model.plot_forecast(time_series=test_data, plot_forecast_uncertainty=True)\nplt.show()\n```\n\n\nFinally, we quantitatively evaluate the model. sMAPE measures the error of the prediction on a scale of 0 to 100\n(lower is better), while MSIS evaluates the quality of the 95% confidence band on a scale of 0 to 100 (lower is better).\n\n```python\n# Evaluate the model's predictions quantitatively\nfrom scipy.stats import norm\nfrom merlion.evaluate.forecast import ForecastMetric\n\n# Compute the sMAPE of the predictions (0 to 100, smaller is better)\nsmape = ForecastMetric.sMAPE.value(ground_truth=test_data, predict=test_pred)\n\n# Compute the MSIS of the model's 95% confidence interval (0 to 100, smaller is better)\nlb = TimeSeries.from_pd(test_pred.to_pd() + norm.ppf(0.025) * test_err.to_pd().values)\nub = TimeSeries.from_pd(test_pred.to_pd() + norm.ppf(0.975) * test_err.to_pd().values)\nmsis = ForecastMetric.MSIS.value(ground_truth=test_data, predict=test_pred,\n insample=train_data, lb=lb, ub=ub)\nprint(f\"sMAPE: {smape:.4f}, MSIS: {msis:.4f}\")\n```\n```\nsMAPE: 4.1944, MSIS: 18.9331\n```\n\n## Evaluation and Benchmarking\n\nOne of Merlion's key features is an evaluation pipeline that simulates the live deployment\nof a model on historical data. This enables you to compare models on the datasets relevant\nto them, under the conditions that they may encounter in a production environment. Our\nevaluation pipeline proceeds as follows:\n1. Train an initial model on recent historical training data (designated as the training split of the time series)\n1. At a regular interval (e.g. once per day), retrain the entire model on the most recent data. This can be either the\n entire history of the time series, or a more limited window (e.g. 4 weeks).\n1. Obtain the model's predictions (anomaly scores or forecasts) for the time series values that occur between\n re-trainings. You may customize whether this should be done in batch (predicting all values at once),\n streaming (updating the model's internal state after each data point without fully re-training it),\n or some intermediate cadence.\n1. Compare the model's predictions against the ground truth (labeled anomalies for anomaly detection, or the actual\n time series values for forecasting), and report quantitative evaluation metrics.\n\nWe provide scripts that allow you to use this pipeline to evaluate arbitrary models on arbitrary datasets.\nFor example, invoking\n```shell script\npython benchmark_anomaly.py --dataset NAB_realAWSCloudwatch --model IsolationForest --retrain_freq 1d\n``` \nwill evaluate the anomaly detection performance of the `IsolationForest` (retrained once a day) on the\n\"realAWSCloudwatch\" subset of the NAB dataset. Similarly, invoking\n```shell script\npython benchmark_forecast.py --dataset M4_Hourly --model ETS\n```\nwill evaluate the batch forecasting performance (i.e. no retraining) of `ETS` on the \"Hourly\" subset of the M4 dataset. \nYou can find the results produced by running these scripts in the Experiments section of the\n[technical report](https://arxiv.org/abs/2109.09265).\n\n## Technical Report and Citing Merlion\nYou can find more details in our technical report: https://arxiv.org/abs/2109.09265\n\nIf you're using Merlion in your research or applications, please cite using this BibTeX:\n```\n@article{bhatnagar2021merlion,\n title={Merlion: A Machine Learning Library for Time Series},\n author={Aadyot Bhatnagar and Paul Kassianik and Chenghao Liu and Tian Lan and Wenzhuo Yang\n and Rowan Cassius and Doyen Sahoo and Devansh Arpit and Sri Subramanian and Gerald Woo\n and Amrita Saha and Arun Kumar Jagota and Gokulakrishnan Gopalakrishnan and Manpreet Singh\n and K C Krithika and Sukumar Maddineni and Daeki Cho and Bo Zong and Yingbo Zhou\n and Caiming Xiong and Silvio Savarese and Steven Hoi and Huan Wang},\n year={2021},\n eprint={2109.09265},\n archivePrefix={arXiv},\n primaryClass={cs.LG}\n}\n```\n",
"bugtrack_url": null,
"license": "3-Clause BSD",
"summary": "Merlion: A Machine Learning Framework for Time Series Intelligence",
"version": "2.0.1",
"split_keywords": [
"time series",
"forecasting",
"anomaly detection",
"machine learning",
"automl",
"ensemble learning",
"benchmarking",
"python",
"scientific toolkit"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "0cf69895b145cf8ff1502cf9345948291be18f6cfb66cff912f2aa818b7d359d",
"md5": "160b86bef61a8c8354b7e74da8321f5c",
"sha256": "b766b526ace83e1abeca00fa73626c7765ef3cc2053bf4b47e7b920edca6369a"
},
"downloads": -1,
"filename": "salesforce_merlion-2.0.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "160b86bef61a8c8354b7e74da8321f5c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7.0",
"size": 1309586,
"upload_time": "2023-02-07T07:48:50",
"upload_time_iso_8601": "2023-02-07T07:48:50.280229Z",
"url": "https://files.pythonhosted.org/packages/0c/f6/9895b145cf8ff1502cf9345948291be18f6cfb66cff912f2aa818b7d359d/salesforce_merlion-2.0.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "fb9453db468198065d576277d276e086414ce9084216b87c0613f2d45219d433",
"md5": "62364fdeda47eacb039aea5dcf51ca83",
"sha256": "eeef166fa618ea7fe8ba11ee790c85742d07287bda7a6dcff912ec4279ea6219"
},
"downloads": -1,
"filename": "salesforce-merlion-2.0.1.tar.gz",
"has_sig": false,
"md5_digest": "62364fdeda47eacb039aea5dcf51ca83",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7.0",
"size": 1243143,
"upload_time": "2023-02-07T07:48:51",
"upload_time_iso_8601": "2023-02-07T07:48:51.818510Z",
"url": "https://files.pythonhosted.org/packages/fb/94/53db468198065d576277d276e086414ce9084216b87c0613f2d45219d433/salesforce-merlion-2.0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-02-07 07:48:51",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "salesforce",
"github_project": "Merlion",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "salesforce-merlion"
}
