<p align="left">
<img width=15% src="https://dai.lids.mit.edu/wp-content/uploads/2018/06/Logo_DAI_highres.png" alt="DAI-Lab" />
<i>A project from Data to AI Lab at MIT.</i>
</p>
<!-- Uncomment these lines after releasing the package to PyPI for version and downloads badges -->
<!--[](https://pypi.python.org/pypi/zephyr_ml)-->
<!--[](https://pepy.tech/project/zephyr_ml)-->
<!--[](https://travis-ci.org/signals-dev/zephyr)-->
<!--[](https://codecov.io/gh/signals-dev/zephyr)-->
# Zephyr
A machine learning library for assisting in the generation of machine learning problems for wind farms operations data by analyzing past occurrences of events.
| Important Links | |
| ----------------------------------- | -------------------------------------------------------------------- |
| :computer: **[Website]** | Check out the Sintel Website for more information about the project. |
| :book: **[Documentation]** | Quickstarts, User and Development Guides, and API Reference. |
| :star: **[Tutorials]** | Checkout our notebooks |
| :octocat: **[Repository]** | The link to the Github Repository of this library. |
| :scroll: **[License]** | The repository is published under the MIT License. |
| :keyboard: **[Development Status]** | This software is in its Pre-Alpha stage. |
| ![][Slack Logo] **[Community]** | Join our Slack Workspace for announcements and discussions. |
[Website]: https://sintel.dev/
[Documentation]: https://dtail.gitbook.io/zephyr/
[Repository]: https://github.com/sintel-dev/Zephyr
[Tutorials]: https://github.com/sintel-dev/Zephyr/blob/master/notebooks
[License]: https://github.com/sintel-dev/Zephyr/blob/master/LICENSE
[Development Status]: https://pypi.org/search/?c=Development+Status+%3A%3A+2+-+Pre-Alpha
[Community]: https://join.slack.com/t/sintel-space/shared_invite/zt-q147oimb-4HcphcxPfDAM0O9_4PaUtw
[Slack Logo]: https://github.com/sintel-dev/Orion/blob/master/docs/images/slack.png
- Homepage: https://github.com/signals-dev/zephyr
# Overview
The **Zephyr** library is a framework designed to assist in the
generation of machine learning problems for wind farms operations data by analyzing past
occurrences of events.
The main features of **Zephyr** are:
* **EntitySet creation**: tools designed to represent wind farm data and the relationship
between different tables. We have functions to create EntitySets for datasets with PI data
and datasets using SCADA data.
* **Labeling Functions**: a collection of functions, as well as tools to create custom versions
of them, ready to be used to analyze past operations data in the search for occurrences of
specific types of events in the past.
* **Prediction Engineering**: a flexible framework designed to apply labeling functions on
wind turbine operations data in a number of different ways to create labels for custom
Machine Learning problems.
* **Feature Engineering**: a guide to using Featuretools to apply automated feature engineerinig
to wind farm data.
# Install
## Requirements
**Zephyr** has been developed and runs on Python 3.6 and 3.7.
Also, although it is not strictly required, the usage of a [virtualenv](
https://virtualenv.pypa.io/en/latest/) is highly recommended in order to avoid interfering
with other software installed in the system where you are trying to run **Zephyr**.
## Download and Install
**Zephyr** can be installed locally using [pip](https://pip.pypa.io/en/stable/) with
the following command:
```bash
pip install zephyr-ml
```
If you want to install from source or contribute to the project please read the
[Contributing Guide](CONTRIBUTING.rst).
# Quickstart
In this short tutorial we will guide you through a series of steps that will help you
getting started with **Zephyr**.
## 1. Loading the data
The first step we will be to use preprocessed data to create an EntitySet. Depending on the
type of data, we will either the `zephyr_ml.create_pidata_entityset` or `zephyr_ml.create_scada_entityset`
functions.
**NOTE**: if you cloned the **Zephyr** repository, you will find some demo data inside the
`notebooks/data` folder which has been preprocessed to fit the `create_entityset` data
requirements.
```python3
import os
import pandas as pd
from zephyr_ml import create_scada_entityset
data_path = 'notebooks/data'
data = {
'turbines': pd.read_csv(os.path.join(data_path, 'turbines.csv')),
'alarms': pd.read_csv(os.path.join(data_path, 'alarms.csv')),
'work_orders': pd.read_csv(os.path.join(data_path, 'work_orders.csv')),
'stoppages': pd.read_csv(os.path.join(data_path, 'stoppages.csv')),
'notifications': pd.read_csv(os.path.join(data_path, 'notifications.csv')),
'scada': pd.read_csv(os.path.join(data_path, 'scada.csv'))
}
scada_es = create_scada_entityset(data)
```
This will load the turbine, alarms, stoppages, work order, notifications, and SCADA data, and return it
as an EntitySet.
```
Entityset: SCADA data
DataFrames:
turbines [Rows: 1, Columns: 10]
alarms [Rows: 2, Columns: 9]
work_orders [Rows: 2, Columns: 20]
stoppages [Rows: 2, Columns: 16]
notifications [Rows: 2, Columns: 15]
scada [Rows: 2, Columns: 5]
Relationships:
alarms.COD_ELEMENT -> turbines.COD_ELEMENT
stoppages.COD_ELEMENT -> turbines.COD_ELEMENT
work_orders.COD_ELEMENT -> turbines.COD_ELEMENT
scada.COD_ELEMENT -> turbines.COD_ELEMENT
notifications.COD_ORDER -> work_orders.COD_ORDER
```
## 2. Selecting a Labeling Function
The second step will be to choose an adequate **Labeling Function**.
We can see the list of available labeling functions using the `zephyr_ml.labeling.get_labeling_functions`
function.
```python3
from zephyr_ml import labeling
labeling.get_labeling_functions()
```
This will return us a dictionary with the name and a short description of each available
function.
```
{'brake_pad_presence': 'Calculates the total power loss over the data slice.',
'converter_replacement_presence': 'Calculates the converter replacement presence.',
'total_power_loss': 'Calculates the total power loss over the data slice.'}
```
In this case, we will choose the `total_power_loss` function, which calculates the total
amount of power lost over a slice of time.
## 3. Generate Target Times
Once we have loaded the data and the Labeling Function, we are ready to start using
the `zephyr_ml.generate_labels` function to generate a Target Times table.
```python3
from zephyr_ml import DataLabeler
data_labeler = DataLabeler(labeling.labeling_functions.total_power_loss)
target_times, metadata = data_labeler.generate_label_times(scada_es)
```
This will return us a `compose.LabelTimes` containing the three columns required to start
working on a Machine Learning problem: the turbine ID (COD_ELEMENT), the cutoff time (time) and the label.
```
COD_ELEMENT time label
0 0 2022-01-01 45801.0
```
## 4. Feature Engineering
Using EntitySets and LabelTimes allows us to easily use Featuretools for automatic feature generation.
```python3
import featuretools as ft
feature_matrix, features = ft.dfs(
entityset=scada_es,
target_dataframe_name='turbines',
cutoff_time_in_index=True,
cutoff_time=target_times,
max_features=20
)
```
Then we get a list of features and the computed `feature_matrix`.
```
TURBINE_PI_ID TURBINE_LOCAL_ID TURBINE_SAP_COD DES_CORE_ELEMENT SITE DES_CORE_PLANT ... MODE(alarms.COD_STATUS) MODE(alarms.DES_NAME) MODE(alarms.DES_TITLE) NUM_UNIQUE(alarms.COD_ALARM) NUM_UNIQUE(alarms.COD_ALARM_INT) label
COD_ELEMENT time ...
0 2022-01-01 TA00 A0 LOC000 T00 LOCATION LOC ... Alarm1 Alarm1 Description of alarm 1 1 1 45801.0
[1 rows x 21 columns]
```
## 5. Modeling
Once we have the feature matrix, we can train a model using the Zephyr interface where you can train, infer, and evaluate a pipeline.
First, we need to prepare our dataset for training by creating ``X`` and ``y`` variables and one-hot encoding features.
```python3
y = list(feature_matrix.pop('label'))
X = pd.get_dummies(feature_matrix).values
```
In this example, we will use an 'xgb' regression pipeline to predict total power loss.
```python3
from zephyr_ml import Zephyr
pipeline_name = 'xgb_regressor'
zephyr = Zephyr(pipeline_name)
```
To train the pipeline, we simply use the `fit` function.
```python3
zephyr.fit(X, y)
```
After it finished training, we can make prediciton using `predict`
```python3
y_pred = zephyr.predict(X)
```
We can also use ``zephyr.evaluate`` to obtain the performance of the pipeline.
# What's Next?
If you want to continue learning about **Zephyr** and all its
features please have a look at the tutorials found inside the [notebooks folder](
https://github.com/signals-dev/zephyr/tree/main/notebooks).
# History
## 0.0.3 - 2024-03-12
SigPro supporting python 3.9, 3.10, and 3.11
* add gearbox_replace_presence labeling function - [Issue #21](https://github.com/signals-dev/Zephyr/issues/21) by @SaraPido
* Update python versions - [Issue #20](https://github.com/signals-dev/Zephyr/issues/20) by @SaraPido
## 0.0.2 - 2023-05-09
SigPro integration for processing signals
* Integrating SigPro - [Issue #7](https://github.com/signals-dev/Zephyr/issues/7) by @frances-h @sarahmish
* Add options to xgb pipeline - [Issue #5](https://github.com/signals-dev/Zephyr/issues/5) by @sarahmish
## 0.0.1 - 2023-03-02
New modeling module using Zephyr class
* Expand GH action tests - [Issue #4](https://github.com/signals-dev/Zephyr/issues/4) by @sarahmish
* Add XGB Pipeline - [Issue #1](https://github.com/signals-dev/Zephyr/issues/1) by @sarahmish
## 0.0.0 - 2022-11-17
First full release
* Prediction Engineering Framework by @frances-h
* EntitySet creation by @frances-h
* DataLabeler and initial labeling functions by @frances-h
Raw data
{
"_id": null,
"home_page": "https://github.com/sintel-dev/zephyr",
"name": "zephyr-ml",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8,<3.12",
"maintainer_email": "",
"keywords": "zephyr Draco Prediction Engineering",
"author": "MIT Data To AI Lab",
"author_email": "dai-lab@mit.edu",
"download_url": "https://files.pythonhosted.org/packages/ab/6f/46abc31fd6941183f5dabc7c9b8c66bf47ece81c2af43530d6673df9faa5/zephyr-ml-0.0.3.tar.gz",
"platform": null,
"description": "<p align=\"left\">\n<img width=15% src=\"https://dai.lids.mit.edu/wp-content/uploads/2018/06/Logo_DAI_highres.png\" alt=\"DAI-Lab\" />\n<i>A project from Data to AI Lab at MIT.</i>\n</p>\n\n<!-- Uncomment these lines after releasing the package to PyPI for version and downloads badges -->\n<!--[](https://pypi.python.org/pypi/zephyr_ml)-->\n<!--[](https://pepy.tech/project/zephyr_ml)-->\n<!--[](https://travis-ci.org/signals-dev/zephyr)-->\n<!--[](https://codecov.io/gh/signals-dev/zephyr)-->\n\n# Zephyr\n\nA machine learning library for assisting in the generation of machine learning problems for wind farms operations data by analyzing past occurrences of events.\n\n | Important Links | |\n | ----------------------------------- | -------------------------------------------------------------------- |\n | :computer: **[Website]** | Check out the Sintel Website for more information about the project. |\n | :book: **[Documentation]** | Quickstarts, User and Development Guides, and API Reference. |\n | :star: **[Tutorials]** | Checkout our notebooks |\n | :octocat: **[Repository]** | The link to the Github Repository of this library. |\n | :scroll: **[License]** | The repository is published under the MIT License. |\n | :keyboard: **[Development Status]** | This software is in its Pre-Alpha stage. |\n | ![][Slack Logo] **[Community]** | Join our Slack Workspace for announcements and discussions. |\n\n [Website]: https://sintel.dev/\n [Documentation]: https://dtail.gitbook.io/zephyr/\n [Repository]: https://github.com/sintel-dev/Zephyr\n [Tutorials]: https://github.com/sintel-dev/Zephyr/blob/master/notebooks\n [License]: https://github.com/sintel-dev/Zephyr/blob/master/LICENSE\n [Development Status]: https://pypi.org/search/?c=Development+Status+%3A%3A+2+-+Pre-Alpha\n [Community]: https://join.slack.com/t/sintel-space/shared_invite/zt-q147oimb-4HcphcxPfDAM0O9_4PaUtw\n [Slack Logo]: https://github.com/sintel-dev/Orion/blob/master/docs/images/slack.png\n\n - Homepage: https://github.com/signals-dev/zephyr\n\n# Overview\n\nThe **Zephyr** library is a framework designed to assist in the\ngeneration of machine learning problems for wind farms operations data by analyzing past\noccurrences of events.\n\nThe main features of **Zephyr** are:\n\n* **EntitySet creation**: tools designed to represent wind farm data and the relationship\nbetween different tables. We have functions to create EntitySets for datasets with PI data\nand datasets using SCADA data.\n* **Labeling Functions**: a collection of functions, as well as tools to create custom versions\nof them, ready to be used to analyze past operations data in the search for occurrences of\nspecific types of events in the past.\n* **Prediction Engineering**: a flexible framework designed to apply labeling functions on\nwind turbine operations data in a number of different ways to create labels for custom\nMachine Learning problems.\n* **Feature Engineering**: a guide to using Featuretools to apply automated feature engineerinig\nto wind farm data.\n\n# Install\n\n## Requirements\n\n**Zephyr** has been developed and runs on Python 3.6 and 3.7.\n\nAlso, although it is not strictly required, the usage of a [virtualenv](\nhttps://virtualenv.pypa.io/en/latest/) is highly recommended in order to avoid interfering\nwith other software installed in the system where you are trying to run **Zephyr**.\n\n## Download and Install\n\n**Zephyr** can be installed locally using [pip](https://pip.pypa.io/en/stable/) with\nthe following command:\n\n```bash\npip install zephyr-ml\n```\n\nIf you want to install from source or contribute to the project please read the\n[Contributing Guide](CONTRIBUTING.rst).\n\n# Quickstart\n\nIn this short tutorial we will guide you through a series of steps that will help you\ngetting started with **Zephyr**.\n\n## 1. Loading the data\n\nThe first step we will be to use preprocessed data to create an EntitySet. Depending on the\ntype of data, we will either the `zephyr_ml.create_pidata_entityset` or `zephyr_ml.create_scada_entityset`\nfunctions.\n\n**NOTE**: if you cloned the **Zephyr** repository, you will find some demo data inside the\n`notebooks/data` folder which has been preprocessed to fit the `create_entityset` data\nrequirements.\n\n```python3\nimport os\nimport pandas as pd\nfrom zephyr_ml import create_scada_entityset\n\ndata_path = 'notebooks/data'\n\ndata = {\n 'turbines': pd.read_csv(os.path.join(data_path, 'turbines.csv')),\n 'alarms': pd.read_csv(os.path.join(data_path, 'alarms.csv')),\n 'work_orders': pd.read_csv(os.path.join(data_path, 'work_orders.csv')),\n 'stoppages': pd.read_csv(os.path.join(data_path, 'stoppages.csv')),\n 'notifications': pd.read_csv(os.path.join(data_path, 'notifications.csv')),\n 'scada': pd.read_csv(os.path.join(data_path, 'scada.csv'))\n}\n\nscada_es = create_scada_entityset(data)\n```\n\nThis will load the turbine, alarms, stoppages, work order, notifications, and SCADA data, and return it\nas an EntitySet.\n\n```\nEntityset: SCADA data\n DataFrames:\n turbines [Rows: 1, Columns: 10]\n alarms [Rows: 2, Columns: 9]\n work_orders [Rows: 2, Columns: 20]\n stoppages [Rows: 2, Columns: 16]\n notifications [Rows: 2, Columns: 15]\n scada [Rows: 2, Columns: 5]\n Relationships:\n alarms.COD_ELEMENT -> turbines.COD_ELEMENT\n stoppages.COD_ELEMENT -> turbines.COD_ELEMENT\n work_orders.COD_ELEMENT -> turbines.COD_ELEMENT\n scada.COD_ELEMENT -> turbines.COD_ELEMENT\n notifications.COD_ORDER -> work_orders.COD_ORDER\n```\n\n## 2. Selecting a Labeling Function\n\nThe second step will be to choose an adequate **Labeling Function**.\n\nWe can see the list of available labeling functions using the `zephyr_ml.labeling.get_labeling_functions`\nfunction.\n\n```python3\nfrom zephyr_ml import labeling\n\nlabeling.get_labeling_functions()\n```\n\nThis will return us a dictionary with the name and a short description of each available\nfunction.\n\n```\n{'brake_pad_presence': 'Calculates the total power loss over the data slice.',\n 'converter_replacement_presence': 'Calculates the converter replacement presence.',\n 'total_power_loss': 'Calculates the total power loss over the data slice.'}\n```\n\nIn this case, we will choose the `total_power_loss` function, which calculates the total\namount of power lost over a slice of time.\n\n## 3. Generate Target Times\n\nOnce we have loaded the data and the Labeling Function, we are ready to start using\nthe `zephyr_ml.generate_labels` function to generate a Target Times table.\n\n\n```python3\nfrom zephyr_ml import DataLabeler\n\ndata_labeler = DataLabeler(labeling.labeling_functions.total_power_loss)\ntarget_times, metadata = data_labeler.generate_label_times(scada_es)\n```\n\nThis will return us a `compose.LabelTimes` containing the three columns required to start\nworking on a Machine Learning problem: the turbine ID (COD_ELEMENT), the cutoff time (time) and the label.\n\n```\n COD_ELEMENT time label\n0 0 2022-01-01 45801.0\n```\n\n## 4. Feature Engineering\nUsing EntitySets and LabelTimes allows us to easily use Featuretools for automatic feature generation.\n\n```python3\nimport featuretools as ft\n\nfeature_matrix, features = ft.dfs(\n entityset=scada_es,\n target_dataframe_name='turbines',\n cutoff_time_in_index=True,\n cutoff_time=target_times,\n max_features=20\n)\n```\n\nThen we get a list of features and the computed `feature_matrix`.\n\n```\n TURBINE_PI_ID TURBINE_LOCAL_ID TURBINE_SAP_COD DES_CORE_ELEMENT SITE DES_CORE_PLANT ... MODE(alarms.COD_STATUS) MODE(alarms.DES_NAME) MODE(alarms.DES_TITLE) NUM_UNIQUE(alarms.COD_ALARM) NUM_UNIQUE(alarms.COD_ALARM_INT) label\nCOD_ELEMENT time ... \n0 2022-01-01 TA00 A0 LOC000 T00 LOCATION LOC ... Alarm1 Alarm1 Description of alarm 1 1 1 45801.0\n\n[1 rows x 21 columns]\n```\n\n\n## 5. Modeling\n\nOnce we have the feature matrix, we can train a model using the Zephyr interface where you can train, infer, and evaluate a pipeline. \nFirst, we need to prepare our dataset for training by creating ``X`` and ``y`` variables and one-hot encoding features.\n\n```python3\ny = list(feature_matrix.pop('label'))\nX = pd.get_dummies(feature_matrix).values\n```\n\nIn this example, we will use an 'xgb' regression pipeline to predict total power loss.\n\n```python3\nfrom zephyr_ml import Zephyr\n\npipeline_name = 'xgb_regressor'\n\nzephyr = Zephyr(pipeline_name)\n```\n\nTo train the pipeline, we simply use the `fit` function.\n```python3\nzephyr.fit(X, y)\n```\n\nAfter it finished training, we can make prediciton using `predict`\n\n```python3\ny_pred = zephyr.predict(X)\n```\n\nWe can also use ``zephyr.evaluate`` to obtain the performance of the pipeline.\n\n# What's Next?\n\nIf you want to continue learning about **Zephyr** and all its\nfeatures please have a look at the tutorials found inside the [notebooks folder](\nhttps://github.com/signals-dev/zephyr/tree/main/notebooks).\n\n\n# History\n\n## 0.0.3 - 2024-03-12\n\nSigPro supporting python 3.9, 3.10, and 3.11\n\n* add gearbox_replace_presence labeling function - [Issue #21](https://github.com/signals-dev/Zephyr/issues/21) by @SaraPido\n* Update python versions - [Issue #20](https://github.com/signals-dev/Zephyr/issues/20) by @SaraPido\n\n\n## 0.0.2 - 2023-05-09\n\nSigPro integration for processing signals\n\n* Integrating SigPro - [Issue #7](https://github.com/signals-dev/Zephyr/issues/7) by @frances-h @sarahmish \n* Add options to xgb pipeline - [Issue #5](https://github.com/signals-dev/Zephyr/issues/5) by @sarahmish\n\n\n## 0.0.1 - 2023-03-02\n\nNew modeling module using Zephyr class\n\n* Expand GH action tests - [Issue #4](https://github.com/signals-dev/Zephyr/issues/4) by @sarahmish \n* Add XGB Pipeline - [Issue #1](https://github.com/signals-dev/Zephyr/issues/1) by @sarahmish\n\n\n## 0.0.0 - 2022-11-17\n\nFirst full release\n\n* Prediction Engineering Framework by @frances-h \n* EntitySet creation by @frances-h \n* DataLabeler and initial labeling functions by @frances-h \n",
"bugtrack_url": null,
"license": "",
"summary": "Prediction engineering methods for Draco.",
"version": "0.0.3",
"project_urls": {
"Homepage": "https://github.com/sintel-dev/zephyr"
},
"split_keywords": [
"zephyr",
"draco",
"prediction",
"engineering"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8e33eb6de6a5173e85abd09a329c48cb191b01cba1c34b414360c7411404abb2",
"md5": "a19f7df646df861396a2caa809e3f02a",
"sha256": "0164b996f49730f20a0ef5d0e52a590b785db39fba256fc92c75f2c15835267b"
},
"downloads": -1,
"filename": "zephyr_ml-0.0.3-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "a19f7df646df861396a2caa809e3f02a",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": ">=3.8,<3.12",
"size": 25925,
"upload_time": "2024-03-12T18:14:02",
"upload_time_iso_8601": "2024-03-12T18:14:02.494253Z",
"url": "https://files.pythonhosted.org/packages/8e/33/eb6de6a5173e85abd09a329c48cb191b01cba1c34b414360c7411404abb2/zephyr_ml-0.0.3-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ab6f46abc31fd6941183f5dabc7c9b8c66bf47ece81c2af43530d6673df9faa5",
"md5": "6b24b193f1755ba53933b1e76667b46d",
"sha256": "c6695385fc8a7768f856ae3b148cf80b25d6158a5bf66c67bdfa01f6616e33ac"
},
"downloads": -1,
"filename": "zephyr-ml-0.0.3.tar.gz",
"has_sig": false,
"md5_digest": "6b24b193f1755ba53933b1e76667b46d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8,<3.12",
"size": 73173,
"upload_time": "2024-03-12T18:14:04",
"upload_time_iso_8601": "2024-03-12T18:14:04.043947Z",
"url": "https://files.pythonhosted.org/packages/ab/6f/46abc31fd6941183f5dabc7c9b8c66bf47ece81c2af43530d6673df9faa5/zephyr-ml-0.0.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-12 18:14:04",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "sintel-dev",
"github_project": "zephyr",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "zephyr-ml"
}
JSON
