<br>
<p align="center">
<h2 align="center"> Atlantic - Automated Data Preprocessing Framework for Supervised Machine Learning
<br>
## Framework Contextualization <a name = "ta"></a>
The `Atlantic` project constitutes an comprehensive and objective approach to simplify and automate data processing through the integration and objectively validated application of various preprocessing mechanisms, ranging from feature engineering, automated feature selection, multiple encoding versions and null imputation methods. The optimization methodology of this framework follows a evaluation structured in tree based models ensembles.
This project aims at providing the following application capabilities:
* General applicability on tabular datasets: The developed preprocessing procedures are applicable on any data table associated with Supervised Machine Learning scopes regardless of the properties or specifications of the Dataset features.
* Automated treatment of tabular data associated with predictive analysis: It implements a global and carefully validated tested data treatment based on the characteristics of each Dataset input columns, assuming its identification with the associated target column.
* Robustness and improvement of predictive results: The implementation of the `atlantic` automated data preprocessing pipeline aims at improving predictive performance directly associated with the processing methods implemented based on the Dataset properties.
#### Main Development Tools <a name = "pre1"></a>
Major frameworks used to built this project:
* [H2O.ai](https://docs.h2o.ai/h2o/latest-stable/h2o-docs/automl.html)
* [Scikit-learn](https://scikit-learn.org/stable/)
* [XGBoost](https://xgboost.readthedocs.io/en/stable/)
* [Optuna](https://optuna.org/)
* [Pandas](https://pandas.pydata.org/)
## Framework Architecture <a name = "ta"></a>
<p align="center">
<img src="https://i.ibb.co/C9dWJmk/ATL-Architecture-Final.png" align="center" width="700" height="680" />
</p>
## Where to get it <a name = "ta"></a>
Binary installer for the latest released version is available at the Python Package Index [PyPI](https://pypi.org/project/atlantic/).
The source code is currently hosted on GitHub at: https://github.com/TsLu1s/Atlantic
## Installation
To install this package from Pypi repository run the following command:
```
pip install atlantic
```
# Usage Examples
## 1. Atlantic - Automated Data Preprocessing Pipeline
In order to be able to apply the automated preprocessing `atlantic` pipeline you need first to import the package.
The following needed step is to load a dataset and define your to be predicted target column name into the variable `target` and define split ratio for your Train and Validation subsets.
You can customize the `fit_processing` method by altering the following running pipeline parameters:
* split_ratio: Division ratio in which the preprocessing methods will be evaluated within the loaded Dataset.
* relevance: Minimal value of the total sum of relative variable\feature importance percentage selected in the `H2O AutoML feature selection` step.
* h2o_fs_models: Quantity of models generated for competition in step `H2O AutoML feature selection` to evaluate the relative importance of each feature (only leaderboard model is selected for evaluation).
* encoding_fs: You can choose if you want to encond your features in order to reduce loading time in `H2O AutoML feature selection` step. If in `True` mode label encoding is applied to categorical features.
* vif_ratio: This value defines the minimal `threshold` for Variance Inflation Factor filtering (default value=10).
Importante Notes:
* Default predictive evaluation metric for regression contexts is `Mean Absolute Error` and classification is `Accuracy`.
* Although functional, `Atlantic` data processing is not optimized for big data purposes yet.
* Major update is now available in **versions>=1.1.0**
```py
from atlantic.pipeline import ATLpipeline
import pandas as pd
from sklearn.model_selection import train_test_split
import warnings
warnings.filterwarnings("ignore", category=Warning) # -> For a clean console
data = pd.read_csv('csv_directory_path', encoding='latin', delimiter=',') # Dataframe Loading Example
train,test = train_test_split(data, train_size=0.8)
train,test = train.reset_index(drop=True), test.reset_index(drop=True) # -> Required
### Fit Data Processing
atl = ATLpipeline(X=train, # X:pd.DataFrame, target:str="Target_Column"
target="Target Column")
atl.fit_processing(split_ratio=0.75, # split_ratio:float=0.75, relevance:float=0.99 [0.5,1]
relevance=0.99, # h2o_fs_models:int [1,50], encoding_fs:bool=True\False
h2o_fs_models=7, # vif_ratio:float=10.0 [3,30]
encoding_fs=True,
vif_ratio=10.0)
### Transform Data Processing
train = atl.data_processing(X=train)
test = atl.data_processing(X=test)
### Export Atlantic Preprocessing Metadata
import pickle
output = open("fit_atl.pkl", 'wb')
pickle.dump(atl, output)
```
## 2. Atlantic - Preprocessing Data
### 2.1 Encoding Versions
There are multiple preprocessing methods available to direct use. This package provides upgrated encoding `LabelEncoder`, `OneHotEncoder` and [IDF](https://pypi.org/project/cane/) methods with an automatic multicolumn application.
```py
from atlantic.processing import AutoLabelEncoder, AutoIdfEncoder, AutoOneHotEncoder
import pandas as pd
from sklearn.model_selection import train_test_split
train,test = train_test_split(data, train_size=0.8)
train,test = train.reset_index(drop=True), test.reset_index(drop=True) # Required
target = "Target_Column" # -> target feature name
cat_cols=[col for col in data.select_dtypes(include=['object']).columns if col != target]
### Encoders
## Create Label Encoder
encoder = AutoLabelEncoder()
## Create IDF Encoder
encoder = AutoIdfEncoder()
## Create One-hot Encoder
encoder = AutoOneHotEncoder()
## Fit
encoder.fit(train[cat_cols])
# Transform the DataFrame using Label\IDF\One-hot Encoding
train = encoder.transform(X=train)
test = encoder.transform(X=test)
# Label Encoding : Perform an inverse transform to convert it back the categorical columns values
test = encoder.inverse_transform(X=test)
# IDF & One-hot Encoding : Perform an inverse transform to convert it back the categorical columns values
# Note: Only decodes the last transformed Dataframe
test = encoder.inverse_transform()
```
### 2.2 Feature Selection Methods
You can get filter your most valuable features from the dataset via this 2 feature selection methods:
* [H2O AutoML Feature Selection](https://docs.h2o.ai/h2o/latest-stable/h2o-docs/variable-importance.html) - This method is based of variable importance evaluation and calculation for tree-based models in H2Os AutoML and it can be customized by use of the following parameters:
* relevance: Minimal value of the total sum of relative variable\feature importance percentage selected.
* h2o_fs_models: Quantity of models generated for competition to evaluate the relative importance of each feature (only leaderboard model will be selected for evaluation).
* encoding_fs: You can choose if you want to encond your features in order to reduce loading time. If in `True` mode label encoding is applied to categorical features.
* [VIF Feature Selection (Variance Inflation Factor)](https://www.investopedia.com/terms/v/variance-inflation-factor.asp) - Variance inflation factor aims at measuring the amount of multicollinearity in a set of multiple regression variables or features, therefore for this filtering method to be applied all input variables need to be of numeric type. It can be customized by changing the column filtering treshold `vif_threshold` designated with a default value of 10.
```py
from atlantic.selector import Selector
fs = Selector(X=train,target="Target_Column")
cols_vif = fs.feature_selection_vif(vif_threshold=10.0) # X: Only numerical values allowed & No nans allowed in VIF
selected_cols, selected_importance = fs.feature_selection_h2o(relevance=0.99, # relevance:float [0.5,1], h2o_fs_models:int [1,50]
h2o_fs_models=7, # encoding_fs:bool=True/False
encoding_fs=True)
```
### 2.3 Null Imputation Auxiliar Methods
Simplified and automated multivariate null imputation methods based from [Sklearn](https://scikit-learn.org/stable/modules/impute.html) are also provided and applicable, as following:
```py
## Simplified Null Imputation (Only numeric features)
from atlantic.imputation import AutoSimpleImputer, AutoKNNImputer, AutoIterativeImputer
# Example usage of AutoSimpleImputer
simple_imputer = AutoSimpleImputer(strategy='mean')
simple_imputer.fit(train) # Fit on the Train DataFrame
df_imputed = simple_imputer.transform(train.copy()) # Transform the Train DataFrame
df_imputed_test = simple_imputer.transform(test.copy()) # Transform the Test DataFrame
# Example usage of AutoKNNImputer
knn_imputer = AutoKNNImputer(n_neighbors=3,
weights="uniform")
knn_imputer.fit(train) # Fit on the Train DataFrame
df_imputed = knn_imputer.transform(train.copy()) # Transform the Train DataFrame
df_imputed_test = knn_imputer.transform(test.copy()) # Transform the Test DataFrame
# Example usage of AutoIterativeImputer
iterative_imputer = AutoIterativeImputer(max_iter=10,
random_state=0,
initial_strategy="mean",
imputation_order="ascending")
iterative_imputer.fit(train) # Fit on the Train DataFrame
df_imputed = iterative_imputer.transform(train.copy()) # Transform the Train DataFrame
df_imputed_test = iterative_imputer.transform(test.copy()) # Transform the Test DataFrame
```
## Citation
Feel free to cite Atlantic as following:
```
@article{SANTOS2023100532,
author = {Luis Santos and Luis Ferreira}
title = {Atlantic - Automated data preprocessing framework for supervised machine learning},
journal = {Software Impacts},
volume = {17},
year = {2023},
issn = {2665-9638},
doi = {http://dx.doi.org/10.1016/j.simpa.2023.100532},
url = {https://www.sciencedirect.com/science/article/pii/S2665963823000696}
}
```
## License
Distributed under the MIT License. See [LICENSE](https://github.com/TsLu1s/Atlantic/blob/main/LICENSE) for more information.
## Contact
[Luis Santos - LinkedIn](https://www.linkedin.com/in/lu%C3%ADsfssantos/)
Raw data
{
"_id": null,
"home_page": "https://github.com/TsLu1s/Atlantic",
"name": "atlantic",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "data science,machine learning,data processing,predictive modeling,data preprocessing,automated data preprocessing,automated machine learning,automl",
"author": "Lu\u00eds Santos",
"author_email": "luisf_ssantos@hotmail.com",
"download_url": "",
"platform": null,
"description": "<br>\r\n<p align=\"center\">\r\n <h2 align=\"center\"> Atlantic - Automated Data Preprocessing Framework for Supervised Machine Learning\r\n <br>\r\n \r\n## Framework Contextualization <a name = \"ta\"></a>\r\n\r\nThe `Atlantic` project constitutes an comprehensive and objective approach to simplify and automate data processing through the integration and objectively validated application of various preprocessing mechanisms, ranging from feature engineering, automated feature selection, multiple encoding versions and null imputation methods. The optimization methodology of this framework follows a evaluation structured in tree based models ensembles.\r\n\r\nThis project aims at providing the following application capabilities:\r\n\r\n* General applicability on tabular datasets: The developed preprocessing procedures are applicable on any data table associated with Supervised Machine Learning scopes regardless of the properties or specifications of the Dataset features.\r\n\r\n* Automated treatment of tabular data associated with predictive analysis: It implements a global and carefully validated tested data treatment based on the characteristics of each Dataset input columns, assuming its identification with the associated target column.\r\n\r\n* Robustness and improvement of predictive results: The implementation of the `atlantic` automated data preprocessing pipeline aims at improving predictive performance directly associated with the processing methods implemented based on the Dataset properties. \r\n \r\n#### Main Development Tools <a name = \"pre1\"></a>\r\n\r\nMajor frameworks used to built this project: \r\n \r\n* [H2O.ai](https://docs.h2o.ai/h2o/latest-stable/h2o-docs/automl.html)\r\n* [Scikit-learn](https://scikit-learn.org/stable/)\r\n* [XGBoost](https://xgboost.readthedocs.io/en/stable/)\r\n* [Optuna](https://optuna.org/)\r\n* [Pandas](https://pandas.pydata.org/)\r\n\r\n \r\n## Framework Architecture <a name = \"ta\"></a>\r\n\r\n<p align=\"center\">\r\n <img src=\"https://i.ibb.co/C9dWJmk/ATL-Architecture-Final.png\" align=\"center\" width=\"700\" height=\"680\" />\r\n</p> \r\n\r\n## Where to get it <a name = \"ta\"></a>\r\n\r\nBinary installer for the latest released version is available at the Python Package Index [PyPI](https://pypi.org/project/atlantic/). \r\n\r\nThe source code is currently hosted on GitHub at: https://github.com/TsLu1s/Atlantic\r\n\r\n## Installation \r\n\r\nTo install this package from Pypi repository run the following command:\r\n\r\n```\r\npip install atlantic\r\n```\r\n\r\n# Usage Examples\r\n \r\n## 1. Atlantic - Automated Data Preprocessing Pipeline\r\n\r\nIn order to be able to apply the automated preprocessing `atlantic` pipeline you need first to import the package. \r\nThe following needed step is to load a dataset and define your to be predicted target column name into the variable `target` and define split ratio for your Train and Validation subsets.\r\nYou can customize the `fit_processing` method by altering the following running pipeline parameters:\r\n* split_ratio: Division ratio in which the preprocessing methods will be evaluated within the loaded Dataset.\r\n* relevance: Minimal value of the total sum of relative variable\\feature importance percentage selected in the `H2O AutoML feature selection` step.\r\n* h2o_fs_models: Quantity of models generated for competition in step `H2O AutoML feature selection` to evaluate the relative importance of each feature (only leaderboard model is selected for evaluation).\r\n* encoding_fs: You can choose if you want to encond your features in order to reduce loading time in `H2O AutoML feature selection` step. If in `True` mode label encoding is applied to categorical features.\r\n* vif_ratio: This value defines the minimal `threshold` for Variance Inflation Factor filtering (default value=10).\r\n \r\nImportante Notes:\r\n \r\n* Default predictive evaluation metric for regression contexts is `Mean Absolute Error` and classification is `Accuracy`.\r\n* Although functional, `Atlantic` data processing is not optimized for big data purposes yet.\r\n* Major update is now available in **versions>=1.1.0**\r\n \r\n```py\r\n \r\nfrom atlantic.pipeline import ATLpipeline\r\nimport pandas as pd\r\nfrom sklearn.model_selection import train_test_split \r\nimport warnings\r\nwarnings.filterwarnings(\"ignore\", category=Warning) # -> For a clean console\r\n \r\ndata = pd.read_csv('csv_directory_path', encoding='latin', delimiter=',') # Dataframe Loading Example\r\n\r\ntrain,test = train_test_split(data, train_size=0.8)\r\ntrain,test = train.reset_index(drop=True), test.reset_index(drop=True) # -> Required \r\n\r\n### Fit Data Processing\r\n\r\natl = ATLpipeline(X=train, # X:pd.DataFrame, target:str=\"Target_Column\"\r\n target=\"Target Column\") \r\n\r\natl.fit_processing(split_ratio=0.75, # split_ratio:float=0.75, relevance:float=0.99 [0.5,1]\r\n relevance=0.99, # h2o_fs_models:int [1,50], encoding_fs:bool=True\\False\r\n h2o_fs_models=7, # vif_ratio:float=10.0 [3,30]\r\n encoding_fs=True,\r\n vif_ratio=10.0)\r\n\r\n### Transform Data Processing\r\n\r\ntrain = atl.data_processing(X=train)\r\ntest = atl.data_processing(X=test)\r\n\r\n### Export Atlantic Preprocessing Metadata\r\n\r\nimport pickle \r\noutput = open(\"fit_atl.pkl\", 'wb')\r\npickle.dump(atl, output)\r\n \r\n``` \r\n\r\n## 2. Atlantic - Preprocessing Data\r\n \r\n### 2.1 Encoding Versions\r\n \r\nThere are multiple preprocessing methods available to direct use. This package provides upgrated encoding `LabelEncoder`, `OneHotEncoder` and [IDF](https://pypi.org/project/cane/) methods with an automatic multicolumn application. \r\n \r\n```py\r\nfrom atlantic.processing import AutoLabelEncoder, AutoIdfEncoder, AutoOneHotEncoder\r\nimport pandas as pd\r\nfrom sklearn.model_selection import train_test_split \r\n\r\ntrain,test = train_test_split(data, train_size=0.8)\r\ntrain,test = train.reset_index(drop=True), test.reset_index(drop=True) # Required\r\n\r\ntarget = \"Target_Column\" # -> target feature name\r\n \r\ncat_cols=[col for col in data.select_dtypes(include=['object']).columns if col != target]\r\n\r\n### Encoders\r\n## Create Label Encoder\r\nencoder = AutoLabelEncoder()\r\n## Create IDF Encoder\r\nencoder = AutoIdfEncoder()\r\n## Create One-hot Encoder\r\nencoder = AutoOneHotEncoder()\r\n\r\n## Fit\r\nencoder.fit(train[cat_cols])\r\n\r\n# Transform the DataFrame using Label\\IDF\\One-hot Encoding\r\ntrain = encoder.transform(X=train)\r\ntest = encoder.transform(X=test)\r\n\r\n# Label Encoding : Perform an inverse transform to convert it back the categorical columns values\r\ntest = encoder.inverse_transform(X=test)\r\n\r\n# IDF & One-hot Encoding : Perform an inverse transform to convert it back the categorical columns values\r\n# Note: Only decodes the last transformed Dataframe\r\ntest = encoder.inverse_transform()\r\n \r\n``` \r\n \r\n### 2.2 Feature Selection Methods\r\n\r\nYou can get filter your most valuable features from the dataset via this 2 feature selection methods:\r\n \r\n* [H2O AutoML Feature Selection](https://docs.h2o.ai/h2o/latest-stable/h2o-docs/variable-importance.html) - This method is based of variable importance evaluation and calculation for tree-based models in H2Os AutoML and it can be customized by use of the following parameters: \r\n * relevance: Minimal value of the total sum of relative variable\\feature importance percentage selected.\r\n * h2o_fs_models: Quantity of models generated for competition to evaluate the relative importance of each feature (only leaderboard model will be selected for evaluation).\r\n * encoding_fs: You can choose if you want to encond your features in order to reduce loading time. If in `True` mode label encoding is applied to categorical features.\r\n \r\n \r\n* [VIF Feature Selection (Variance Inflation Factor)](https://www.investopedia.com/terms/v/variance-inflation-factor.asp) - Variance inflation factor aims at measuring the amount of multicollinearity in a set of multiple regression variables or features, therefore for this filtering method to be applied all input variables need to be of numeric type. It can be customized by changing the column filtering treshold `vif_threshold` designated with a default value of 10.\r\n \r\n \r\n```py \r\nfrom atlantic.selector import Selector\r\n\r\nfs = Selector(X=train,target=\"Target_Column\")\r\n\r\ncols_vif = fs.feature_selection_vif(vif_threshold=10.0) # X: Only numerical values allowed & No nans allowed in VIF\r\n\r\nselected_cols, selected_importance = fs.feature_selection_h2o(relevance=0.99, # relevance:float [0.5,1], h2o_fs_models:int [1,50]\r\n h2o_fs_models=7, # encoding_fs:bool=True/False\r\n encoding_fs=True)\r\n```\r\n \r\n### 2.3 Null Imputation Auxiliar Methods\r\n \r\nSimplified and automated multivariate null imputation methods based from [Sklearn](https://scikit-learn.org/stable/modules/impute.html) are also provided and applicable, as following:\r\n \r\n```py \r\n\r\n## Simplified Null Imputation (Only numeric features)\r\nfrom atlantic.imputation import AutoSimpleImputer, AutoKNNImputer, AutoIterativeImputer\r\n\r\n# Example usage of AutoSimpleImputer\r\nsimple_imputer = AutoSimpleImputer(strategy='mean')\r\nsimple_imputer.fit(train) # Fit on the Train DataFrame\r\ndf_imputed = simple_imputer.transform(train.copy()) # Transform the Train DataFrame\r\ndf_imputed_test = simple_imputer.transform(test.copy()) # Transform the Test DataFrame\r\n\r\n# Example usage of AutoKNNImputer\r\nknn_imputer = AutoKNNImputer(n_neighbors=3,\r\n weights=\"uniform\")\r\nknn_imputer.fit(train) # Fit on the Train DataFrame\r\ndf_imputed = knn_imputer.transform(train.copy()) # Transform the Train DataFrame\r\ndf_imputed_test = knn_imputer.transform(test.copy()) # Transform the Test DataFrame\r\n\r\n# Example usage of AutoIterativeImputer\r\niterative_imputer = AutoIterativeImputer(max_iter=10, \r\n random_state=0, \r\n initial_strategy=\"mean\", \r\n imputation_order=\"ascending\")\r\niterative_imputer.fit(train) # Fit on the Train DataFrame\r\ndf_imputed = iterative_imputer.transform(train.copy()) # Transform the Train DataFrame\r\ndf_imputed_test = iterative_imputer.transform(test.copy()) # Transform the Test DataFrame\r\n\r\n``` \r\n\r\n## Citation\r\n\r\nFeel free to cite Atlantic as following:\r\n\r\n```\r\n\r\n@article{SANTOS2023100532,\r\n author = {Luis Santos and Luis Ferreira}\r\n title = {Atlantic - Automated data preprocessing framework for supervised machine learning},\r\n journal = {Software Impacts},\r\n volume = {17},\r\n year = {2023},\r\n issn = {2665-9638},\r\n doi = {http://dx.doi.org/10.1016/j.simpa.2023.100532},\r\n url = {https://www.sciencedirect.com/science/article/pii/S2665963823000696}\r\n}\r\n\r\n```\r\n\r\n \r\n## License\r\n\r\nDistributed under the MIT License. See [LICENSE](https://github.com/TsLu1s/Atlantic/blob/main/LICENSE) for more information.\r\n\r\n## Contact \r\n \r\n[Luis Santos - LinkedIn](https://www.linkedin.com/in/lu%C3%ADsfssantos/)\r\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Atlantic is an automated preprocessing framework for Supervised Machine Learning",
"version": "1.1.25",
"project_urls": {
"Homepage": "https://github.com/TsLu1s/Atlantic"
},
"split_keywords": [
"data science",
"machine learning",
"data processing",
"predictive modeling",
"data preprocessing",
"automated data preprocessing",
"automated machine learning",
"automl"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "45593f434e514b7ca3ab787c44079e3d669674e52c2d6e59f78c89bbd2c733b3",
"md5": "30bea0c096e6f42373f8b740742b36c0",
"sha256": "8236cad90bcb7ae9be077933c1724623f5938aca9504cd7fc40bf84936c909c8"
},
"downloads": -1,
"filename": "atlantic-1.1.25-py3-none-any.whl",
"has_sig": false,
"md5_digest": "30bea0c096e6f42373f8b740742b36c0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 19342,
"upload_time": "2024-01-27T23:02:00",
"upload_time_iso_8601": "2024-01-27T23:02:00.308534Z",
"url": "https://files.pythonhosted.org/packages/45/59/3f434e514b7ca3ab787c44079e3d669674e52c2d6e59f78c89bbd2c733b3/atlantic-1.1.25-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-27 23:02:00",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "TsLu1s",
"github_project": "Atlantic",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "atlantic"
}
JSON
