[](https://clearbox-preprocessor.readthedocs.io/en/latest/?badge=latest)
[](https://badge.fury.io/py/clearbox-preprocessor)
[](https://pepy.tech/project/clearbox-preprocessor)
[](https://github.com/Clearbox-AI/preprocessor)
# Clearbox AI Preprocessor
This repository contains the continuation of the work presented in our series of blogposts *"The whys and hows of data preparation"* ([part 1](https://www.clearbox.ai/blog/2022-01-25-the-whys-and-hows-of-data-preparation), [part 2](https://www.clearbox.ai/blog/2022-02-22-the-whys-and-hows-of-data-preparation-part-2), [part 3](https://www.clearbox.ai/blog/2022-07-05-the-whys-and-hows-of-data-preparation-part-3)).
The new version of Preprocessor exploits Polars library's features to achieve blazing fast tabular data manipulation.
It is possible to input the Preprocessor a `Pandas.DataFrame`, a `Polars.DataFrame` or a `Polars.LazyFrame`.
## Installation
You can install the preprocessor by running the following command:
```shell
$ pip install clearbox-preprocessor
```
## Preprocessing customization
A bunch of options are available to customize the preprocessing.
The `Preprocessor` class features the following input arguments, besides the input dataset:
- `discarding_threshold: float (default = 0.9)`
Float number between 0 and 1 to set the threshold for discarding categorical features.
If more than discarding_threshold * 100 % of values in a categorical feature are different from each other, then the column is discarded.
For example, if discarding_threshold=0.9, a column will be discarded if more than 90% of its values are unique.
- `get_discarded_info: bool (defatult = False)`
When set to 'True', the preprocessor will feature the methods preprocessor.get_discarded_features_reason, which provides information on which columns were discarded and the reason why, and preprocessor.get_single_valued_columns, which provides the values of the single-valued discarded columns.
Note that setting get_discarded_info=True will considerably slow down the processing operation!
The list of discarded columns will be available even if get_discarded_info=False, so consider setting this flag to True only if you need to know why a column was discarded or, if it contained just one value, what that value was.
- `excluded_col: (default = [])`
List containing the names of the columns to be excluded from processing. These columns will be returned in the final dataframe withouth being manipulated.
- `time: (default = None)`
String name of the time column by which to sort the dataframe in case of time series.
- `scaling_method: (default="none")`
Specifies the scaling operation to perform on numerical features.
- "none" : no scaling is applied
- "normalize" : applies normalization to numerical features
- "standardize" : applies standardization to numerical features
- "quantile" : Transforms numerical features using quantiles information
- `num_fill_null: (default = "mean")`
Specifies the value to fill null values with or the strategy for filling null values in numerical features.
- value : fills null values with the specified value
- "mean" : fills null values with the average of the column
- "forward" : fills null values with the previous non-null value in the column
- "backward" : fills null values with the following non-null value in the column
- "min" : fills null values with the minimum value of the column
- "max" : fills null values with the maximum value of the column
- "zero" : fills null values with zeros
- "one" : fills null values with ones
- `n_bins: (default = 0)`
Integer number that determines the number of bins into which numerical features are discretized. When set to 0, the preprocessing step defaults to the scaling method specified in the 'scaling' atgument instead of discretization.
Note that if n_bins is different than 0, discretization will take place instead of scaling, regardless of whether the 'scaling' argument is specified.
### Timeseries
The Prperocessor also features a timeseries manipulation and feature extraction method called `extract_ts_features()`.
This method takes as input:
- the preprocessed dataframe
- the target vector in the form of a `Pandas.Series` or a `Polars.Series`
- the name of the time column
- the name of the id column to group by
It returns the most relevant features selected among a wide range of features.
## Usage
You can start using the Preprocessor by importing it and creating a `Pandas.DataFrame` or a `Polars.LazyFrame`:
```python
import polars as pl
from clearbox_preprocessor import Preprocessor
q = pl.LazyFrame(
{
"cha": ["x", None, "z", "w", "x", "k"],
"int": [123, 124, 223, 431, 435, 432],
"dat": ["2023-1-5T00:34:12.000Z", "2023-2-3T04:31:45.000Z", "2023-2-4T04:31:45.000Z", None, "2023-5-12T21:41:58.000Z", "2023-6-1T17:52:22.000Z"],
"boo": [True, False, None, True, False, False],
"equ": ["a", "a", "a", "a", None, "a"],
"flo": [43.034, 343.1, 224.23, 75.3, None, 83.2],
"str": ["asd", "fgh", "fgh", "", None, "cvb"]
}
).with_columns(pl.col('dat').str.to_datetime("%Y-%m-%dT%H:%M:%S.000Z"))
q.collect()
```
<img src="https://github.com/Clearbox-AI/preprocessor/assets/152599516/c2e1878d-6af3-4157-8a7d-61fdccfde270" alt="image" width="40%" height="auto">
At this point, you can initialize the Preprocessor by passing the LazyFrame or DataFrame created to it and then calling the `transform()` method to materialize the processed dataframe.
Note that if no argument is specified beyond the dataframe *q*, the default settings are employed for preprocessing:
```python
preprocessor = Preprocessor(q)
df = preprocessor.transform(q)
df
```
<img src="https://github.com/Clearbox-AI/preprocessor/assets/152599516/7cd5b6f6-26f9-4af9-8250-751f43cac7d5" alt="image" width="70%" height="auto">
### Customization example
In the following example, when the Preprocessor is initialized:
1. The discarding threshold is lowered from 90% to 80% (a column will be discarded if more than 80% of its values are unique).
2. The discarding featrues informations are stored in the `preprocessor` instance.
3. The column "boo" is excluded from the preprocessing and is preserved unchanged.
4. The scaling method of the numerical features chosen is standardization
5. The fill null strategy for numerical features is "forward".
```python
preprocessor = Preprocessor(q,
get_discarded_info=True,
discarding_threshold = 0.8,
excluded_col = ["boo"],
scaling = "standardize",
num_fill_null = "forward"
)
df = preprocessor.transform(q)
df
```
<img src="https://github.com/Clearbox-AI/preprocessor/assets/152599516/ba61531d-a462-4d58-847a-47127d6050fd" alt="image" width="52%" height="auto">
It is possible to inverse transform the processed dataframe with the method `preprocessor.inverse_transform()`.
```python
preprocessor = Preprocessor(q)
df = preprocessor.transform(q)
inverse_df = preprocessor.inverse_transform(df)
```
If the Processor's argument `get_discarded_info` is set to `True` during initialization, it is possible to call the method `get_discarded_features_reason()` to display the discarded features.
In the case of discarded single-valued columns, the value contained is also displayed and is available in a dictionary called `single_value_columns`, stored in the Preprocessor instance, and can be used as metadata.
```python
preprocessor.get_discarded_features_reason()
```
<img src="https://github.com/Clearbox-AI/preprocessor/assets/152599516/5c5d7a42-d52d-4d32-862d-fc3c95c31d67" alt="image" width="90%" height="auto">
## To do
- [ ] Implement unit tests
Raw data
{
"_id": null,
"home_page": "https://github.com/Clearbox-AI/preprocessor",
"name": "clearbox-preprocessor",
"maintainer": null,
"docs_url": null,
"requires_python": ">3.9",
"maintainer_email": null,
"keywords": null,
"author": "Dario Brunelli",
"author_email": "dario@clearbox.ai",
"download_url": "https://files.pythonhosted.org/packages/f5/84/4af8c87c538afde10d99aeb071d50ad901db3a3607c514f8a9f182fcfe86/clearbox-preprocessor-0.11.14.tar.gz",
"platform": null,
"description": "[](https://clearbox-preprocessor.readthedocs.io/en/latest/?badge=latest)\n[](https://badge.fury.io/py/clearbox-preprocessor)\n[](https://pepy.tech/project/clearbox-preprocessor)\n[](https://github.com/Clearbox-AI/preprocessor)\n\n# Clearbox AI Preprocessor\n\nThis repository contains the continuation of the work presented in our series of blogposts *\"The whys and hows of data preparation\"* ([part 1](https://www.clearbox.ai/blog/2022-01-25-the-whys-and-hows-of-data-preparation), [part 2](https://www.clearbox.ai/blog/2022-02-22-the-whys-and-hows-of-data-preparation-part-2), [part 3](https://www.clearbox.ai/blog/2022-07-05-the-whys-and-hows-of-data-preparation-part-3)). \n\nThe new version of Preprocessor exploits Polars library's features to achieve blazing fast tabular data manipulation.\n\nIt is possible to input the Preprocessor a `Pandas.DataFrame`, a `Polars.DataFrame` or a `Polars.LazyFrame`.\n\n## Installation\n\nYou can install the preprocessor by running the following command:\n\n```shell\n$ pip install clearbox-preprocessor\n```\n\n## Preprocessing customization\nA bunch of options are available to customize the preprocessing. \n\nThe `Preprocessor` class features the following input arguments, besides the input dataset:\n - `discarding_threshold: float (default = 0.9)`\n \n Float number between 0 and 1 to set the threshold for discarding categorical features. \n If more than discarding_threshold * 100 % of values in a categorical feature are different from each other, then the column is discarded. \n For example, if discarding_threshold=0.9, a column will be discarded if more than 90% of its values are unique.\n \n - `get_discarded_info: bool (defatult = False)`\n \n When set to 'True', the preprocessor will feature the methods preprocessor.get_discarded_features_reason, which provides information on which columns were discarded and the reason why, and preprocessor.get_single_valued_columns, which provides the values of the single-valued discarded columns.\n Note that setting get_discarded_info=True will considerably slow down the processing operation!\n The list of discarded columns will be available even if get_discarded_info=False, so consider setting this flag to True only if you need to know why a column was discarded or, if it contained just one value, what that value was.\n \n - `excluded_col: (default = [])`\n \n List containing the names of the columns to be excluded from processing. These columns will be returned in the final dataframe withouth being manipulated. \n - `time: (default = None)`\n \n String name of the time column by which to sort the dataframe in case of time series.\n \n - `scaling_method: (default=\"none\")`\n \n Specifies the scaling operation to perform on numerical features.\n - \"none\" : no scaling is applied\n - \"normalize\" : applies normalization to numerical features\n - \"standardize\" : applies standardization to numerical features\n - \"quantile\" : Transforms numerical features using quantiles information\n \n - `num_fill_null: (default = \"mean\")`\n \n Specifies the value to fill null values with or the strategy for filling null values in numerical features.\n - value : fills null values with the specified value \n - \"mean\" : fills null values with the average of the column\n - \"forward\" : fills null values with the previous non-null value in the column\n - \"backward\" : fills null values with the following non-null value in the column\n - \"min\" : fills null values with the minimum value of the column\n - \"max\" : fills null values with the maximum value of the column\n - \"zero\" : fills null values with zeros\n - \"one\" : fills null values with ones\n\n - `n_bins: (default = 0)`\n \n Integer number that determines the number of bins into which numerical features are discretized. When set to 0, the preprocessing step defaults to the scaling method specified in the 'scaling' atgument instead of discretization.\n \n Note that if n_bins is different than 0, discretization will take place instead of scaling, regardless of whether the 'scaling' argument is specified.\n\n### Timeseries\nThe Prperocessor also features a timeseries manipulation and feature extraction method called `extract_ts_features()`.\n\nThis method takes as input:\n- the preprocessed dataframe\n- the target vector in the form of a `Pandas.Series` or a `Polars.Series`\n- the name of the time column\n- the name of the id column to group by\n \nIt returns the most relevant features selected among a wide range of features.\n\n## Usage\nYou can start using the Preprocessor by importing it and creating a `Pandas.DataFrame` or a `Polars.LazyFrame`:\n\n```python\nimport polars as pl\nfrom clearbox_preprocessor import Preprocessor\n\nq = pl.LazyFrame(\n {\n \"cha\": [\"x\", None, \"z\", \"w\", \"x\", \"k\"],\n \"int\": [123, 124, 223, 431, 435, 432],\n \"dat\": [\"2023-1-5T00:34:12.000Z\", \"2023-2-3T04:31:45.000Z\", \"2023-2-4T04:31:45.000Z\", None, \"2023-5-12T21:41:58.000Z\", \"2023-6-1T17:52:22.000Z\"],\n \"boo\": [True, False, None, True, False, False],\n \"equ\": [\"a\", \"a\", \"a\", \"a\", None, \"a\"],\n \"flo\": [43.034, 343.1, 224.23, 75.3, None, 83.2],\n \"str\": [\"asd\", \"fgh\", \"fgh\", \"\", None, \"cvb\"]\n }\n).with_columns(pl.col('dat').str.to_datetime(\"%Y-%m-%dT%H:%M:%S.000Z\"))\n\nq.collect()\n```\n<img src=\"https://github.com/Clearbox-AI/preprocessor/assets/152599516/c2e1878d-6af3-4157-8a7d-61fdccfde270\" alt=\"image\" width=\"40%\" height=\"auto\">\n\nAt this point, you can initialize the Preprocessor by passing the LazyFrame or DataFrame created to it and then calling the `transform()` method to materialize the processed dataframe.\n\nNote that if no argument is specified beyond the dataframe *q*, the default settings are employed for preprocessing:\n\n```python\npreprocessor = Preprocessor(q)\ndf = preprocessor.transform(q)\ndf\n```\n<img src=\"https://github.com/Clearbox-AI/preprocessor/assets/152599516/7cd5b6f6-26f9-4af9-8250-751f43cac7d5\" alt=\"image\" width=\"70%\" height=\"auto\">\n\n\n### Customization example\nIn the following example, when the Preprocessor is initialized:\n1. The discarding threshold is lowered from 90% to 80% (a column will be discarded if more than 80% of its values are unique).\n2. The discarding featrues informations are stored in the `preprocessor` instance.\n3. The column \"boo\" is excluded from the preprocessing and is preserved unchanged.\n4. The scaling method of the numerical features chosen is standardization\n5. The fill null strategy for numerical features is \"forward\".\n\n```python\npreprocessor = Preprocessor(q, \n get_discarded_info=True, \n discarding_threshold = 0.8, \n excluded_col = [\"boo\"], \n scaling = \"standardize\", \n num_fill_null = \"forward\"\n )\ndf = preprocessor.transform(q)\ndf\n```\n<img src=\"https://github.com/Clearbox-AI/preprocessor/assets/152599516/ba61531d-a462-4d58-847a-47127d6050fd\" alt=\"image\" width=\"52%\" height=\"auto\">\n\nIt is possible to inverse transform the processed dataframe with the method `preprocessor.inverse_transform()`.\n\n```python\npreprocessor = Preprocessor(q)\ndf = preprocessor.transform(q)\ninverse_df = preprocessor.inverse_transform(df)\n```\n\nIf the Processor's argument `get_discarded_info` is set to `True` during initialization, it is possible to call the method `get_discarded_features_reason()` to display the discarded features. \nIn the case of discarded single-valued columns, the value contained is also displayed and is available in a dictionary called `single_value_columns`, stored in the Preprocessor instance, and can be used as metadata.\n\n```python\npreprocessor.get_discarded_features_reason()\n```\n<img src=\"https://github.com/Clearbox-AI/preprocessor/assets/152599516/5c5d7a42-d52d-4d32-862d-fc3c95c31d67\" alt=\"image\" width=\"90%\" height=\"auto\">\n\n\n## To do\n- [ ] Implement unit tests\n",
"bugtrack_url": null,
"license": null,
"summary": "A fast polars based data pre-processor for ML datasets",
"version": "0.11.14",
"project_urls": {
"Homepage": "https://github.com/Clearbox-AI/preprocessor"
},
"split_keywords": [],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "a56085d196e793a7c4f02ddfa81e3c67ea9c287a7a51a18e864071c353057f32",
"md5": "fe49df395dc3c700deefd13c65598a21",
"sha256": "0d10f88c29493c6b060c3660373443539fbe0b2546acb38e735f327e76d2100d"
},
"downloads": -1,
"filename": "clearbox_preprocessor-0.11.14-py3-none-any.whl",
"has_sig": false,
"md5_digest": "fe49df395dc3c700deefd13c65598a21",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">3.9",
"size": 20964,
"upload_time": "2025-03-21T11:41:53",
"upload_time_iso_8601": "2025-03-21T11:41:53.726950Z",
"url": "https://files.pythonhosted.org/packages/a5/60/85d196e793a7c4f02ddfa81e3c67ea9c287a7a51a18e864071c353057f32/clearbox_preprocessor-0.11.14-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "f5844af8c87c538afde10d99aeb071d50ad901db3a3607c514f8a9f182fcfe86",
"md5": "c8b2648cc86026f499538492087303ed",
"sha256": "2046ac27d15a78d7bda8889dccac4c35aa30174c5e8b981a62611e3aefa72c61"
},
"downloads": -1,
"filename": "clearbox-preprocessor-0.11.14.tar.gz",
"has_sig": false,
"md5_digest": "c8b2648cc86026f499538492087303ed",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">3.9",
"size": 21599,
"upload_time": "2025-03-21T11:41:55",
"upload_time_iso_8601": "2025-03-21T11:41:55.464474Z",
"url": "https://files.pythonhosted.org/packages/f5/84/4af8c87c538afde10d99aeb071d50ad901db3a3607c514f8a9f182fcfe86/clearbox-preprocessor-0.11.14.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-03-21 11:41:55",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Clearbox-AI",
"github_project": "preprocessor",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "numpy",
"specs": []
},
{
"name": "pandas",
"specs": []
},
{
"name": "scikit-learn",
"specs": []
},
{
"name": "scipy",
"specs": [
[
"<=",
"1.14"
]
]
},
{
"name": "polars",
"specs": [
[
"==",
"1.17.1"
]
]
},
{
"name": "tsfresh",
"specs": [
[
"==",
"0.20.3"
]
]
},
{
"name": "sphinx",
"specs": []
},
{
"name": "myst_parser",
"specs": []
},
{
"name": "sphinxemoji",
"specs": []
},
{
"name": "numpydoc",
"specs": []
},
{
"name": "sphinx_rtd_theme",
"specs": []
}
],
"lcname": "clearbox-preprocessor"
}
JSON
