<div align="center">
<br/>
<p align="center">
<i>This repository is part of <a href="https://sdv.dev">The Synthetic Data Vault Project</a>, a project from <a href="https://datacebo.com">DataCebo</a>.</i>
</p>
[](https://pypi.org/search/?q=&o=&c=Development+Status+%3A%3A+5+-+Production%2FStable)
[](https://pypi.python.org/pypi/RDT)
[](https://github.com/sdv-dev/RDT/actions/workflows/unit.yml)
[](https://pepy.tech/project/rdt)
[](https://codecov.io/gh/sdv-dev/RDT)
[](https://bit.ly/sdv-slack-invite)
<div align="left">
<br/>
<p align="center">
<a href="https://github.com/sdv-dev/RDT">
<img align="center" width=40% src="https://github.com/sdv-dev/SDV/blob/stable/docs/images/RDT-DataCebo.png"></img>
</a>
</p>
</div>
</div>
# Overview
RDT (Reversible Data Transforms) is a Python library that transforms raw data into fully numerical
data, ready for data science. The transforms are reversible, allowing you to convert from numerical
data back into your original format.
<img align="center" src="https://github.com/sdv-dev/SDV/blob/stable/docs/images/rdt_main_tranformation.png"></img>
# Install
Install **RDT** using ``pip`` or ``conda``. We recommend using a virtual environment to avoid
conflicts with other software on your device.
```bash
pip install rdt
```
```bash
conda install -c conda-forge rdt
```
For more information about using reversible data transformations, visit the [RDT Documentation](https://docs.sdv.dev/rdt).
# Quickstart
In this short series of tutorials we will guide you through a series of steps that will
help you getting started using **RDT** to transform columns, tables and datasets.
## Load the demo data
After you have installed RDT, you can get started using the demo dataset.
```python3
from rdt import get_demo
customers = get_demo()
```
This dataset contains some randomly generated values that describe the customers of an online
marketplace.
```
last_login email_optin credit_card age dollars_spent
0 2021-06-26 False VISA 29 99.99
1 2021-02-10 False VISA 18 NaN
2 NaT False AMEX 21 2.50
3 2020-09-26 True NaN 45 25.00
4 2020-12-22 NaN DISCOVER 32 19.99
```
Let's transform this data so that each column is converted to full, numerical data ready for data
science.
## Creating the HyperTransformer & config
The ``HyperTransformer`` is capable of transforming multi-column datasets.
```python3
from rdt import HyperTransformer
ht = HyperTransformer()
```
The `HyperTransformer` needs to know about the columns in your dataset and which transformers to
apply to each. These are described by a config. We can ask the `HyperTransformer` to automatically
detect it based on the data we plan to use.
```python3
ht.detect_initial_config(data=customers)
```
This will create and set the config.
```
Config:
{
"sdtypes": {
"last_login": "datetime",
"email_optin": "boolean",
"credit_card": "categorical",
"age": "numerical",
"dollars_spent": "numerical"
},
"transformers": {
"last_login": "UnixTimestampEncoder()",
"email_optin": "BinaryEncoder()",
"credit_card": "FrequencyEncoder()",
"age": "FloatFormatter()",
"dollars_spent": "FloatFormatter()"
}
}
```
The `sdtypes` dictionary describes the semantic data types of each of your columns and the
`transformers` dictionary describes which transformer to use for each column. You can customize the
transformers and their settings. (See the [Transformers Glossary](https://docs.sdv.dev/rdt/transformers-glossary/browse-transformers) for more information).
## Fitting & using the HyperTransformer
The `HyperTransformer` references the config while learning the data during the `fit` stage.
```python3
ht.fit(customers)
```
Once the transformer is fit, it's ready to use. Use the transform method to transform all columns
of your dataset at once.
```python3
transformed_data = ht.transform(customers)
```
```
last_login.value email_optin.value credit_card.value age.value dollars_spent.value
0 1.624666e+18 0.0 0.2 29 99.99
1 1.612915e+18 0.0 0.2 18 36.87
2 1.611814e+18 0.0 0.5 21 2.50
3 1.601078e+18 1.0 0.7 45 25.00
4 1.608595e+18 0.0 0.9 32 19.99
```
The ``HyperTransformer`` applied the assigned transformer to each individual column. Each column
now contains fully numerical data that you can use for your project!
When you're done with your project, you can also transform the data back to the original format
using the `reverse_transform` method.
```python3
original_format_data = ht.reverse_transform(transformed_data)
```
```
last_login email_optin credit_card age dollars_spent
0 NaT False VISA 29 99.99
1 2021-02-10 False VISA 18 NaN
2 NaT False AMEX 21 NaN
3 2020-09-26 True NaN 45 25.00
4 2020-12-22 False DISCOVER 32 19.99
```
# What's Next?
To learn more about reversible data transformations, visit the [RDT Documentation](https://docs.sdv.dev/rdt).
---
<div align="center">
<a href="https://datacebo.com"><img align="center" width=40% src="https://github.com/sdv-dev/SDV/blob/stable/docs/images/DataCebo.png"></img></a>
</div>
<br/>
<br/>
[The Synthetic Data Vault Project](https://sdv.dev) was first created at MIT's [Data to AI Lab](
https://dai.lids.mit.edu/) in 2016. After 4 years of research and traction with enterprise, we
created [DataCebo](https://datacebo.com) in 2020 with the goal of growing the project.
Today, DataCebo is the proud developer of SDV, the largest ecosystem for
synthetic data generation & evaluation. It is home to multiple libraries that support synthetic
data, including:
* 🔄 Data discovery & transformation. Reverse the transforms to reproduce realistic data.
* 🧠Multiple machine learning models -- ranging from Copulas to Deep Learning -- to create tabular,
multi table and time series data.
* 📊 Measuring quality and privacy of synthetic data, and comparing different synthetic data
generation models.
[Get started using the SDV package](https://sdv.dev/SDV/getting_started/install.html) -- a fully
integrated solution and your one-stop shop for synthetic data. Or, use the standalone libraries
for specific needs.
Raw data
{
"_id": null,
"home_page": "https://github.com/sdv-dev/RDT",
"name": "rdt-identity",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8,<3.12",
"maintainer_email": "",
"keywords": "rdt,rdt_identity",
"author": "DataCebo, Inc.",
"author_email": "info@sdv.dev",
"download_url": "https://files.pythonhosted.org/packages/1a/a4/b83036a3bed1c206c5208b4ac86151c0a8b51b1b953fdb4c8f5ff1692ba0/rdt_identity-1.9.1.tar.gz",
"platform": null,
"description": "<div align=\"center\">\n<br/>\n<p align=\"center\">\n <i>This repository is part of <a href=\"https://sdv.dev\">The Synthetic Data Vault Project</a>, a project from <a href=\"https://datacebo.com\">DataCebo</a>.</i>\n</p>\n\n[](https://pypi.org/search/?q=&o=&c=Development+Status+%3A%3A+5+-+Production%2FStable)\n[](https://pypi.python.org/pypi/RDT)\n[](https://github.com/sdv-dev/RDT/actions/workflows/unit.yml)\n[](https://pepy.tech/project/rdt)\n[](https://codecov.io/gh/sdv-dev/RDT)\n[](https://bit.ly/sdv-slack-invite)\n\n<div align=\"left\">\n<br/>\n<p align=\"center\">\n<a href=\"https://github.com/sdv-dev/RDT\">\n<img align=\"center\" width=40% src=\"https://github.com/sdv-dev/SDV/blob/stable/docs/images/RDT-DataCebo.png\"></img>\n</a>\n</p>\n</div>\n\n</div>\n\n# Overview\n\nRDT (Reversible Data Transforms) is a Python library that transforms raw data into fully numerical\ndata, ready for data science. The transforms are reversible, allowing you to convert from numerical\ndata back into your original format.\n\n<img align=\"center\" src=\"https://github.com/sdv-dev/SDV/blob/stable/docs/images/rdt_main_tranformation.png\"></img>\n\n\n# Install\n\nInstall **RDT** using ``pip`` or ``conda``. We recommend using a virtual environment to avoid\nconflicts with other software on your device.\n\n```bash\npip install rdt\n```\n\n```bash\nconda install -c conda-forge rdt\n```\n\nFor more information about using reversible data transformations, visit the [RDT Documentation](https://docs.sdv.dev/rdt).\n\n\n# Quickstart\n\nIn this short series of tutorials we will guide you through a series of steps that will\nhelp you getting started using **RDT** to transform columns, tables and datasets.\n\n## Load the demo data\n\nAfter you have installed RDT, you can get started using the demo dataset.\n\n```python3\nfrom rdt import get_demo\n\ncustomers = get_demo()\n```\n\nThis dataset contains some randomly generated values that describe the customers of an online\nmarketplace.\n\n```\n last_login email_optin credit_card age dollars_spent\n0 2021-06-26 False VISA 29 99.99\n1 2021-02-10 False VISA 18 NaN\n2 NaT False AMEX 21 2.50\n3 2020-09-26 True NaN 45 25.00\n4 2020-12-22 NaN DISCOVER 32 19.99\n```\n\nLet's transform this data so that each column is converted to full, numerical data ready for data\nscience.\n\n## Creating the HyperTransformer & config\n\nThe ``HyperTransformer`` is capable of transforming multi-column datasets.\n\n```python3\nfrom rdt import HyperTransformer\n\nht = HyperTransformer()\n```\n\nThe `HyperTransformer` needs to know about the columns in your dataset and which transformers to\napply to each. These are described by a config. We can ask the `HyperTransformer` to automatically\ndetect it based on the data we plan to use.\n\n```python3\nht.detect_initial_config(data=customers)\n```\n\nThis will create and set the config.\n\n```\nConfig:\n{\n \"sdtypes\": {\n \"last_login\": \"datetime\",\n \"email_optin\": \"boolean\",\n \"credit_card\": \"categorical\",\n \"age\": \"numerical\",\n \"dollars_spent\": \"numerical\"\n },\n \"transformers\": {\n \"last_login\": \"UnixTimestampEncoder()\",\n \"email_optin\": \"BinaryEncoder()\",\n \"credit_card\": \"FrequencyEncoder()\",\n \"age\": \"FloatFormatter()\",\n \"dollars_spent\": \"FloatFormatter()\"\n }\n}\n```\n\nThe `sdtypes` dictionary describes the semantic data types of each of your columns and the\n`transformers` dictionary describes which transformer to use for each column. You can customize the\ntransformers and their settings. (See the [Transformers Glossary](https://docs.sdv.dev/rdt/transformers-glossary/browse-transformers) for more information).\n\n## Fitting & using the HyperTransformer\n\nThe `HyperTransformer` references the config while learning the data during the `fit` stage.\n\n```python3\nht.fit(customers)\n```\n\nOnce the transformer is fit, it's ready to use. Use the transform method to transform all columns\nof your dataset at once.\n\n```python3\ntransformed_data = ht.transform(customers)\n```\n\n```\n last_login.value email_optin.value credit_card.value age.value dollars_spent.value\n0 1.624666e+18 0.0 0.2 29 99.99\n1 1.612915e+18 0.0 0.2 18 36.87\n2 1.611814e+18 0.0 0.5 21 2.50\n3 1.601078e+18 1.0 0.7 45 25.00\n4 1.608595e+18 0.0 0.9 32 19.99\n```\n\nThe ``HyperTransformer`` applied the assigned transformer to each individual column. Each column\nnow contains fully numerical data that you can use for your project!\n\nWhen you're done with your project, you can also transform the data back to the original format\nusing the `reverse_transform` method.\n\n```python3\noriginal_format_data = ht.reverse_transform(transformed_data)\n```\n\n```\n last_login email_optin credit_card age dollars_spent\n0 NaT False VISA 29 99.99\n1 2021-02-10 False VISA 18 NaN\n2 NaT False AMEX 21 NaN\n3 2020-09-26 True NaN 45 25.00\n4 2020-12-22 False DISCOVER 32 19.99\n```\n\n# What's Next?\n\nTo learn more about reversible data transformations, visit the [RDT Documentation](https://docs.sdv.dev/rdt).\n\n\n---\n\n\n<div align=\"center\">\n<a href=\"https://datacebo.com\"><img align=\"center\" width=40% src=\"https://github.com/sdv-dev/SDV/blob/stable/docs/images/DataCebo.png\"></img></a>\n</div>\n<br/>\n<br/>\n\n[The Synthetic Data Vault Project](https://sdv.dev) was first created at MIT's [Data to AI Lab](\nhttps://dai.lids.mit.edu/) in 2016. After 4 years of research and traction with enterprise, we\ncreated [DataCebo](https://datacebo.com) in 2020 with the goal of growing the project.\nToday, DataCebo is the proud developer of SDV, the largest ecosystem for\nsynthetic data generation & evaluation. It is home to multiple libraries that support synthetic\ndata, including:\n\n* \ud83d\udd04 Data discovery & transformation. Reverse the transforms to reproduce realistic data.\n* \ud83e\udde0 Multiple machine learning models -- ranging from Copulas to Deep Learning -- to create tabular,\n multi table and time series data.\n* \ud83d\udcca Measuring quality and privacy of synthetic data, and comparing different synthetic data\n generation models.\n\n[Get started using the SDV package](https://sdv.dev/SDV/getting_started/install.html) -- a fully\nintegrated solution and your one-stop shop for synthetic data. Or, use the standalone libraries\nfor specific needs.\n",
"bugtrack_url": null,
"license": "BSL-1.1",
"summary": "Reversible Data Transforms",
"version": "1.9.1",
"project_urls": {
"Homepage": "https://github.com/sdv-dev/RDT"
},
"split_keywords": [
"rdt",
"rdt_identity"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "311565402c894f38c9928df0243cc205b6d06101a2ff02c08ed5b2996f9640fa",
"md5": "2e2496aae54638c5951e7d271b42f078",
"sha256": "a0e137b8c7c4269c0089613af1fcdd258da44de5c85d6dc438f451746034c696"
},
"downloads": -1,
"filename": "rdt_identity-1.9.1-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "2e2496aae54638c5951e7d271b42f078",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": ">=3.8,<3.12",
"size": 11912,
"upload_time": "2024-01-10T22:44:11",
"upload_time_iso_8601": "2024-01-10T22:44:11.280943Z",
"url": "https://files.pythonhosted.org/packages/31/15/65402c894f38c9928df0243cc205b6d06101a2ff02c08ed5b2996f9640fa/rdt_identity-1.9.1-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1aa4b83036a3bed1c206c5208b4ac86151c0a8b51b1b953fdb4c8f5ff1692ba0",
"md5": "5f9d2d8cd004f2e770359eb1082047ea",
"sha256": "9e33b2f72ce8e213fe339f4a259de959c020d76b3da61dbfbad447231c94ef31"
},
"downloads": -1,
"filename": "rdt_identity-1.9.1.tar.gz",
"has_sig": false,
"md5_digest": "5f9d2d8cd004f2e770359eb1082047ea",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8,<3.12",
"size": 117138,
"upload_time": "2024-01-10T22:44:15",
"upload_time_iso_8601": "2024-01-10T22:44:15.361338Z",
"url": "https://files.pythonhosted.org/packages/1a/a4/b83036a3bed1c206c5208b4ac86151c0a8b51b1b953fdb4c8f5ff1692ba0/rdt_identity-1.9.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-10 22:44:15",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "sdv-dev",
"github_project": "RDT",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"tox": true,
"lcname": "rdt-identity"
}
JSON
