| Name | metasyn JSON |
| Version |
1.1.0
JSON |
| download |
| home_page | None |
| Summary | Package for creating synthetic datasets while preserving privacy. |
| upload_time | 2024-11-25 12:28:37 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.8 |
| license | MIT License Copyright (c) 2024 SoDa Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
metadata
open-data
privacy
synthetic-data
tabular datasets
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<p align="center">
<img src="https://github.com/sodascience/metasyn/blob/main/docs/source/images/logos/blue.svg" width="600px" alt="Metasyn logo"></img>
<h3 align="center">Transparent and privacy-friendly synthetic data generation</h3>
<p align="center">
<span>
<a href="https://www.repostatus.org/#active"><img src="https://www.repostatus.org/badges/latest/active.svg" alt="Project Status: Active – The project has reached a stable, usable state and is being actively developed." /></a>
<a href="https://pypi.org/project/metasyn"><img src="https://img.shields.io/pypi/pyversions/metasyn" alt="metasyn on pypi"></img></a>
<a href="https://colab.research.google.com/github/sodascience/metasyn/blob/main/examples/getting_started.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="open getting started on colab"></img></a>
<a href="https://metasyn.readthedocs.io/en/latest/index.html"><img src="https://readthedocs.org/projects/metasyn/badge/?version=latest" alt="Readthedocs"></img></a>
<a href="https://hub.docker.com/r/sodateam/metasyn"><img src="https://img.shields.io/docker/v/sodateam/metasyn?logo=docker&label=docker&color=blue" alt="Docker image version"></img></a>
<a href="https://zenodo.org/doi/10.5281/zenodo.7696031"><img src="https://zenodo.org/badge/DOI/10.5281/zenodo.7696031.svg" alt="DOI"></a>
<a href="https://joss.theoj.org/papers/43fd4234e18bfd94b952aea35db8b883"><img src="https://joss.theoj.org/papers/43fd4234e18bfd94b952aea35db8b883/status.svg"></a>
</span>
</p>
</p>
<br/>
__Generate synthetic tabular data__ in a transparent, understandable, and privacy-friendly way. Metasyn makes it possible for owners of sensitive data to create test data, do open science, improve code reproducibility, encourage data reuse, and enhance accessibility of their datasets, without worrying about leaking private information.
With metasyn you can __fit__ a model to an existing dataframe, __save__ it to a transparent and auditable `.json` file, and __synthesize__ a dataframe that looks a lot like the real one. In contrast to most other synthetic data software, we make the explicit choice to strictly limit the statistical information in our model in order to adhere to the highest privacy standards.
## Highlights
- 👋 __Accessible__. Metasyn is designed to be easy to use and understand, and we do our best to be welcoming to newcomers and novice users. [Let us know](https://github.com/sodascience/metasyn/issues/new) if we can improve!
- ✨ __Fully featured__. Out of the box, metasyn natively handles a wide range of data types, missing values, categorical data, key columns with unique values, and structured strings such as postcodes or identifiers.
- 🔎 __Transparent__. With metasyn you share not only synthetic data, but also the model and settings used to create it through a traceable, auditable metadata format. Everyone can read and understand what the model does; it is crystal clear which information becomes public.
- 🔐 __Private__. By default, metasyn does not incorporate multivariate information, meaning less risk of privacy issues such as identity, attribute, or group disclosure. On top of this, we support privacy plugins such as our own [disclosure control plugin](https://github.com/sodascience/metasyn-disclosure-control) to further enhance privacy in critically sensitive situations.
- 🔗 __Integrated__. We integrate closely with popular, modern tools in the python ecosystem, building on the wonderful [polars](https://pola.rs/) dataframe library ([pandas](https://pandas.pydata.org/) is supported too), as well as [faker](https://faker.readthedocs.io/en/master/) to generate localized data for names, emails, and phone numbers, and more.
- 📦 __Extensible__. Are you missing features? Do you have a different definition of privacy? Our plugin system allows you (or your organisation) to create their own extension to adjust metasyn to what you need. Or you can [contribute](#contributing) directly to the project.
## Installation
Metasyn can be installed directly from PyPI using the following command in the terminal:
```sh
pip install metasyn
```
The latest (possibly unstable) development version can be installed directly from GitHub like so:
```sh
pip install git+https://github.com/sodascience/metasyn
```
## Usage
To generate synthetic data, `metasyn` first needs to fit a `MetaFrame` to the data which can then be used to produce new synthetic rows:
The above image closely matches the Python code:
```python
import polars as pl
from metasyn import MetaFrame, demo_file
# Get the csv file path for built-in demo dataset
csv_path = demo_file("fruit")
# Create a polars dataframe from the csv file.
# It is important to ensure the data types are correct
# when creating your dataframe, especially categorical data!
df = pl.read_csv(csv_path, schema_overrides={
"fruits": pl.Categorical,
"cars": pl.Categorical
})
# Create a MetaFrame from the DataFrame.
mf = MetaFrame.fit_dataframe(df)
# Generate a new DataFrame with 5 rows from the MetaFrame.
df_synth = mf.synthesize(5)
# This DataFrame can be exported to csv, parquet, excel and more.
df_synth.write_csv("output.csv")
```
To explore more options and try this out online, take a look at our interactive tutorial:
[](https://colab.research.google.com/github/sodascience/metasyn/blob/main/examples/getting_started.ipynb)
For more information on how to create dataframes with polars, refer to the [Polars documentation](https://pola.rs/). Metasyn also works with pandas dataframes!
## Where to go next
- To explore more options and try this out online, take a look at our interactive tutorial: [](https://colab.research.google.com/github/sodascience/metasyn/blob/main/examples/getting_started.ipynb)
- As a next step to learn more about generating synthetic data with metasyn we recommend to check out the [user guide](https://metasyn.readthedocs.io/en/latest/usage/usage.html) and other [documentation](https://metasyn.readthedocs.io/en/latest).
- For even more privacy, have a look at our [disclosure control plugin](https://github.com/sodascience/metasyn-disclosure-control).
- Want to create programs that build on metasyn? Take a look at our versioned [Docker containers](https://hub.docker.com/r/sodateam/metasyn) and our [CLI](https://metasyn.readthedocs.io/en/latest/usage/cli.html).
## Contributing
Metasyn is an open-source project, and we welcome contributions from the community, from bug reports & feature requests to code contributions. Read our [contributing guidelines](.github/CONTRIBUTING.md) for more information and to get started!
## Contact
**Metasyn** is a project by the [ODISSEI Social Data Science (SoDa)](https://odissei-data.nl/nl/soda/) team.
Do you have questions, suggestions, or remarks on the technical implementation? Create an issue in the [issue tracker](https://github.com/sodascience/metasyn/issues) or feel free to contact [Erik-Jan van Kesteren](https://github.com/vankesteren) or [Raoul Schram](https://github.com/qubixes).
<img src="docs/source/images/logos/soda.png" alt="SoDa logo" width="250px"/>
Raw data
{
"_id": null,
"home_page": null,
"name": "metasyn",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "metadata, open-data, privacy, synthetic-data, tabular datasets",
"author": null,
"author_email": "Raoul Schram <r.d.schram@uu.nl>, Erik-Jan van Kesteren <e.vankesteren1@uu.nl>",
"download_url": "https://files.pythonhosted.org/packages/62/79/5cd3287fedac1d02fbd0415cb97c160442d3aa594796887d764f85c1a3bd/metasyn-1.1.0.tar.gz",
"platform": null,
"description": "<p align=\"center\">\n <img src=\"https://github.com/sodascience/metasyn/blob/main/docs/source/images/logos/blue.svg\" width=\"600px\" alt=\"Metasyn logo\"></img>\n <h3 align=\"center\">Transparent and privacy-friendly synthetic data generation</h3>\n <p align=\"center\">\n <span>\n <a href=\"https://www.repostatus.org/#active\"><img src=\"https://www.repostatus.org/badges/latest/active.svg\" alt=\"Project Status: Active \u2013 The project has reached a stable, usable state and is being actively developed.\" /></a>\n <a href=\"https://pypi.org/project/metasyn\"><img src=\"https://img.shields.io/pypi/pyversions/metasyn\" alt=\"metasyn on pypi\"></img></a>\n <a href=\"https://colab.research.google.com/github/sodascience/metasyn/blob/main/examples/getting_started.ipynb\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"open getting started on colab\"></img></a>\n <a href=\"https://metasyn.readthedocs.io/en/latest/index.html\"><img src=\"https://readthedocs.org/projects/metasyn/badge/?version=latest\" alt=\"Readthedocs\"></img></a>\n <a href=\"https://hub.docker.com/r/sodateam/metasyn\"><img src=\"https://img.shields.io/docker/v/sodateam/metasyn?logo=docker&label=docker&color=blue\" alt=\"Docker image version\"></img></a>\n <a href=\"https://zenodo.org/doi/10.5281/zenodo.7696031\"><img src=\"https://zenodo.org/badge/DOI/10.5281/zenodo.7696031.svg\" alt=\"DOI\"></a>\n <a href=\"https://joss.theoj.org/papers/43fd4234e18bfd94b952aea35db8b883\"><img src=\"https://joss.theoj.org/papers/43fd4234e18bfd94b952aea35db8b883/status.svg\"></a>\n </span>\n </p>\n</p>\n<br/>\n\n__Generate synthetic tabular data__ in a transparent, understandable, and privacy-friendly way. Metasyn makes it possible for owners of sensitive data to create test data, do open science, improve code reproducibility, encourage data reuse, and enhance accessibility of their datasets, without worrying about leaking private information. \n\nWith metasyn you can __fit__ a model to an existing dataframe, __save__ it to a transparent and auditable `.json` file, and __synthesize__ a dataframe that looks a lot like the real one. In contrast to most other synthetic data software, we make the explicit choice to strictly limit the statistical information in our model in order to adhere to the highest privacy standards.\n\n## Highlights\n- \ud83d\udc4b __Accessible__. Metasyn is designed to be easy to use and understand, and we do our best to be welcoming to newcomers and novice users. [Let us know](https://github.com/sodascience/metasyn/issues/new) if we can improve!\n- \u2728 __Fully featured__. Out of the box, metasyn natively handles a wide range of data types, missing values, categorical data, key columns with unique values, and structured strings such as postcodes or identifiers.\n- \ud83d\udd0e __Transparent__. With metasyn you share not only synthetic data, but also the model and settings used to create it through a traceable, auditable metadata format. Everyone can read and understand what the model does; it is crystal clear which information becomes public.\n- \ud83d\udd10 __Private__. By default, metasyn does not incorporate multivariate information, meaning less risk of privacy issues such as identity, attribute, or group disclosure. On top of this, we support privacy plugins such as our own [disclosure control plugin](https://github.com/sodascience/metasyn-disclosure-control) to further enhance privacy in critically sensitive situations.\n- \ud83d\udd17 __Integrated__. We integrate closely with popular, modern tools in the python ecosystem, building on the wonderful [polars](https://pola.rs/) dataframe library ([pandas](https://pandas.pydata.org/) is supported too), as well as [faker](https://faker.readthedocs.io/en/master/) to generate localized data for names, emails, and phone numbers, and more.\n- \ud83d\udce6 __Extensible__. Are you missing features? Do you have a different definition of privacy? Our plugin system allows you (or your organisation) to create their own extension to adjust metasyn to what you need. Or you can [contribute](#contributing) directly to the project.\n\n## Installation\nMetasyn can be installed directly from PyPI using the following command in the terminal:\n\n```sh\npip install metasyn\n```\n\nThe latest (possibly unstable) development version can be installed directly from GitHub like so:\n\n```sh\npip install git+https://github.com/sodascience/metasyn\n```\n\n## Usage\n\n\n\nTo generate synthetic data, `metasyn` first needs to fit a `MetaFrame` to the data which can then be used to produce new synthetic rows:\n\n\n\nThe above image closely matches the Python code:\n\n```python\nimport polars as pl\nfrom metasyn import MetaFrame, demo_file\n\n# Get the csv file path for built-in demo dataset\ncsv_path = demo_file(\"fruit\")\n\n# Create a polars dataframe from the csv file.\n# It is important to ensure the data types are correct \n# when creating your dataframe, especially categorical data!\ndf = pl.read_csv(csv_path, schema_overrides={\n \"fruits\": pl.Categorical, \n \"cars\": pl.Categorical\n})\n\n# Create a MetaFrame from the DataFrame.\nmf = MetaFrame.fit_dataframe(df)\n\n# Generate a new DataFrame with 5 rows from the MetaFrame.\ndf_synth = mf.synthesize(5)\n\n# This DataFrame can be exported to csv, parquet, excel and more.\ndf_synth.write_csv(\"output.csv\")\n```\n\nTo explore more options and try this out online, take a look at our interactive tutorial:\n\n[](https://colab.research.google.com/github/sodascience/metasyn/blob/main/examples/getting_started.ipynb)\n\nFor more information on how to create dataframes with polars, refer to the [Polars documentation](https://pola.rs/). Metasyn also works with pandas dataframes!\n\n## Where to go next\n\n- To explore more options and try this out online, take a look at our interactive tutorial: [](https://colab.research.google.com/github/sodascience/metasyn/blob/main/examples/getting_started.ipynb)\n- As a next step to learn more about generating synthetic data with metasyn we recommend to check out the [user guide](https://metasyn.readthedocs.io/en/latest/usage/usage.html) and other [documentation](https://metasyn.readthedocs.io/en/latest).\n- For even more privacy, have a look at our [disclosure control plugin](https://github.com/sodascience/metasyn-disclosure-control).\n- Want to create programs that build on metasyn? Take a look at our versioned [Docker containers](https://hub.docker.com/r/sodateam/metasyn) and our [CLI](https://metasyn.readthedocs.io/en/latest/usage/cli.html).\n\n## Contributing\nMetasyn is an open-source project, and we welcome contributions from the community, from bug reports & feature requests to code contributions. Read our [contributing guidelines](.github/CONTRIBUTING.md) for more information and to get started!\n\n## Contact\n**Metasyn** is a project by the [ODISSEI Social Data Science (SoDa)](https://odissei-data.nl/nl/soda/) team.\nDo you have questions, suggestions, or remarks on the technical implementation? Create an issue in the [issue tracker](https://github.com/sodascience/metasyn/issues) or feel free to contact [Erik-Jan van Kesteren](https://github.com/vankesteren) or [Raoul Schram](https://github.com/qubixes).\n\n<img src=\"docs/source/images/logos/soda.png\" alt=\"SoDa logo\" width=\"250px\"/> \n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) 2024 SoDa Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
"summary": "Package for creating synthetic datasets while preserving privacy.",
"version": "1.1.0",
"project_urls": {
"GitHub": "https://github.com/sodascience/metasyn",
"documentation": "https://metasyn.readthedocs.io/en/latest/index.html"
},
"split_keywords": [
"metadata",
" open-data",
" privacy",
" synthetic-data",
" tabular datasets"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "662924e31d7a40c717b0452d441bc44a2ed72d44b73622c8400dc22661885ba2",
"md5": "0feb58797a8155aa1bcf73a7199fad49",
"sha256": "a2635f1d689a91a44ed90874d523f8032f8abe042259a10030d5bc1e95e16b5f"
},
"downloads": -1,
"filename": "metasyn-1.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0feb58797a8155aa1bcf73a7199fad49",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 341363,
"upload_time": "2024-11-25T12:28:35",
"upload_time_iso_8601": "2024-11-25T12:28:35.411670Z",
"url": "https://files.pythonhosted.org/packages/66/29/24e31d7a40c717b0452d441bc44a2ed72d44b73622c8400dc22661885ba2/metasyn-1.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "62795cd3287fedac1d02fbd0415cb97c160442d3aa594796887d764f85c1a3bd",
"md5": "b2837fa9a7546a98a9f60a8ef9d41295",
"sha256": "ba983e20afce9cbcf75eb118a4ac818173b5d6faf8e703af2af8621d992b7e6b"
},
"downloads": -1,
"filename": "metasyn-1.1.0.tar.gz",
"has_sig": false,
"md5_digest": "b2837fa9a7546a98a9f60a8ef9d41295",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 2923712,
"upload_time": "2024-11-25T12:28:37",
"upload_time_iso_8601": "2024-11-25T12:28:37.369581Z",
"url": "https://files.pythonhosted.org/packages/62/79/5cd3287fedac1d02fbd0415cb97c160442d3aa594796887d764f85c1a3bd/metasyn-1.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-25 12:28:37",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "sodascience",
"github_project": "metasyn",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "metasyn"
}
