dataops-testgen


Namedataops-testgen JSON
Version 2.15.2 PyPI version JSON
download
home_pageNone
SummaryDataKitchen's Data Quality DataOps TestGen
upload_time2024-10-12 00:43:28
maintainerNone
docs_urlNone
authorNone
requires_python>=3.10
licenseNone
keywords dataops data quality testing database profiling
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # DataOps Data Quality TestGen
![apache 2.0 license Badge](https://img.shields.io/badge/License%20-%20Apache%202.0%20-%20blue) ![PRs Badge](https://img.shields.io/badge/PRs%20-%20Welcome%20-%20green) [![Latest Version](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fhub.docker.com%2Fv2%2Frepositories%2Fdatakitchen%2Fdataops-testgen%2Ftags%2F&query=results%5B0%5D.name&label=latest%20version&color=06A04A)](https://hub.docker.com/r/datakitchen/dataops-testgen) [![Docker Pulls](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fhub.docker.com%2Fv2%2Frepositories%2Fdatakitchen%2Fdataops-testgen%2F&query=pull_count&style=flat&label=docker%20pulls&color=06A04A)](https://hub.docker.com/r/datakitchen/dataops-testgen) [![Documentation](https://img.shields.io/badge/docs-On%20datakitchen.io-06A04A?style=flat)](https://docs.datakitchen.io/articles/#!dataops-testgen-help/dataops-testgen-help) [![Static Badge](https://img.shields.io/badge/Slack-Join%20Discussion-blue?style=flat&logo=slack)](https://data-observability-slack.datakitchen.io/join)

*<p style="text-align: center;">DataOps Data Quality TestGen, or "TestGen" for short, can help you find data issues so you can alert your users and notify your suppliers. It does this by delivering simple, fast data quality test generation and execution by data profiling, new dataset screening and hygiene review, algorithmic generation of data quality validation tests, ongoing production testing of new data refreshes, and continuous anomaly monitoring of datasets. TestGen is part of DataKitchen's Open Source Data Observability.</p>*

## Features

What does DataKitchen's DataOps Data Quality TestGen do? It helps you understand and <b>find data issues in new data</b>.
<p align="center">
<img alt="DatKitchen Open Source Data Quality TestGen Features - New Data" src="https://datakitchen.io/wp-content/uploads/2024/07/Screenshot-2024-07-23-at-2.22.57 PM.png" width="70%">
</p>
It constantly <b>watches your data for data quality anomalies</b> and lets you drill into problems.
<br></br>
<p align="center">
<img alt="DataKitchen Open Source Data Quality TestGen Features - Data Ingestion and Quality Testing" src="https://datakitchen.io/wp-content/uploads/2024/07/Screenshot-2024-07-23-at-2.23.07 PM.png" width="70%">
</p>
A <b>single place to manage Data Quality</b> across data sets, locations, and teams.
<br></br>
<p align="center">
<img alt="DataKitchen Open Source Data Quality TestGen Features - Single Place" src="https://datakitchen.io/wp-content/uploads/2024/07/Screenshot-dataops-testgen-centralize.png" width="70%">
</p>

## Installation with dk-installer (recommended)

The [dk-installer](https://github.com/DataKitchen/data-observability-installer/?tab=readme-ov-file#install-the-testgen-application) program installs DataOps Data Quality TestGen as a [Docker Compose](https://docs.docker.com/compose/) application. This is the recommended mode of installation as Docker encapsulates and isolates the application from other software on your machine and does not require you to manage Python dependencies.

### Install the prerequisite software

| Software                | Tested Versions               | Command to check version                |
|-------------------------|-------------------------|-------------------------------|
| [Python](https://www.python.org/downloads/) <br/>- Most Linux and macOS systems have Python pre-installed. <br/>- On Windows machines, you will need to download and install it.        | 3.9, 3.10, 3.11, 3.12                | `python3 --version`                |
| [Docker](https://docs.docker.com/get-docker/) <br/>[Docker Compose](https://docs.docker.com/compose/install/)         | 25.0.3, 26.1.1, <br/> 2.24.6, 2.27.0, 2.28.1        | `docker -v` <br/> `docker compose version`         |

### Download the installer

On Unix-based operating systems, use the following command to download it to the current directory. We recommend creating a new, empty directory.

```shell
curl -o dk-installer.py 'https://raw.githubusercontent.com/DataKitchen/data-observability-installer/main/dk-installer.py'
```

* Alternatively, you can manually download the [`dk-installer.py`](https://github.com/DataKitchen/data-observability-installer/blob/main/dk-installer.py) file from the [data-observability-installer](https://github.com/DataKitchen/data-observability-installer) repository.
* All commands listed below should be run from the folder containing this file.
* For usage help and command options, run `python3 dk-installer.py --help` or `python3 dk-installer.py <command> --help`.

### Install the TestGen application

The installation downloads the latest Docker images for TestGen and deploys a new Docker Compose application. The process may take 5~10 minutes depending on your machine and network connection.

```shell
python3 dk-installer.py tg install
```

The `--port` option may be used to set a custom localhost port for the application (default: 8501).

To enable SSL for HTTPS support, use the `--ssl-cert-file` and `--ssl-key-file` options to specify local file paths to your SSL certificate and key files.

Once the installation completes, verify that you can login to the UI with the URL and credentials provided in the output.

### Optional: Run the TestGen demo setup

The [Data Observability quickstart](https://docs.datakitchen.io/articles/open-source-data-observability/data-observability-overview) walks you through DataOps Data Quality TestGen capabilities to demonstrate how it covers critical use cases for data and analytic teams.

```shell
python3 dk-installer.py tg run-demo
```

In the TestGen UI, you will see that new data profiling and test results have been generated.

## Installation with pip

As an alternative to the Docker Compose [installation with dk-installer (recommended)](#installation-with-dk-installer-recommended), DataOps Data Quality TestGen can also be installed as a Python package via [pip](https://pip.pypa.io/en/stable/). This mode of installation uses the [dataops-testgen](https://pypi.org/project/dataops-testgen/) package published to PyPI, and it requires a PostgreSQL instance to be provisioned for the application database.

### Install the prerequisite software

| Software                                                                                                                                                                         | Tested Versions  | Command to check version               |
|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------|------------------------------|
| [Python](https://www.python.org/downloads/) <br/>- Most Linux and macOS systems have Python pre-installed. <br/>- On Windows machines, you will need to download and install it. | 3.10, 3.11, 3.12 | `python3 --version`               |
| [PostgreSQL](https://www.postgresql.org/download/)                                                                                                                                                                     | 14.1, 15.8, 16.4       | `psql --version`|

### Install the TestGen package

We recommend using a Python virtual environment to avoid any dependency conflicts with other applications installed on your machine. The [venv](https://docs.python.org/3/library/venv.html#creating-virtual-environments) module, which is part of the Python standard library, or other third-party tools, like [virtualenv](https://virtualenv.pypa.io/en/latest/) or [conda](https://docs.conda.io/en/latest/), can be used.

Create and activate a virtual environment with a TestGen-compatible version of Python (`>=3.10`). The steps may vary based on your operating system and Python installation - the [Python packaging user guide](https://packaging.python.org/en/latest/tutorials/installing-packages/) is a useful reference.

_On Linux/Mac_
```shell
python3 -m venv venv
source venv/bin/activate
```

_On Windows_
```powershell
py -3.10 -m venv venv
venv\Scripts\activate
```

Within the virtual environment, install the TestGen package with pip.
```shell
pip install dataops-testgen
```

Verify that the [_testgen_ command line](https://docs.datakitchen.io/articles/#!dataops-testgen-help/testgen-commands-and-details) works.
```shell
testgen --help
```

### Set up the application database in PostgresSQL

Create a `local.env` file with the following environment variables, replacing the `<value>` placeholders with appropriate values. Refer to the [TestGen Configuration](docs/configuration.md) document for more details, defaults, and other supported configuration.
```shell
# Connection parameters for the PostgreSQL server
export TG_METADATA_DB_HOST=<postgres_hostname>
export TG_METADATA_DB_PORT=<postgres_port>

# Connection credentials for the PostgreSQL server
# This role must have privileges to create roles, users, database and schema so that the application database can be initialized
export TG_METADATA_DB_USER=<postgres_username>
export TG_METADATA_DB_PASSWORD=<postgres_password>

# Set a password and arbitrary string (the "salt") to be used for encrypting secrets in the application database
export TG_DECRYPT_PASSWORD=<encryption_password>
export TG_DECRYPT_SALT=<encryption_salt>

# Set credentials for the default admin user to be created for TestGen
export TESTGEN_USERNAME=<username>
export TESTGEN_PASSWORD=<password>

# Set an accessible path for storing application logs
export TESTGEN_LOG_FILE_PATH=<path_for_logs>
```

Source the file to apply the environment variables. For the Windows equivalent, refer to [this guide](https://bennett4.medium.com/windows-alternative-to-source-env-for-setting-environment-variables-606be2a6d3e1).
```shell
source local.env
```

Make sure the PostgreSQL database server is up and running. Initialize the application database for TestGen. 
```shell
testgen setup-system-db --yes
```

### Run the TestGen UI

Run the following command to start the TestGen UI. It will open the browser at [http://localhost:8501](http://localhost:8501).

```shell
testgen ui run
```

Verify that you can login to the UI with the `TESTGEN_USERNAME` and `TESTGEN_PASSWORD` values that you configured in the environment variables.

### Optional: Run the TestGen demo setup

The [Data Observability quickstart](https://docs.datakitchen.io/articles/open-source-data-observability/data-observability-overview) walks you through DataOps Data Quality TestGen capabilities to demonstrate how it covers critical use cases for data and analytic teams.

```shell
testgen quick-start --delete-target-db
testgen run-profile --table-group-id 0ea85e17-acbe-47fe-8394-9970725ad37d
testgen run-test-generation --table-group-id 0ea85e17-acbe-47fe-8394-9970725ad37d
testgen run-tests --project-key DEFAULT --test-suite-key default-suite-1
testgen quick-start --simulate-fast-forward
```

In the TestGen UI, you will see that new data profiling and test results have been generated.


## Product Documentation

[DataOps Data Quality TestGen](https://docs.datakitchen.io/articles/dataops-testgen-help/dataops-testgen-help)

## Useful Commands

The [dk-installer](https://github.com/DataKitchen/data-observability-installer/?tab=readme-ov-file#install-the-testgen-application) and [docker compose CLI](https://docs.docker.com/compose/reference/) can be used to operate the TestGen application installed using dk-installer. All commands must be run in the same folder that contains the `dk-installer.py` and `docker-compose.yml` files used by the installation.

### Remove demo data

After completing the quickstart, you can remove the demo data from the application with the following command.

```shell
python3 dk-installer.py tg delete-demo
```

### Upgrade to latest version

New releases of TestGen are announced on the `#releases` channel on [Data Observability Slack](https://data-observability-slack.datakitchen.io/join), and release notes can be found on the [DataKitchen documentation portal](https://docs.datakitchen.io/articles/#!dataops-testgen-help/testgen-release-notes/a/h1_1691719522). Use the following command to upgrade to the latest released version.

 ```shell
 python3 dk-installer.py tg upgrade
 ```

### Uninstall the application

The following command uninstalls the Docker Compose application and removes all data, containers, and images related to TestGen from your machine.

```shell
python3 dk-installer.py tg delete
```

### Access the _testgen_ CLI

The [_testgen_ command line](https://docs.datakitchen.io/articles/#!dataops-testgen-help/testgen-commands-and-details) can be accessed within the running container.

```shell
docker compose exec engine bash
```

Use `exit` to return to the regular terminal.

### Stop the application

```shell
docker compose down
```

### Restart the application

```shell
docker compose up -d
```

## What Next?

### Getting started guide
We recommend you start by going through the [Data Observability Overview Demo](https://docs.datakitchen.io/articles/open-source-data-observability/data-observability-overview).

### Support
For support requests, [join the Data Observability Slack](https://data-observability-slack.datakitchen.io/join) 👋 and post on the `#support` channel.

### Connect to your database
Follow [these instructions](https://docs.datakitchen.io/articles/#!dataops-testgen-help/connect-your-database) to improve the quality of data in your database.

### Community
Talk and learn with other data practitioners who are building with DataKitchen. Share knowledge, get help, and contribute to our open-source project.

Join our community here:

* 👋 [Join us on Slack](https://data-observability-slack.datakitchen.io/join), this is also how you get support (see above)

* 🌟 [Star us on GitHub](https://github.com/DataKitchen/data-observability-installer)

* 🐦 [Follow us on Twitter](https://twitter.com/i/flow/login?redirect_after_login=%2Fdatakitchen_io)

* 🕴️ [Follow us on LinkedIn](https://www.linkedin.com/company/datakitchen)

* 📺 [Get Free DataOps Fundamentals Certification](https://info.datakitchen.io/training-certification-dataops-fundamentals)

* 📚 [Read our blog posts](https://datakitchen.io/blog/)

* 🗃 [Sign The DataOps Manifesto](https://DataOpsManifesto.org)

* 🗃 [Sign The Data Journey Manifesto](https://DataJourneyManifesto.org)


### Contributing
For details on contributing or running the project for development, check out our [contributing guide](CONTRIBUTING.md).

### License
DataKitchen's DataOps Data Quality TestGen is Apache 2.0 licensed.

            

Raw data

            {
    "_id": null,
    "home_page": null,
    "name": "dataops-testgen",
    "maintainer": null,
    "docs_url": null,
    "requires_python": ">=3.10",
    "maintainer_email": "\"DataKitchen, Inc.\" <info@datakitchen.io>",
    "keywords": "dataops, data, quality, testing, database, profiling",
    "author": null,
    "author_email": "\"DataKitchen, Inc.\" <info@datakitchen.io>",
    "download_url": "https://files.pythonhosted.org/packages/ff/ce/ce60a041def4b81b610832a9b62d5c498c4b73f8f7038bccd8d4e584aebe/dataops-testgen-2.15.2.tar.gz",
    "platform": null,
    "description": "# DataOps Data Quality TestGen\n![apache 2.0 license Badge](https://img.shields.io/badge/License%20-%20Apache%202.0%20-%20blue) ![PRs Badge](https://img.shields.io/badge/PRs%20-%20Welcome%20-%20green) [![Latest Version](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fhub.docker.com%2Fv2%2Frepositories%2Fdatakitchen%2Fdataops-testgen%2Ftags%2F&query=results%5B0%5D.name&label=latest%20version&color=06A04A)](https://hub.docker.com/r/datakitchen/dataops-testgen) [![Docker Pulls](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fhub.docker.com%2Fv2%2Frepositories%2Fdatakitchen%2Fdataops-testgen%2F&query=pull_count&style=flat&label=docker%20pulls&color=06A04A)](https://hub.docker.com/r/datakitchen/dataops-testgen) [![Documentation](https://img.shields.io/badge/docs-On%20datakitchen.io-06A04A?style=flat)](https://docs.datakitchen.io/articles/#!dataops-testgen-help/dataops-testgen-help) [![Static Badge](https://img.shields.io/badge/Slack-Join%20Discussion-blue?style=flat&logo=slack)](https://data-observability-slack.datakitchen.io/join)\n\n*<p style=\"text-align: center;\">DataOps Data Quality TestGen, or \"TestGen\" for short, can help you find data issues so you can alert your users and notify your suppliers. It does this by delivering simple, fast data quality test generation and execution by data profiling, new dataset screening and hygiene review, algorithmic generation of data quality validation tests, ongoing production testing of new data refreshes, and continuous anomaly monitoring of datasets. TestGen is part of DataKitchen's Open Source Data Observability.</p>*\n\n## Features\n\nWhat does DataKitchen's DataOps Data Quality TestGen do? It helps you understand and <b>find data issues in new data</b>.\n<p align=\"center\">\n<img alt=\"DatKitchen Open Source Data Quality TestGen Features - New Data\" src=\"https://datakitchen.io/wp-content/uploads/2024/07/Screenshot-2024-07-23-at-2.22.57\u202fPM.png\" width=\"70%\">\n</p>\nIt constantly <b>watches your data for data quality anomalies</b> and lets you drill into problems.\n<br></br>\n<p align=\"center\">\n<img alt=\"DataKitchen Open Source Data Quality TestGen Features - Data Ingestion and Quality Testing\" src=\"https://datakitchen.io/wp-content/uploads/2024/07/Screenshot-2024-07-23-at-2.23.07\u202fPM.png\" width=\"70%\">\n</p>\nA <b>single place to manage Data Quality</b> across data sets, locations, and teams.\n<br></br>\n<p align=\"center\">\n<img alt=\"DataKitchen Open Source Data Quality TestGen Features - Single Place\" src=\"https://datakitchen.io/wp-content/uploads/2024/07/Screenshot-dataops-testgen-centralize.png\" width=\"70%\">\n</p>\n\n## Installation with dk-installer (recommended)\n\nThe [dk-installer](https://github.com/DataKitchen/data-observability-installer/?tab=readme-ov-file#install-the-testgen-application) program installs DataOps Data Quality TestGen as a [Docker Compose](https://docs.docker.com/compose/) application. This is the recommended mode of installation as Docker encapsulates and isolates the application from other software on your machine and does not require you to manage Python dependencies.\n\n### Install the prerequisite software\n\n| Software                | Tested Versions               | Command to check version                |\n|-------------------------|-------------------------|-------------------------------|\n| [Python](https://www.python.org/downloads/) <br/>- Most Linux and macOS systems have Python pre-installed. <br/>- On Windows machines, you will need to download and install it.        | 3.9, 3.10, 3.11, 3.12                | `python3 --version`                |\n| [Docker](https://docs.docker.com/get-docker/) <br/>[Docker Compose](https://docs.docker.com/compose/install/)         | 25.0.3, 26.1.1, <br/> 2.24.6, 2.27.0, 2.28.1        | `docker -v` <br/> `docker compose version`         |\n\n### Download the installer\n\nOn Unix-based operating systems, use the following command to download it to the current directory. We recommend creating a new, empty directory.\n\n```shell\ncurl -o dk-installer.py 'https://raw.githubusercontent.com/DataKitchen/data-observability-installer/main/dk-installer.py'\n```\n\n* Alternatively, you can manually download the [`dk-installer.py`](https://github.com/DataKitchen/data-observability-installer/blob/main/dk-installer.py) file from the [data-observability-installer](https://github.com/DataKitchen/data-observability-installer) repository.\n* All commands listed below should be run from the folder containing this file.\n* For usage help and command options, run `python3 dk-installer.py --help` or `python3 dk-installer.py <command> --help`.\n\n### Install the TestGen application\n\nThe installation downloads the latest Docker images for TestGen and deploys a new Docker Compose application. The process may take 5~10 minutes depending on your machine and network connection.\n\n```shell\npython3 dk-installer.py tg install\n```\n\nThe `--port` option may be used to set a custom localhost port for the application (default: 8501).\n\nTo enable SSL for HTTPS support, use the `--ssl-cert-file` and `--ssl-key-file` options to specify local file paths to your SSL certificate and key files.\n\nOnce the installation completes, verify that you can login to the UI with the URL and credentials provided in the output.\n\n### Optional: Run the TestGen demo setup\n\nThe [Data Observability quickstart](https://docs.datakitchen.io/articles/open-source-data-observability/data-observability-overview) walks you through DataOps Data Quality TestGen capabilities to demonstrate how it covers critical use cases for data and analytic teams.\n\n```shell\npython3 dk-installer.py tg run-demo\n```\n\nIn the TestGen UI, you will see that new data profiling and test results have been generated.\n\n## Installation with pip\n\nAs an alternative to the Docker Compose [installation with dk-installer (recommended)](#installation-with-dk-installer-recommended), DataOps Data Quality TestGen can also be installed as a Python package via [pip](https://pip.pypa.io/en/stable/). This mode of installation uses the [dataops-testgen](https://pypi.org/project/dataops-testgen/) package published to PyPI, and it requires a PostgreSQL instance to be provisioned for the application database.\n\n### Install the prerequisite software\n\n| Software                                                                                                                                                                         | Tested Versions  | Command to check version               |\n|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------|------------------------------|\n| [Python](https://www.python.org/downloads/) <br/>- Most Linux and macOS systems have Python pre-installed. <br/>- On Windows machines, you will need to download and install it. | 3.10, 3.11, 3.12 | `python3 --version`               |\n| [PostgreSQL](https://www.postgresql.org/download/)                                                                                                                                                                     | 14.1, 15.8, 16.4       | `psql --version`|\n\n### Install the TestGen package\n\nWe recommend using a Python virtual environment to avoid any dependency conflicts with other applications installed on your machine. The [venv](https://docs.python.org/3/library/venv.html#creating-virtual-environments) module, which is part of the Python standard library, or other third-party tools, like [virtualenv](https://virtualenv.pypa.io/en/latest/) or [conda](https://docs.conda.io/en/latest/), can be used.\n\nCreate and activate a virtual environment with a TestGen-compatible version of Python (`>=3.10`). The steps may vary based on your operating system and Python installation - the [Python packaging user guide](https://packaging.python.org/en/latest/tutorials/installing-packages/) is a useful reference.\n\n_On Linux/Mac_\n```shell\npython3 -m venv venv\nsource venv/bin/activate\n```\n\n_On Windows_\n```powershell\npy -3.10 -m venv venv\nvenv\\Scripts\\activate\n```\n\nWithin the virtual environment, install the TestGen package with pip.\n```shell\npip install dataops-testgen\n```\n\nVerify that the [_testgen_ command line](https://docs.datakitchen.io/articles/#!dataops-testgen-help/testgen-commands-and-details) works.\n```shell\ntestgen --help\n```\n\n### Set up the application database in PostgresSQL\n\nCreate a `local.env` file with the following environment variables, replacing the `<value>` placeholders with appropriate values. Refer to the [TestGen Configuration](docs/configuration.md) document for more details, defaults, and other supported configuration.\n```shell\n# Connection parameters for the PostgreSQL server\nexport TG_METADATA_DB_HOST=<postgres_hostname>\nexport TG_METADATA_DB_PORT=<postgres_port>\n\n# Connection credentials for the PostgreSQL server\n# This role must have privileges to create roles, users, database and schema so that the application database can be initialized\nexport TG_METADATA_DB_USER=<postgres_username>\nexport TG_METADATA_DB_PASSWORD=<postgres_password>\n\n# Set a password and arbitrary string (the \"salt\") to be used for encrypting secrets in the application database\nexport TG_DECRYPT_PASSWORD=<encryption_password>\nexport TG_DECRYPT_SALT=<encryption_salt>\n\n# Set credentials for the default admin user to be created for TestGen\nexport TESTGEN_USERNAME=<username>\nexport TESTGEN_PASSWORD=<password>\n\n# Set an accessible path for storing application logs\nexport TESTGEN_LOG_FILE_PATH=<path_for_logs>\n```\n\nSource the file to apply the environment variables. For the Windows equivalent, refer to [this guide](https://bennett4.medium.com/windows-alternative-to-source-env-for-setting-environment-variables-606be2a6d3e1).\n```shell\nsource local.env\n```\n\nMake sure the PostgreSQL database server is up and running. Initialize the application database for TestGen. \n```shell\ntestgen setup-system-db --yes\n```\n\n### Run the TestGen UI\n\nRun the following command to start the TestGen UI. It will open the browser at [http://localhost:8501](http://localhost:8501).\n\n```shell\ntestgen ui run\n```\n\nVerify that you can login to the UI with the `TESTGEN_USERNAME` and `TESTGEN_PASSWORD` values that you configured in the environment variables.\n\n### Optional: Run the TestGen demo setup\n\nThe [Data Observability quickstart](https://docs.datakitchen.io/articles/open-source-data-observability/data-observability-overview) walks you through DataOps Data Quality TestGen capabilities to demonstrate how it covers critical use cases for data and analytic teams.\n\n```shell\ntestgen quick-start --delete-target-db\ntestgen run-profile --table-group-id 0ea85e17-acbe-47fe-8394-9970725ad37d\ntestgen run-test-generation --table-group-id 0ea85e17-acbe-47fe-8394-9970725ad37d\ntestgen run-tests --project-key DEFAULT --test-suite-key default-suite-1\ntestgen quick-start --simulate-fast-forward\n```\n\nIn the TestGen UI, you will see that new data profiling and test results have been generated.\n\n\n## Product Documentation\n\n[DataOps Data Quality TestGen](https://docs.datakitchen.io/articles/dataops-testgen-help/dataops-testgen-help)\n\n## Useful Commands\n\nThe [dk-installer](https://github.com/DataKitchen/data-observability-installer/?tab=readme-ov-file#install-the-testgen-application) and [docker compose CLI](https://docs.docker.com/compose/reference/) can be used to operate the TestGen application installed using dk-installer. All commands must be run in the same folder that contains the `dk-installer.py` and `docker-compose.yml` files used by the installation.\n\n### Remove demo data\n\nAfter completing the quickstart, you can remove the demo data from the application with the following command.\n\n```shell\npython3 dk-installer.py tg delete-demo\n```\n\n### Upgrade to latest version\n\nNew releases of TestGen are announced on the `#releases` channel on [Data Observability Slack](https://data-observability-slack.datakitchen.io/join), and release notes can be found on the [DataKitchen documentation portal](https://docs.datakitchen.io/articles/#!dataops-testgen-help/testgen-release-notes/a/h1_1691719522). Use the following command to upgrade to the latest released version.\n\n ```shell\n python3 dk-installer.py tg upgrade\n ```\n\n### Uninstall the application\n\nThe following command uninstalls the Docker Compose application and removes all data, containers, and images related to TestGen from your machine.\n\n```shell\npython3 dk-installer.py tg delete\n```\n\n### Access the _testgen_ CLI\n\nThe [_testgen_ command line](https://docs.datakitchen.io/articles/#!dataops-testgen-help/testgen-commands-and-details) can be accessed within the running container.\n\n```shell\ndocker compose exec engine bash\n```\n\nUse `exit` to return to the regular terminal.\n\n### Stop the application\n\n```shell\ndocker compose down\n```\n\n### Restart the application\n\n```shell\ndocker compose up -d\n```\n\n## What Next?\n\n### Getting started guide\nWe recommend you start by going through the [Data Observability Overview Demo](https://docs.datakitchen.io/articles/open-source-data-observability/data-observability-overview).\n\n### Support\nFor support requests, [join the Data Observability Slack](https://data-observability-slack.datakitchen.io/join) \ud83d\udc4b and post on the `#support` channel.\n\n### Connect to your database\nFollow [these instructions](https://docs.datakitchen.io/articles/#!dataops-testgen-help/connect-your-database) to improve the quality of data in your database.\n\n### Community\nTalk and learn with other data practitioners who are building with DataKitchen. Share knowledge, get help, and contribute to our open-source project.\n\nJoin our community here:\n\n* \ud83d\udc4b [Join us on Slack](https://data-observability-slack.datakitchen.io/join), this is also how you get support (see above)\n\n* \ud83c\udf1f [Star us on GitHub](https://github.com/DataKitchen/data-observability-installer)\n\n* \ud83d\udc26 [Follow us on Twitter](https://twitter.com/i/flow/login?redirect_after_login=%2Fdatakitchen_io)\n\n* \ud83d\udd74\ufe0f [Follow us on LinkedIn](https://www.linkedin.com/company/datakitchen)\n\n* \ud83d\udcfa [Get Free DataOps Fundamentals Certification](https://info.datakitchen.io/training-certification-dataops-fundamentals)\n\n* \ud83d\udcda [Read our blog posts](https://datakitchen.io/blog/)\n\n* \ud83d\uddc3 [Sign The DataOps Manifesto](https://DataOpsManifesto.org)\n\n* \ud83d\uddc3 [Sign The Data Journey Manifesto](https://DataJourneyManifesto.org)\n\n\n### Contributing\nFor details on contributing or running the project for development, check out our [contributing guide](CONTRIBUTING.md).\n\n### License\nDataKitchen's DataOps Data Quality TestGen is Apache 2.0 licensed.\n",
    "bugtrack_url": null,
    "license": null,
    "summary": "DataKitchen's Data Quality DataOps TestGen",
    "version": "2.15.2",
    "project_urls": {
        "Bug Tracker": "https://github.com/DataKitchen/dataops-testgen/issues",
        "Documentation": "https://docs.datakitchen.io/articles/#!dataops-testgen-help/dataops-testgen-help",
        "Homepage": "https://example.com",
        "Release Notes": "https://docs.datakitchen.io/articles/#!dataops-testgen-help/testgen-release-notes",
        "Slack": "https://data-observability-slack.datakitchen.io/join",
        "Source Code": "https://github.com/DataKitchen/dataops-testgen"
    },
    "split_keywords": [
        "dataops",
        " data",
        " quality",
        " testing",
        " database",
        " profiling"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "25578a303de0fb6d02a2a3ff7ab373d35c5342aecc2d47e79a6c8cc4f84d7d5f",
                "md5": "8d2ec4656bd7863064afd7f89f6101a0",
                "sha256": "38629b420a43f19b249d6d133b094853a14dd22fe29196f8e3593db955c4f016"
            },
            "downloads": -1,
            "filename": "dataops_testgen-2.15.2-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "8d2ec4656bd7863064afd7f89f6101a0",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.10",
            "size": 4424068,
            "upload_time": "2024-10-12T00:43:23",
            "upload_time_iso_8601": "2024-10-12T00:43:23.449768Z",
            "url": "https://files.pythonhosted.org/packages/25/57/8a303de0fb6d02a2a3ff7ab373d35c5342aecc2d47e79a6c8cc4f84d7d5f/dataops_testgen-2.15.2-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "ffcece60a041def4b81b610832a9b62d5c498c4b73f8f7038bccd8d4e584aebe",
                "md5": "abb4764569d8db3177cae3d7074612b6",
                "sha256": "9e5f160d107aea3156773cdc881abac7b5af2baf60e57ab7cedde3caa3b6e88a"
            },
            "downloads": -1,
            "filename": "dataops-testgen-2.15.2.tar.gz",
            "has_sig": false,
            "md5_digest": "abb4764569d8db3177cae3d7074612b6",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.10",
            "size": 4325718,
            "upload_time": "2024-10-12T00:43:28",
            "upload_time_iso_8601": "2024-10-12T00:43:28.427146Z",
            "url": "https://files.pythonhosted.org/packages/ff/ce/ce60a041def4b81b610832a9b62d5c498c4b73f8f7038bccd8d4e584aebe/dataops-testgen-2.15.2.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-10-12 00:43:28",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "DataKitchen",
    "github_project": "dataops-testgen",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "lcname": "dataops-testgen"
}
        
Elapsed time: 0.79548s