# OpenDataDiscovery dbt tests metadata collecting
[](https://badge.fury.io/py/odd-dbt)
CLI tool helps run and ingest dbt test to platform.
It can be used as separated CLI tool or within [ODD CLI](https://github.com/opendatadiscovery/odd-cli) package which
provides some useful additional features for working with OpenDataDiscovery.
## Supported adapters
| Adapter | version |
|-----------|---------|
| Snowflake | ^1.6 |
| Postgres | ^1.6 |
Profiles inside the file looks different for each type of data source.
**Snowflake** host_settings value is created from field `account`. Field value should be `<account_identifier>`
For example the URL for an account uses the following format: `<account_identifier>`.snowflakecomputing.com
Example Snowflake account identifier `hj1234.eu-central-1`.
## Supported tests types
1. [x] Generic tests
2. [ ] Singular tests. Currently Singular tests are not supported.
## Installation
```pip install odd-dbt```
## To see all available commands
```
odd_dbt_test --help
```
## Example
For each command that involves sending information to OpenDataDiscovery platform exists set of env variables:
1. `ODD_PLATFORM_HOST` - Where you platform is
2. `ODD_PLATFORM_TOKEN` - Token for ingesting data to platform (How to create [token](https://docs.opendatadiscovery.org/configuration-and-deployment/trylocally#create-collector-entity)?)
3. `DBT_DATA_SOURCE_ODDRN` - Unique oddrn string describes dbt source, i.e '//dbt/host/localhost'
It is recommended to add them as ENV variables or provide as flags to each command
```
export ODD_PLATFORM_HOST=http://localhost:8080
export ODD_PLATFORM_TOKEN=token***
export DBT_DATA_SOURCE_ODDRN=//dbt/host/localhost
```
### Commands
`create-datasource` - helps to register dbt as data source at OpenDataDiscovery platform. User later for ingesting metadata.
```commandline
odd_dbt_test create-datasource --name=my_local_dbt --dbt-host=localhost
```
`ingest-test` - Read results_run file under the target folder to parse and ingest metadata.
```commandline
odd_dbt_test ingest-test --profile=my_profile
```
`test` - Proxy command to `dbt test`, then reads results_run file under the target folder to parse and ingest metadata.
```commandline
odd_dbt_test test --profile=my_profile
```
### Run commands programmatically
You could run that scrip to read, parse and ingest test results to the platform.
```python
# ingest_test_result.py
from odd_dbt import config
from odd_dbt.domain.cli_args import CliArgs
from odd_dbt.libs.dbt import get_context
from odd_dbt.libs.odd import create_dbt_generator_from_oddrn
from odd_dbt.service.odd import ingest_entities
from odd_dbt.mapper.test_results import DbtTestMapper
from odd_dbt.mapper.lineage import DbtLineageMapper
cfg = config.Config() # All fields can be set manually or read from ENV variables
client = config.create_odd_client(host=cfg.odd_platform_host, token=cfg.odd_platform_token)
generator = create_dbt_generator_from_oddrn(oddrn=cfg.dbt_data_source_oddrn)
cli_args = CliArgs.default()
context = get_context(cli_args=cli_args)
# Ingest lineage
data_entities = DbtLineageMapper(context=context, generator=generator).map()
ingest_entities(data_entities, client)
# Or ingest test results
data_entities = DbtTestMapper(context=context, generator=generator).map()
ingest_entities(data_entities, client)
```
Raw data
{
"_id": null,
"home_page": "",
"name": "odd-dbt",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.9,<4.0",
"maintainer_email": "",
"keywords": "Open Data Discovery,dbt,Metadata,Data Discovery,Data Observability",
"author": "Mateusz Kulas",
"author_email": "mkulas@provectus.com",
"download_url": "https://files.pythonhosted.org/packages/87/5b/e240b7129ec3665d08164172b8ee73f2772f91b9018109191e3fb89be080/odd_dbt-0.2.11.tar.gz",
"platform": null,
"description": "# OpenDataDiscovery dbt tests metadata collecting\n\n[](https://badge.fury.io/py/odd-dbt)\n\nCLI tool helps run and ingest dbt test to platform.\n\nIt can be used as separated CLI tool or within [ODD CLI](https://github.com/opendatadiscovery/odd-cli) package which\nprovides some useful additional features for working with OpenDataDiscovery.\n\n## Supported adapters\n\n| Adapter | version |\n|-----------|---------|\n| Snowflake | ^1.6 |\n| Postgres | ^1.6 |\n\nProfiles inside the file looks different for each type of data source.\n\n**Snowflake** host_settings value is created from field `account`. Field value should be `<account_identifier>`\nFor example the URL for an account uses the following format: `<account_identifier>`.snowflakecomputing.com\nExample Snowflake account identifier `hj1234.eu-central-1`.\n\n## Supported tests types\n\n1. [x] Generic tests\n2. [ ] Singular tests. Currently Singular tests are not supported.\n\n## Installation\n```pip install odd-dbt```\n\n## To see all available commands\n```\nodd_dbt_test --help\n```\n\n## Example\nFor each command that involves sending information to OpenDataDiscovery platform exists set of env variables:\n1. `ODD_PLATFORM_HOST` - Where you platform is\n2. `ODD_PLATFORM_TOKEN` - Token for ingesting data to platform (How to create [token](https://docs.opendatadiscovery.org/configuration-and-deployment/trylocally#create-collector-entity)?)\n3. `DBT_DATA_SOURCE_ODDRN` - Unique oddrn string describes dbt source, i.e '//dbt/host/localhost'\n\nIt is recommended to add them as ENV variables or provide as flags to each command\n```\nexport ODD_PLATFORM_HOST=http://localhost:8080\nexport ODD_PLATFORM_TOKEN=token***\nexport DBT_DATA_SOURCE_ODDRN=//dbt/host/localhost\n```\n\n### Commands\n`create-datasource` - helps to register dbt as data source at OpenDataDiscovery platform. User later for ingesting metadata.\n```commandline\nodd_dbt_test create-datasource --name=my_local_dbt --dbt-host=localhost\n```\n\n`ingest-test` - Read results_run file under the target folder to parse and ingest metadata.\n```commandline\nodd_dbt_test ingest-test --profile=my_profile\n```\n\n`test` - Proxy command to `dbt test`, then reads results_run file under the target folder to parse and ingest metadata.\n```commandline\nodd_dbt_test test --profile=my_profile\n```\n\n### Run commands programmatically\nYou could run that scrip to read, parse and ingest test results to the platform.\n\n```python\n# ingest_test_result.py\n\nfrom odd_dbt import config\nfrom odd_dbt.domain.cli_args import CliArgs\nfrom odd_dbt.libs.dbt import get_context\nfrom odd_dbt.libs.odd import create_dbt_generator_from_oddrn\nfrom odd_dbt.service.odd import ingest_entities\nfrom odd_dbt.mapper.test_results import DbtTestMapper\nfrom odd_dbt.mapper.lineage import DbtLineageMapper\n\ncfg = config.Config() # All fields can be set manually or read from ENV variables\nclient = config.create_odd_client(host=cfg.odd_platform_host, token=cfg.odd_platform_token)\ngenerator = create_dbt_generator_from_oddrn(oddrn=cfg.dbt_data_source_oddrn)\n\ncli_args = CliArgs.default()\ncontext = get_context(cli_args=cli_args)\n\n# Ingest lineage\ndata_entities = DbtLineageMapper(context=context, generator=generator).map()\ningest_entities(data_entities, client)\n\n# Or ingest test results\ndata_entities = DbtTestMapper(context=context, generator=generator).map()\ningest_entities(data_entities, client)\n```",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "OpenDataDiscovery Action for dbt",
"version": "0.2.11",
"project_urls": null,
"split_keywords": [
"open data discovery",
"dbt",
"metadata",
"data discovery",
"data observability"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "bea596f98729263df45d86b87343447abe23dc3fff0760e76bacf6a7ae72b26d",
"md5": "c2e963543f6ca43745f7aecec8a5bdd0",
"sha256": "aa57952db2940f7b79a84dd1fc998d623183dc4eeb4e9393edc0d91849b29aef"
},
"downloads": -1,
"filename": "odd_dbt-0.2.11-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c2e963543f6ca43745f7aecec8a5bdd0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9,<4.0",
"size": 24154,
"upload_time": "2023-11-03T15:22:34",
"upload_time_iso_8601": "2023-11-03T15:22:34.448673Z",
"url": "https://files.pythonhosted.org/packages/be/a5/96f98729263df45d86b87343447abe23dc3fff0760e76bacf6a7ae72b26d/odd_dbt-0.2.11-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "875be240b7129ec3665d08164172b8ee73f2772f91b9018109191e3fb89be080",
"md5": "4409409278577085bc771b96c1d4fffa",
"sha256": "932eedb5ae81d9c01bbb725e6ac138b2c26f623e9ac84cb4a8675538192732b1"
},
"downloads": -1,
"filename": "odd_dbt-0.2.11.tar.gz",
"has_sig": false,
"md5_digest": "4409409278577085bc771b96c1d4fffa",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9,<4.0",
"size": 18266,
"upload_time": "2023-11-03T15:22:36",
"upload_time_iso_8601": "2023-11-03T15:22:36.216706Z",
"url": "https://files.pythonhosted.org/packages/87/5b/e240b7129ec3665d08164172b8ee73f2772f91b9018109191e3fb89be080/odd_dbt-0.2.11.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-03 15:22:36",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "odd-dbt"
}
JSON