# OpenDataDiscovery dbt tests metadata collecting
[](https://badge.fury.io/py/odd-dbt)
CLI tool helps run and ingest dbt test to platform.
It can be used as separated CLI tool or within [ODD CLI](https://github.com/opendatadiscovery/odd-cli) package which
provides some useful additional features for working with OpenDataDiscovery.
## Supported adapters
| Adapter | version |
|-----------|---------|
| Snowflake | ^1.6 |
| Postgres | ^1.6 |
Profiles inside the file looks different for each type of data source.
**Snowflake** host_settings value is created from field `account`. Field value should be `<account_identifier>`
For example the URL for an account uses the following format: `<account_identifier>`.snowflakecomputing.com
Example Snowflake account identifier `hj1234.eu-central-1`.
## Supported tests types
1. [x] Generic tests
2. [ ] Singular tests. Currently Singular tests are not supported.
## Installation
```pip install odd-dbt```
## To see all available commands
```
odd_dbt_test --help
```
## Example
For each command that involves sending information to OpenDataDiscovery platform exists set of env variables:
1. `ODD_PLATFORM_HOST` - Where you platform is
2. `ODD_PLATFORM_TOKEN` - Token for ingesting data to platform (How to create [token](https://docs.opendatadiscovery.org/configuration-and-deployment/trylocally#create-collector-entity)?)
3. `DBT_DATA_SOURCE_ODDRN` - Unique oddrn string describes dbt source, i.e '//dbt/host/localhost'
It is recommended to add them as ENV variables or provide as flags to each command
```
export ODD_PLATFORM_HOST=http://localhost:8080
export ODD_PLATFORM_TOKEN=token***
export DBT_DATA_SOURCE_ODDRN=//dbt/host/localhost
```
### Commands
`create-datasource` - helps to register dbt as data source at OpenDataDiscovery platform. User later for ingesting metadata.
```commandline
odd_dbt_test create-datasource --name=my_local_dbt --dbt-host=localhost
```
`ingest-test` - Read results_run file under the target folder to parse and ingest metadata.
```commandline
odd_dbt_test ingest-test --profile=my_profile
```
`test` - Proxy command to `dbt test`, then reads results_run file under the target folder to parse and ingest metadata.
```commandline
odd_dbt_test test --profile=my_profile
```
### Run commands programmatically
You could run that scrip to read, parse and ingest test results to the platform.
```python
# ingest_test_result.py
from odd_dbt import config
from odd_dbt.domain.cli_args import CliArgs
from odd_dbt.libs.dbt import get_context
from odd_dbt.libs.odd import create_dbt_generator_from_oddrn
from odd_dbt.service.odd import ingest_entities
from odd_dbt.mapper.test_results import DbtTestMapper
from odd_dbt.mapper.lineage import DbtLineageMapper
cfg = config.Config() # All fields can be set manually or read from ENV variables
client = config.create_odd_client(host=cfg.odd_platform_host, token=cfg.odd_platform_token)
generator = create_dbt_generator_from_oddrn(oddrn=cfg.dbt_data_source_oddrn)
cli_args = CliArgs.default()
context = get_context(cli_args=cli_args)
# Ingest lineage
data_entities = DbtLineageMapper(context=context, generator=generator).map()
ingest_entities(data_entities, client)
# Or ingest test results
data_entities = DbtTestMapper(context=context, generator=generator).map()
ingest_entities(data_entities, client)
```
Raw data
{
"_id": null,
"home_page": null,
"name": "odd-dbt",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.9",
"maintainer_email": null,
"keywords": "Open Data Discovery, dbt, Metadata, Data Discovery, Data Observability",
"author": "Mateusz Kulas",
"author_email": "mkulas@provectus.com",
"download_url": "https://files.pythonhosted.org/packages/5c/25/9220fe49d50de4d63e17dc43b79ef8b70d258130b01572c3b7e4e02ec66a/odd_dbt-0.2.12.tar.gz",
"platform": null,
"description": "# OpenDataDiscovery dbt tests metadata collecting\n\n[](https://badge.fury.io/py/odd-dbt)\n\nCLI tool helps run and ingest dbt test to platform.\n\nIt can be used as separated CLI tool or within [ODD CLI](https://github.com/opendatadiscovery/odd-cli) package which\nprovides some useful additional features for working with OpenDataDiscovery.\n\n## Supported adapters\n\n| Adapter | version |\n|-----------|---------|\n| Snowflake | ^1.6 |\n| Postgres | ^1.6 |\n\nProfiles inside the file looks different for each type of data source.\n\n**Snowflake** host_settings value is created from field `account`. Field value should be `<account_identifier>`\nFor example the URL for an account uses the following format: `<account_identifier>`.snowflakecomputing.com\nExample Snowflake account identifier `hj1234.eu-central-1`.\n\n## Supported tests types\n\n1. [x] Generic tests\n2. [ ] Singular tests. Currently Singular tests are not supported.\n\n## Installation\n```pip install odd-dbt```\n\n## To see all available commands\n```\nodd_dbt_test --help\n```\n\n## Example\nFor each command that involves sending information to OpenDataDiscovery platform exists set of env variables:\n1. `ODD_PLATFORM_HOST` - Where you platform is\n2. `ODD_PLATFORM_TOKEN` - Token for ingesting data to platform (How to create [token](https://docs.opendatadiscovery.org/configuration-and-deployment/trylocally#create-collector-entity)?)\n3. `DBT_DATA_SOURCE_ODDRN` - Unique oddrn string describes dbt source, i.e '//dbt/host/localhost'\n\nIt is recommended to add them as ENV variables or provide as flags to each command\n```\nexport ODD_PLATFORM_HOST=http://localhost:8080\nexport ODD_PLATFORM_TOKEN=token***\nexport DBT_DATA_SOURCE_ODDRN=//dbt/host/localhost\n```\n\n### Commands\n`create-datasource` - helps to register dbt as data source at OpenDataDiscovery platform. User later for ingesting metadata.\n```commandline\nodd_dbt_test create-datasource --name=my_local_dbt --dbt-host=localhost\n```\n\n`ingest-test` - Read results_run file under the target folder to parse and ingest metadata.\n```commandline\nodd_dbt_test ingest-test --profile=my_profile\n```\n\n`test` - Proxy command to `dbt test`, then reads results_run file under the target folder to parse and ingest metadata.\n```commandline\nodd_dbt_test test --profile=my_profile\n```\n\n### Run commands programmatically\nYou could run that scrip to read, parse and ingest test results to the platform.\n\n```python\n# ingest_test_result.py\n\nfrom odd_dbt import config\nfrom odd_dbt.domain.cli_args import CliArgs\nfrom odd_dbt.libs.dbt import get_context\nfrom odd_dbt.libs.odd import create_dbt_generator_from_oddrn\nfrom odd_dbt.service.odd import ingest_entities\nfrom odd_dbt.mapper.test_results import DbtTestMapper\nfrom odd_dbt.mapper.lineage import DbtLineageMapper\n\ncfg = config.Config() # All fields can be set manually or read from ENV variables\nclient = config.create_odd_client(host=cfg.odd_platform_host, token=cfg.odd_platform_token)\ngenerator = create_dbt_generator_from_oddrn(oddrn=cfg.dbt_data_source_oddrn)\n\ncli_args = CliArgs.default()\ncontext = get_context(cli_args=cli_args)\n\n# Ingest lineage\ndata_entities = DbtLineageMapper(context=context, generator=generator).map()\ningest_entities(data_entities, client)\n\n# Or ingest test results\ndata_entities = DbtTestMapper(context=context, generator=generator).map()\ningest_entities(data_entities, client)\n```",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "OpenDataDiscovery Action for dbt",
"version": "0.2.12",
"project_urls": null,
"split_keywords": [
"open data discovery",
" dbt",
" metadata",
" data discovery",
" data observability"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "616c2cec867af7f77c609483aaf7d7baf94738b77ffd106cd4da0efd388c9b9c",
"md5": "baca8c5464a317d9464ad2f5acb90661",
"sha256": "e4d43c99fbc587d200ec7689860c709e7493a1e3d67e8a47c38c5a1b580e2093"
},
"downloads": -1,
"filename": "odd_dbt-0.2.12-py3-none-any.whl",
"has_sig": false,
"md5_digest": "baca8c5464a317d9464ad2f5acb90661",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.9",
"size": 24155,
"upload_time": "2024-05-08T10:22:14",
"upload_time_iso_8601": "2024-05-08T10:22:14.030136Z",
"url": "https://files.pythonhosted.org/packages/61/6c/2cec867af7f77c609483aaf7d7baf94738b77ffd106cd4da0efd388c9b9c/odd_dbt-0.2.12-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "5c259220fe49d50de4d63e17dc43b79ef8b70d258130b01572c3b7e4e02ec66a",
"md5": "e531d6909cb04d0a13461fb654da50c4",
"sha256": "250d19d31f259b34a1d6b1af6342afca78139eab0101f96bf992e86f2616cf24"
},
"downloads": -1,
"filename": "odd_dbt-0.2.12.tar.gz",
"has_sig": false,
"md5_digest": "e531d6909cb04d0a13461fb654da50c4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.9",
"size": 18252,
"upload_time": "2024-05-08T10:22:15",
"upload_time_iso_8601": "2024-05-08T10:22:15.859662Z",
"url": "https://files.pythonhosted.org/packages/5c/25/9220fe49d50de4d63e17dc43b79ef8b70d258130b01572c3b7e4e02ec66a/odd_dbt-0.2.12.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-08 10:22:15",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "odd-dbt"
}
JSON