# tap-dbt
[](https://pypi.org/project/tap-dbt/)
[](https://pypi.org/project/tap-dbt/)
[](https://hub.meltano.com/taps/dbt)
[](https://github.com/MeltanoLabs/tap-dbt/actions/workflows/ci.yml)
`tap-dbt` is a Singer tap for the [dbt Cloud API v2][dbtcloud].
Built with the [Singer SDK][sdk].
- [Installation](#Installation)
- [Features](#Features)
- [Configuration](#Configuration)
- [Inputs](#Inputs)
- [JSON example](#JSON-example)
- [Environment variables example](#Environment-variables-example)
- [Meltano Example](#Meltano-Example)
- [Usage](#Usage)
- [Executing the Tap Directly](#Executing-the-Tap-Directly)
- [With Meltano](#With-Meltano)
## Installation
```shell
pip install tap-dbt
```
## Features
### Streams
Full stream metadata is available in the dbt Labs repository: [openapi_schema]
#### Selected by default
The following will be extracted by default if no catalog is used:
- [x] Stream: accounts
- [x] Stream: jobs
- [x] Stream: projects
- [x] Stream: runs
#### Configurable
Can be enabled by setting `selected` in the catalog:
- [x] Stream: connections
- [x] Stream: environments
- [x] Stream: repositories
- [x] Stream: users
### Incremental Run Stream
Ordering the query from the Runs endpoint by `-finished_at`, i.e. descending Run Finished Datetime, yields:
|id|finished_at|updated_at|created_at|
|---|---|---|---|
|314516|None|2023-05-27 21:05:16.109548+00:00|2023-05-27 21:05:05.664170+00:00|
|314514|None|2023-05-27 21:00:16.847296+00:00|2023-05-27 21:00:05.458908+00:00|
|314513|None|2023-05-27 21:00:16.355680+00:00|2023-05-27 21:00:05.427258+00:00|
|314517|None|2023-05-27 21:05:17.094309+00:00|2023-05-27 21:05:05.696222+00:00|
|314515|2023-05-27 21:01:28.568431+00:00|2023-05-27 21:01:29.269048+00:00|2023-05-27 21:00:05.488543+00:00|
|314512|2023-05-27 20:48:59.342035+00:00|2023-05-27 20:48:59.844412+00:00|2023-05-27 20:45:04.509746+00:00|
|314511|2023-05-27 20:48:46.571106+00:00|2023-05-27 20:48:47.079130+00:00|2023-05-27 20:40:04.257950+00:00|
|314505|2023-05-27 20:41:35.591976+00:00|2023-05-27 20:41:36.305364+00:00|2023-05-27 20:15:02.808079+00:00|
|314510|2023-05-27 20:39:27.162437+00:00|2023-05-27 20:39:28.628257+00:00|2023-05-27 20:35:03.939439+00:00|
|314509|2023-05-27 20:37:39.965974+00:00|2023-05-27 20:37:40.496212+00:00|2023-05-27 20:30:03.802620+00:00|
The incremental sync has been set up so that it works on `replication_key = "finished_at"`, when an INCREMENTAL sync is run:
- If the bookmark is set, the stream is queried in reverse `finished_at` order.
- If the `finished_at` value is not set, the run is assumed to still be running so the record is included, plus the sort order implies that there should be records with populated `finished_at` appearing later in the stream - *Repeated sync operation will yield the same records if the dbt Job Run is still underway, however this adheres to the 'at least once' delivery promise - https://sdk.meltano.com/en/latest/implementation/at_least_once.html*
- Once the sync operation reaches records with populated `finished_at`, the values are compared with the bookmark and once the `finished_at` value becomes less than the bookmark the stream finishes syncing.
## Configuration
Visit the [API docs][apidocs] for instructions on how to get your API key.
You can pass configuration using environment variables with the `TAP_DBT_` prefix followed by the uppercased field name
```shell
tap-dbt --config=ENV
```
or a JSON file
```shell
tap-dbt --config=config.json
```
### Inputs
| Field | Description | Type | Required | Default |
|---------------|-----------------------------------------------------------------|----------------|----------|--------------------------------------------------|
| `api_key` | API key for the dbt Cloud API | `string` | yes | |
| `account_ids` | dbt Cloud account IDs | `list(string)` | yes | |
| `user_agent` | User-Agent to make requests with | `string` | no | `tap-dbt/0.1.0 Singer Tap for the dbt Cloud API` |
| `base_url` | Base URL for the dbt Cloud API | `string` | no | `https://cloud.getdbt.com/api/v2` |
| `page_size` | Number of records per API call, sets the `limit=` url parameter | `integer` | no | 5000 |
A full list of supported settings and capabilities for this tap is available by running:
```shell
tap-dbt --about --format json
```
### JSON example
```json
{
"api_key": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
"account_ids": ["51341"],
"user_agent": "tap-dbt/0.1.0 Singer Tap for the dbt Cloud API",
"base_url": "https://my-dbt-cloud-api.com",
"page_size": 5000
}
```
### Environment variables example
```dotenv
TAP_DBT_API_KEY=da39a3ee5e6b4b0d3255bfef95601890afd80709
TAP_DBT_ACCOUNT_IDS=51341
TAP_DBT_USER_AGENT='tap-dbt/0.1.0 Singer Tap for the dbt Cloud API'
TAP_DBT_BASE_URL=https://my-dbt-cloud-api.com"
TAP_DBT_PAGE_SIZE=5000
```
### Meltano Example
```yaml
plugins:
extractors:
- name: tap-dbt
logo_url: https://hub.meltano.com/assets/logos/taps/dbt.png
label: dbt Cloud
docs: https://hub.meltano.com/taps/dbt
repo: https://github.com/edgarrmondragon/tap-dbt
namespace: dbt
pip_url: tap-dbt
executable: tap-dbt
capabilities:
- catalog
- discover
settings:
- name: base_url
label: dbt Cloud URL
placeholder: "https://cloud.getdbt.com/api/v2"
- name: api_key
kind: password
label: API Key
docs: "https://docs.getdbt.com/dbt-cloud/api#section/Authentication"
- name: account_ids
kind: array
label: Account IDs
- name: user_agent
label: User-Agent
placeholder: "tap-dbt/0.1.0 Singer Tap for the dbt Cloud API"
- name: page_size
kind: integer
label: Page Size
```
## Usage
You can easily run `tap-dbt` with the CLI or using [Meltano][meltano].
### Executing the Tap Directly
```shell
tap-dbt --version
tap-dbt --help
tap-dbt --config .secrets/example.json --discover > ./catalog/json
```
### With Meltano
```shell
meltano elt tap-dbt target-snowflake --job_id dbt_snowflake
```
[dbtcloud]: https://cloud.getdbt.com
[sdk]: https://gitlab.com/meltano/singer-sdk
[apidocs]: https://docs.getdbt.com/dbt-cloud/api#section/Authentication
[meltano]: https://www.meltano.com
[openapi_schema]: https://github.com/dbt-labs/dbt-cloud-openapi-spec/blob/master/openapi-v3.yaml
Raw data
{
"_id": null,
"home_page": "https://github.com/edgarrmondragon/tap-dbt",
"name": "tap-dbt",
"maintainer": "Edgar Ram\u00edrez Mondrag\u00f3n",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "edgarrm358@sample.com",
"keywords": "singer.io,elt,dbt,singer-sdk",
"author": "Edgar Ram\u00edrez Mondrag\u00f3n",
"author_email": "edgarrm358@sample.com",
"download_url": "https://files.pythonhosted.org/packages/bf/4b/aafb3b97105d54d5d4fbac1a9dab2098a2c56fc1af0ec3515e522eb0cf2a/tap_dbt-0.10.0.tar.gz",
"platform": null,
"description": "# tap-dbt\n\n[](https://pypi.org/project/tap-dbt/)\n[](https://pypi.org/project/tap-dbt/)\n[](https://hub.meltano.com/taps/dbt)\n[](https://github.com/MeltanoLabs/tap-dbt/actions/workflows/ci.yml)\n\n`tap-dbt` is a Singer tap for the [dbt Cloud API v2][dbtcloud].\n\nBuilt with the [Singer SDK][sdk].\n\n- [Installation](#Installation)\n- [Features](#Features)\n- [Configuration](#Configuration)\n - [Inputs](#Inputs)\n - [JSON example](#JSON-example)\n - [Environment variables example](#Environment-variables-example)\n - [Meltano Example](#Meltano-Example)\n- [Usage](#Usage)\n - [Executing the Tap Directly](#Executing-the-Tap-Directly)\n - [With Meltano](#With-Meltano)\n\n## Installation\n\n```shell\npip install tap-dbt\n```\n\n## Features\n\n### Streams\n\nFull stream metadata is available in the dbt Labs repository: [openapi_schema]\n\n#### Selected by default\nThe following will be extracted by default if no catalog is used:\n\n- [x] Stream: accounts\n- [x] Stream: jobs\n- [x] Stream: projects\n- [x] Stream: runs\n\n\n#### Configurable\nCan be enabled by setting `selected` in the catalog:\n\n- [x] Stream: connections\n- [x] Stream: environments\n- [x] Stream: repositories\n- [x] Stream: users\n\n\n### Incremental Run Stream\n\nOrdering the query from the Runs endpoint by `-finished_at`, i.e. descending Run Finished Datetime, yields:\n\n|id|finished_at|updated_at|created_at|\n|---|---|---|---|\n|314516|None|2023-05-27 21:05:16.109548+00:00|2023-05-27 21:05:05.664170+00:00|\n|314514|None|2023-05-27 21:00:16.847296+00:00|2023-05-27 21:00:05.458908+00:00|\n|314513|None|2023-05-27 21:00:16.355680+00:00|2023-05-27 21:00:05.427258+00:00|\n|314517|None|2023-05-27 21:05:17.094309+00:00|2023-05-27 21:05:05.696222+00:00|\n|314515|2023-05-27 21:01:28.568431+00:00|2023-05-27 21:01:29.269048+00:00|2023-05-27 21:00:05.488543+00:00|\n|314512|2023-05-27 20:48:59.342035+00:00|2023-05-27 20:48:59.844412+00:00|2023-05-27 20:45:04.509746+00:00|\n|314511|2023-05-27 20:48:46.571106+00:00|2023-05-27 20:48:47.079130+00:00|2023-05-27 20:40:04.257950+00:00|\n|314505|2023-05-27 20:41:35.591976+00:00|2023-05-27 20:41:36.305364+00:00|2023-05-27 20:15:02.808079+00:00|\n|314510|2023-05-27 20:39:27.162437+00:00|2023-05-27 20:39:28.628257+00:00|2023-05-27 20:35:03.939439+00:00|\n|314509|2023-05-27 20:37:39.965974+00:00|2023-05-27 20:37:40.496212+00:00|2023-05-27 20:30:03.802620+00:00|\n\nThe incremental sync has been set up so that it works on `replication_key = \"finished_at\"`, when an INCREMENTAL sync is run:\n\n- If the bookmark is set, the stream is queried in reverse `finished_at` order.\n- If the `finished_at` value is not set, the run is assumed to still be running so the record is included, plus the sort order implies that there should be records with populated `finished_at` appearing later in the stream - *Repeated sync operation will yield the same records if the dbt Job Run is still underway, however this adheres to the 'at least once' delivery promise - https://sdk.meltano.com/en/latest/implementation/at_least_once.html*\n- Once the sync operation reaches records with populated `finished_at`, the values are compared with the bookmark and once the `finished_at` value becomes less than the bookmark the stream finishes syncing.\n\n\n## Configuration\n\nVisit the [API docs][apidocs] for instructions on how to get your API key.\n\nYou can pass configuration using environment variables with the `TAP_DBT_` prefix followed by the uppercased field name\n\n```shell\ntap-dbt --config=ENV\n```\n\nor a JSON file\n\n```shell\ntap-dbt --config=config.json\n```\n\n### Inputs\n\n| Field | Description | Type | Required | Default |\n|---------------|-----------------------------------------------------------------|----------------|----------|--------------------------------------------------|\n| `api_key` | API key for the dbt Cloud API | `string` | yes | |\n| `account_ids` | dbt Cloud account IDs | `list(string)` | yes | |\n| `user_agent` | User-Agent to make requests with | `string` | no | `tap-dbt/0.1.0 Singer Tap for the dbt Cloud API` |\n| `base_url` | Base URL for the dbt Cloud API | `string` | no | `https://cloud.getdbt.com/api/v2` |\n| `page_size` | Number of records per API call, sets the `limit=` url parameter | `integer` | no | 5000 |\n\nA full list of supported settings and capabilities for this tap is available by running:\n\n```shell\ntap-dbt --about --format json\n```\n\n### JSON example\n\n```json\n{\n \"api_key\": \"da39a3ee5e6b4b0d3255bfef95601890afd80709\",\n \"account_ids\": [\"51341\"],\n \"user_agent\": \"tap-dbt/0.1.0 Singer Tap for the dbt Cloud API\",\n \"base_url\": \"https://my-dbt-cloud-api.com\",\n \"page_size\": 5000\n}\n```\n\n### Environment variables example\n\n```dotenv\nTAP_DBT_API_KEY=da39a3ee5e6b4b0d3255bfef95601890afd80709\nTAP_DBT_ACCOUNT_IDS=51341\nTAP_DBT_USER_AGENT='tap-dbt/0.1.0 Singer Tap for the dbt Cloud API'\nTAP_DBT_BASE_URL=https://my-dbt-cloud-api.com\"\nTAP_DBT_PAGE_SIZE=5000\n```\n\n### Meltano Example\n\n```yaml\nplugins:\n extractors:\n - name: tap-dbt\n logo_url: https://hub.meltano.com/assets/logos/taps/dbt.png\n label: dbt Cloud\n docs: https://hub.meltano.com/taps/dbt\n repo: https://github.com/edgarrmondragon/tap-dbt\n namespace: dbt\n pip_url: tap-dbt\n executable: tap-dbt\n capabilities:\n - catalog\n - discover\n settings:\n - name: base_url\n label: dbt Cloud URL\n placeholder: \"https://cloud.getdbt.com/api/v2\"\n - name: api_key\n kind: password\n label: API Key\n docs: \"https://docs.getdbt.com/dbt-cloud/api#section/Authentication\"\n - name: account_ids\n kind: array\n label: Account IDs\n - name: user_agent\n label: User-Agent\n placeholder: \"tap-dbt/0.1.0 Singer Tap for the dbt Cloud API\"\n - name: page_size\n kind: integer\n label: Page Size\n\n```\n\n## Usage\n\nYou can easily run `tap-dbt` with the CLI or using [Meltano][meltano].\n\n### Executing the Tap Directly\n\n```shell\ntap-dbt --version\ntap-dbt --help\ntap-dbt --config .secrets/example.json --discover > ./catalog/json\n```\n\n### With Meltano\n\n```shell\nmeltano elt tap-dbt target-snowflake --job_id dbt_snowflake\n```\n\n[dbtcloud]: https://cloud.getdbt.com\n[sdk]: https://gitlab.com/meltano/singer-sdk\n[apidocs]: https://docs.getdbt.com/dbt-cloud/api#section/Authentication\n[meltano]: https://www.meltano.com\n[openapi_schema]: https://github.com/dbt-labs/dbt-cloud-openapi-spec/blob/master/openapi-v3.yaml\n\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Singer tap for dbt, built with the Singer SDK.",
"version": "0.10.0",
"project_urls": {
"Homepage": "https://github.com/edgarrmondragon/tap-dbt",
"Repository": "https://github.com/edgarrmondragon/tap-dbt"
},
"split_keywords": [
"singer.io",
"elt",
"dbt",
"singer-sdk"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "24233acae80605a3d4e01f6302719f9bacd7d299081f933639c5b1830860eb4b",
"md5": "31aec9cc166d1e25eb7318cf3a20bc08",
"sha256": "f6151a0340e9c3ea77bd73ef055e67ac3b52105bd35bb70d1fee8f5ee8844cfc"
},
"downloads": -1,
"filename": "tap_dbt-0.10.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "31aec9cc166d1e25eb7318cf3a20bc08",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 15159,
"upload_time": "2024-02-05T17:56:10",
"upload_time_iso_8601": "2024-02-05T17:56:10.763643Z",
"url": "https://files.pythonhosted.org/packages/24/23/3acae80605a3d4e01f6302719f9bacd7d299081f933639c5b1830860eb4b/tap_dbt-0.10.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "bf4baafb3b97105d54d5d4fbac1a9dab2098a2c56fc1af0ec3515e522eb0cf2a",
"md5": "32ce287cb555b876974d1d4f6b674438",
"sha256": "42def60a5cd45d31b6c586c12d6f0035a6601d61e72071338faec173c97e5af8"
},
"downloads": -1,
"filename": "tap_dbt-0.10.0.tar.gz",
"has_sig": false,
"md5_digest": "32ce287cb555b876974d1d4f6b674438",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 15085,
"upload_time": "2024-02-05T17:56:12",
"upload_time_iso_8601": "2024-02-05T17:56:12.013776Z",
"url": "https://files.pythonhosted.org/packages/bf/4b/aafb3b97105d54d5d4fbac1a9dab2098a2c56fc1af0ec3515e522eb0cf2a/tap_dbt-0.10.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-05 17:56:12",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "edgarrmondragon",
"github_project": "tap-dbt",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "tap-dbt"
}
JSON
