# dbt-cloud-jobs
[](https://img.shields.io/pypi/v/dbt-cloud-jobs)
[](https://opensource.org/licenses/MIT)
Version control your dbt Cloud jobs with YML.
# Installation
```bash
pip install dbt-cloud-jobs
```
# Quickstart
1. Create an API token in dbt Cloud:
- [Generate a service account token](https://docs.getdbt.com/docs/dbt-cloud-apis/service-tokens#generate-service-account-tokens), make sure to grant the `Jobs Admin` permission set.
- If you do not have access to create a service account token you can create a [User API token](https://docs.getdbt.com/docs/dbt-cloud-apis/user-tokens). Note that the service account method is preferred.
1. Set an environment variable with the value of the token:
```bash
export DBT_API_TOKEN="<VALUE_FROM_PREVIOUS_STEP>"
```
1. Set an environment variable for the region where your dbt Cloud account is hosted. The value must be one of "AU", "Europe" or "US" (see docs [here](https://docs.getdbt.com/dbt-cloud/api-v2#/)):
```bash
export DBT_CLOUD_REGION="<REGION>"
```
1. Import your existing dbt Cloud jobs:
```bash
dbt_cloud_jobs --import --account-id 123456 --file dbt_cloud_jobs.yml
```
1. Edit the definition of your jobs in `dbt_cloud_jobs.yml`.
1. Sync the updated definitions to dbt Cloud:
```bash
dbt_cloud_jobs --sync --file dbt_cloud_jobs.yml
```
# Recommended usage in CI/CD
## CI
In CI `dbt-cloud-jobs` should be used to verify that the provided YML file is valid. For example:
```yaml
- name: Install dbt_cloud_jobs
run: pip install dbt-cloud-jobs
- name: Validate `dbt_cloud_jobs.yml`
run: dbt_cloud_jobs --validate --file dbt_cloud_jobs.yml
```
## CD
In CD `dbt-cloud-jobs` should be used to sync the provided YML file to dbt Cloud. For example:
```yaml
- name: Install dbt_cloud_jobs
run: pip install dbt-cloud-jobs
- name: Sync `dbt_cloud_jobs.yml`
run: dbt_cloud_jobs --sync --file dbt_cloud_jobs.yml
```
## Example
For an example of how this package can be used, take a look at [`dbt-cloud-jobs-example-repo`](https://github.com/pgoslatara/dbt-cloud-jobs-example-repo), specifically:
* The [`dbt_cloud_jobs.yml`](https://github.com/pgoslatara/dbt-cloud-jobs-example-repo/blob/main/dbt_cloud_jobs.yml) file.
* The [CI pipeline](https://github.com/pgoslatara/dbt-cloud-jobs-example-repo/blob/main/.github/workflows/ci_pipeline.yml).
* The [CD pipeline](https://github.com/pgoslatara/dbt-cloud-jobs-example-repo/blob/main/.github/workflows/cd_pipeline.yml).
* This [CI pipeline](https://github.com/pgoslatara/dbt-cloud-jobs-example-repo/actions/runs/8238754815/job/22530416583) run that validated the jobs definitions added in [PR1](https://github.com/pgoslatara/dbt-cloud-jobs-example-repo/pull/1).
* This [CD pipeline](https://github.com/pgoslatara/dbt-cloud-jobs-example-repo/actions/runs/8238763252/job/22530445750) run that updated the `Daily job` dbt Cloud job.
# Limitations/Warnings
* Service account tokens are created at the account level. This means that if you have multiple dbt Cloud accounts you will need to create different `dbt_cloud_jobs.yml` files for each account. If you try to use `dbt-cloud-jobs` with a file that contains multiple `account_id` values, an error will be raised.
* ⚠️ `dbt_cloud_jobs` expects to "own" all the jobs in the projects where it is used. This means that if you are running `dbt_cloud_jobs --sync --file dbt_cloud_jobs.yml` for the first time in a project, any pre-existing jobs not present in your `dbt_cloud_jobs.yml` file will be deleted.
# Development
To setup your development environment, fork this repository and run:
```bash
poetry shell
poetry install
```
Set the following environment variables:
```bash
export DBT_ACCOUNT_ID=<DBT_ACCOUNT_ID>
export DBT_CLOUD_REGION="<DBT_CLOUD_REGION>"
export DBT_ENVIRONMENT_ID=<DBT_ENVIRONMENT_ID>
export DBT_PROJECT_ID=<DBT_PROJECT_ID>
export DBT_API_TOKEN="<DBT_API_TOKEN>"
```
It is highly recommended that a dedicated dbt Cloud environment be used for development.
All tests can be run via:
```bash
make test
```
# Release
Trigger the `Publish to PyPi` workflow, inputting the version to publish to PyPi. This workflow will:
- Publish the version to [PyPi](https://pypi.org/project/dbt-cloud-jobs/).
- Tag the HEAD commit of the `main` branches (tags visible [here](https://github.com/pgoslatara/dbt-cloud-jobs/tags)).
- Create a release (releases visible [here](https://github.com/pgoslatara/dbt-cloud-jobs/releases)).
Raw data
{
"_id": null,
"home_page": "https://github.com/pgoslatara/dbt-cloud-jobs",
"name": "dbt-cloud-jobs",
"maintainer": "Padraic Slattery",
"docs_url": null,
"requires_python": "<3.13,>=3.8",
"maintainer_email": "pgoslatara@gmail.com",
"keywords": "python, cli, dbt, CI/CD",
"author": "Padraic Slattery",
"author_email": "pgoslatara@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/45/99/10316c434dc78afb380379dff5bcc171344135ccdbdfdc22b7b18ec800cd/dbt_cloud_jobs-0.1.0.tar.gz",
"platform": null,
"description": "# dbt-cloud-jobs\n\n[](https://img.shields.io/pypi/v/dbt-cloud-jobs)\n\n\n\n[](https://opensource.org/licenses/MIT)\n\nVersion control your dbt Cloud jobs with YML.\n\n# Installation\n\n```bash\npip install dbt-cloud-jobs\n```\n\n# Quickstart\n\n1. Create an API token in dbt Cloud:\n\n - [Generate a service account token](https://docs.getdbt.com/docs/dbt-cloud-apis/service-tokens#generate-service-account-tokens), make sure to grant the `Jobs Admin` permission set.\n - If you do not have access to create a service account token you can create a [User API token](https://docs.getdbt.com/docs/dbt-cloud-apis/user-tokens). Note that the service account method is preferred.\n\n1. Set an environment variable with the value of the token:\n\n ```bash\n export DBT_API_TOKEN=\"<VALUE_FROM_PREVIOUS_STEP>\"\n ```\n\n1. Set an environment variable for the region where your dbt Cloud account is hosted. The value must be one of \"AU\", \"Europe\" or \"US\" (see docs [here](https://docs.getdbt.com/dbt-cloud/api-v2#/)):\n\n ```bash\n export DBT_CLOUD_REGION=\"<REGION>\"\n ```\n\n1. Import your existing dbt Cloud jobs:\n\n ```bash\n dbt_cloud_jobs --import --account-id 123456 --file dbt_cloud_jobs.yml\n ```\n\n1. Edit the definition of your jobs in `dbt_cloud_jobs.yml`.\n\n1. Sync the updated definitions to dbt Cloud:\n\n ```bash\n dbt_cloud_jobs --sync --file dbt_cloud_jobs.yml\n ```\n\n# Recommended usage in CI/CD\n\n## CI\n\nIn CI `dbt-cloud-jobs` should be used to verify that the provided YML file is valid. For example:\n\n```yaml\n - name: Install dbt_cloud_jobs\n run: pip install dbt-cloud-jobs\n\n - name: Validate `dbt_cloud_jobs.yml`\n run: dbt_cloud_jobs --validate --file dbt_cloud_jobs.yml\n```\n\n## CD\n\nIn CD `dbt-cloud-jobs` should be used to sync the provided YML file to dbt Cloud. For example:\n\n```yaml\n - name: Install dbt_cloud_jobs\n run: pip install dbt-cloud-jobs\n\n - name: Sync `dbt_cloud_jobs.yml`\n run: dbt_cloud_jobs --sync --file dbt_cloud_jobs.yml\n```\n\n## Example\n\nFor an example of how this package can be used, take a look at [`dbt-cloud-jobs-example-repo`](https://github.com/pgoslatara/dbt-cloud-jobs-example-repo), specifically:\n\n* The [`dbt_cloud_jobs.yml`](https://github.com/pgoslatara/dbt-cloud-jobs-example-repo/blob/main/dbt_cloud_jobs.yml) file.\n* The [CI pipeline](https://github.com/pgoslatara/dbt-cloud-jobs-example-repo/blob/main/.github/workflows/ci_pipeline.yml).\n* The [CD pipeline](https://github.com/pgoslatara/dbt-cloud-jobs-example-repo/blob/main/.github/workflows/cd_pipeline.yml).\n* This [CI pipeline](https://github.com/pgoslatara/dbt-cloud-jobs-example-repo/actions/runs/8238754815/job/22530416583) run that validated the jobs definitions added in [PR1](https://github.com/pgoslatara/dbt-cloud-jobs-example-repo/pull/1).\n* This [CD pipeline](https://github.com/pgoslatara/dbt-cloud-jobs-example-repo/actions/runs/8238763252/job/22530445750) run that updated the `Daily job` dbt Cloud job.\n\n# Limitations/Warnings\n\n* Service account tokens are created at the account level. This means that if you have multiple dbt Cloud accounts you will need to create different `dbt_cloud_jobs.yml` files for each account. If you try to use `dbt-cloud-jobs` with a file that contains multiple `account_id` values, an error will be raised.\n\n* \u26a0\ufe0f `dbt_cloud_jobs` expects to \"own\" all the jobs in the projects where it is used. This means that if you are running `dbt_cloud_jobs --sync --file dbt_cloud_jobs.yml` for the first time in a project, any pre-existing jobs not present in your `dbt_cloud_jobs.yml` file will be deleted.\n\n# Development\n\nTo setup your development environment, fork this repository and run:\n\n```bash\npoetry shell\npoetry install\n```\n\nSet the following environment variables:\n```bash\nexport DBT_ACCOUNT_ID=<DBT_ACCOUNT_ID>\nexport DBT_CLOUD_REGION=\"<DBT_CLOUD_REGION>\"\nexport DBT_ENVIRONMENT_ID=<DBT_ENVIRONMENT_ID>\nexport DBT_PROJECT_ID=<DBT_PROJECT_ID>\nexport DBT_API_TOKEN=\"<DBT_API_TOKEN>\"\n```\nIt is highly recommended that a dedicated dbt Cloud environment be used for development.\n\nAll tests can be run via:\n```bash\nmake test\n```\n\n# Release\n\nTrigger the `Publish to PyPi` workflow, inputting the version to publish to PyPi. This workflow will:\n\n- Publish the version to [PyPi](https://pypi.org/project/dbt-cloud-jobs/).\n- Tag the HEAD commit of the `main` branches (tags visible [here](https://github.com/pgoslatara/dbt-cloud-jobs/tags)).\n- Create a release (releases visible [here](https://github.com/pgoslatara/dbt-cloud-jobs/releases)).\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Version control your dbt Cloud jobs with YML.",
"version": "0.1.0",
"project_urls": {
"Homepage": "https://github.com/pgoslatara/dbt-cloud-jobs",
"Repository": "https://github.com/pgoslatara/dbt-cloud-jobs"
},
"split_keywords": [
"python",
" cli",
" dbt",
" ci/cd"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "12f090cd978928981d2db80b1e094d49263c9ea749d72313a24fb7c72bb5da19",
"md5": "8203b14d2213d50394e6fc9b7f7a3e97",
"sha256": "3b8cac90e067e9290e196de68a9d42546c2ed58a293a6b9643214c4c464dc2dd"
},
"downloads": -1,
"filename": "dbt_cloud_jobs-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "8203b14d2213d50394e6fc9b7f7a3e97",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.13,>=3.8",
"size": 13396,
"upload_time": "2024-06-23T18:46:15",
"upload_time_iso_8601": "2024-06-23T18:46:15.703927Z",
"url": "https://files.pythonhosted.org/packages/12/f0/90cd978928981d2db80b1e094d49263c9ea749d72313a24fb7c72bb5da19/dbt_cloud_jobs-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "459910316c434dc78afb380379dff5bcc171344135ccdbdfdc22b7b18ec800cd",
"md5": "1792c5b36e2a26fc7de6431f26fc643d",
"sha256": "3e515cecd37300e1759631b8a53d7b55d7e11d9bfff60fb34b0a7b3ea685b448"
},
"downloads": -1,
"filename": "dbt_cloud_jobs-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "1792c5b36e2a26fc7de6431f26fc643d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.13,>=3.8",
"size": 11982,
"upload_time": "2024-06-23T18:46:17",
"upload_time_iso_8601": "2024-06-23T18:46:17.222811Z",
"url": "https://files.pythonhosted.org/packages/45/99/10316c434dc78afb380379dff5bcc171344135ccdbdfdc22b7b18ec800cd/dbt_cloud_jobs-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-06-23 18:46:17",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "pgoslatara",
"github_project": "dbt-cloud-jobs",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "dbt-cloud-jobs"
}
JSON
