| Name | dot-for-dbt JSON |
| Version |
0.3.0
JSON |
| download |
| home_page | None |
| Summary | dot - the Data Orchestration Tool (for dbt) |
| upload_time | 2025-09-07 10:50:45 |
| maintainer | None |
| docs_url | None |
| author | Adam MacLeod |
| requires_python | >=3.12 |
| license | MIT License Copyright (c) 2025 Adam MacLeod (adam@macleod.id.au) Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice (including the next paragraph) shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
analytics
cli
data
dbt
orchestration
reproducibility
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# The Data Orchestration Tool for dbt (dot-for-dbt)
`dot` is a lightweight companion CLI for dbt that lets you run any dbt command for an optional named environment and an exact git commit/ref using the shorthand `<environment?>@<ref>`. Adding `@<ref>` builds that historical version into a schema automatically suffixed with the commit’s short hash (e.g. `analytics_a1b2c3d4`) so your current schemas stay untouched. This enables reproducible historical builds, safe experimentation, side‑by‑side diffing, and confident migration or release validation.
## Installation
Requires Python >= 3.12 (resolved automatically by `uv`).
Using `uv` (recommended persistent install):
```bash
uv tool install dot-for-dbt
dot --help
```
Upgrade:
```bash
uv tool upgrade dot-for-dbt
```
Ephemeral run (no global install):
```bash
uvx dot-for-dbt --help
```
Pin version:
```bash
uv tool install dot-for-dbt==0.1.1
uvx dot-for-dbt==0.1.1 --help
```
Via `pip` (alternative):
```bash
pip install dot-for-dbt
```
Uninstall:
```bash
uv tool uninstall dot-for-dbt
```
## Quick Example
```bash
# Build current project using default environment
dot build
# Build historical commit (isolated schema)
dot build @abc1234
# Build specific environment at a ref
dot run dev@feature/my-branch
```
## CLI Usage
Basic usage:
```sh
dot <dbt_command> <environment> [--dry-run] [--no-gitignore-check]
```
- `<dbt_command>` is any supported dbt command (e.g., build, run, test).
- `<environment>` (Optional) is the environment which you want to target as defined in your `vars.yml` under the top-level `environment:` key. If you do not specify an environment, the default environment from `vars.yml` will be used.
To build or run against a specific git commit in an isolated schema, append `@<gitref or commit>` to the environment:
```sh
dot <dbt_command> <environment>@<gitref or commit>
```
You can also build into the default environment at a certain commit:
```sh
dot <dbt_command> @<gitref or commit>
```
This will check out the specified commit in a git worktree, generate a dedicated `profiles.yml`, and build into `yourschema_<short git hash>`. This enables reproducible, isolated builds for any point in your repository history.
## .gitignore Requirement
The `.dot` directory contains build artifacts and must be ignored by git. By default, the CLI enforces this by checking for a `.dot/` entry in your `.gitignore` file before running any commands. If missing, you will be prompted to add it automatically. The CLI will refuse to run if `.dot/` is not ignored.
To bypass this enforcement (not recommended for normal use), use the `--no-gitignore-check` flag:
```
dot build --no-gitignore-check
```
To ensure correct setup, add the following line to your `.gitignore`:
```
.dot/
```
## vars.yml Behavior
Top-level structure (relevant parts):
```yaml
vars:
...
environment:
default: dev
all:
indirect-selection: buildable
dev:
target: dev
vars:
some_var: true
prod:
target: prod
vars:
some_var: false
```
Rules:
- `vars.yml` is optional. If it does not exist in your working directory, dot will proceed with default settings and no environment-based variables.
- If `vars.yml` exists but is malformed (invalid YAML), dot will print an error and exit.
- If you specify an environment that does not exist in `vars.yml`, dot will print an error and exit.
- If no environment is specified and no default is set in `vars.yml`, dot will proceed with default settings.
## Isolated Builds
Isolated builds let you execute a dbt command against the exact contents of any git commit (or ref) in a clean, temporary worktree while writing all database objects into a schema that is namespaced by the commit hash. This provides:
- Reproducibility (build exactly what existed at that commit)
- Confidence to roll forward/back by inspecting isolated artifacts
Future features are planned to make more extensive use of isolated builds.
### Quick Start
To build using the default environment specified in `vars.yml` at a particular historical git reference, simply omit the environment and use `@<ref>`:
```sh
dot build @abc1234
```
Build an explicit environment against a ref:
```sh
dot run dev@feature/my-branch
dot test prod@v1.2.0
```
Build using a short or symbolic ref (branch, tag, HEAD~N, etc.):
```sh
dot run dev@HEAD~1
dot build prod@main
```
### Syntax Summary
```
<environment?>@<gitref>
```
- `environment` (optional) — name defined under `environment` in `vars.yml`
- `gitref` (optional) — branch, tag, full/short hash, reflog expression, etc.
- If `@<gitref>` is supplied with no leading environment, the default environment is used.
- If no `@` suffix is provided, this is a normal (non‑isolated) build against the current state of your project.
### What Happens Internally
1. Resolve `<gitref>` to the full 40‑char commit hash and the abbreviated short hash via:
- `git rev-parse <gitref>`
- `git rev-parse --short <gitref>`
2. Construct: `.dot/build/<short_hash>/`
3. Create (or reuse) a clean git worktree at:
```
.dot/build/<short_hash>/worktree/
```
4. Locate the dbt project inside that worktree matching the original project path.
5. Detect the active `profiles.yml` location (`dbt debug --config-dir`).
6. Read the selected profile + target (environment name).
7. Write an isolated `profiles.yml` to:
```
.dot/build/<short_hash>/env/<environment>/profiles.yml
```
with the target schema updated to `<schema>_<short_hash>`.
8. Set dbt CLI args so that:
- `--project-dir` points at the isolated worktree project
- `--profiles-dir` points at `.dot/build/<short_hash>/env/<environment>`
- `--target-path` is `.dot/build/<short_hash>/env/<environment>/target`
- `--log-path` is `.dot/build/<short_hash>/env/<environment>/logs`
9. Write the full hash to:
```
.dot/build/<short_hash>/commit
```
10. Execute the dbt command.
### Schema Naming
The target schema becomes:
```
<original_schema>_<short_hash>
```
Where `<short_hash>` is the abbreviated commit hash reported by `git rev-parse --short <ref>` (length chosen automatically by git to avoid ambiguity). For example, if your original target schema is `analytics` and the short hash is `6b777b8c`, the isolated schema is:
```
analytics_6b777b8c
```
### Directory Layout
Example layout for an isolated build:
```
.dot/
build/
<short_hash>/ # Directory keyed by abbreviated hash
worktree/ # Clean checkout at that commit
commit # File containing full 40-char commit hash
env/ # Parent directory for all environment-specific artifacts
dev/
profiles.yml # Auto-generated, schema rewritten with _<short_hash>
target/ # dbt artifacts (manifest, run results, etc.)
logs/ # dbt logs for this isolated run
prod/
profiles.yml
target/
logs/
```
If you build multiple environments (`dev`, `prod`) for the same commit, each gets its own environment subdirectory under `env/`.
### Examples
Diff models between current development and a feature branch:
```sh
dot build dev
dot build dev@feature/new-metric
# Compare artifacts or query both schemas: analytics vs analytics_<short_hash>
```
Test a migration before merging:
```sh
dot run prod@migration/rename-columns
dot test prod@migration/rename-columns
```
Roll forward validation (red/green):
```sh
dot build prod@current_prod_tag
dot build prod@next_release_candidate
# Validate row counts, constraints, performance before switching consumers
```
Historical investigation:
```sh
dot run dev@2024-12-01-tag
```
### profiles.yml Detection & Rewriting
`dot` invokes `dbt debug --config-dir` (wrapped through its own command builder) to locate the effective `profiles.yml`. It then:
- Loads the user’s configured profile
- Extracts the target matching the active environment
- Updates only the `schema` field (preserving credentials, threads, etc.)
- Writes a minimal isolated `profiles.yml` containing just that profile + target
### Passing Additional dbt Args
Anything after `--` is passed through untouched:
```sh
dot run dev@main -- --select my_model+
```
### Cleanup
Currently there is no automatic cleanup. To reclaim space:
- Drop old schemas manually from your warehouse
- Remove stale directories under `.dot/build/`
Automatic management of old build artifacts and schemas is planned for a future release. Please let me know if this would be important to you!
### Troubleshooting
| Symptom | Cause | Action |
|---------|-------|--------|
| Error: Profile not found | Active environment or profile missing | Verify `profiles.yml` and environment name |
| Commit not found | Bad ref | Run `git show<ref>` to validate |
| Schema clutter | Many builds kept | Periodically prune `.dot/build` and drop old schemas |
| Wrong default environment | `environment.default` unset or unexpected | Set `default` under `environment` in `vars.yml` |
### Reference
For architectural rationale see: [ADR 0001: Isolated Builds](adr/0001-isolated-builds.md).
## Architectural Decision Records
Architectural decisions are documented in the [adr/](adr/) directory.
- [ADR 0001: Isolated Builds](adr/0001-isolated-builds.md)
## Changelog
See [CHANGELOG.md](CHANGELOG.md) for released versions.
## License
This project is licensed under the MIT License. See the `LICENSE` file for full details.
SPDX-License-Identifier: MIT
Raw data
{
"_id": null,
"home_page": null,
"name": "dot-for-dbt",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.12",
"maintainer_email": null,
"keywords": "analytics, cli, data, dbt, orchestration, reproducibility",
"author": "Adam MacLeod",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/6a/4b/41de8ba87eaaf36028f1ab099eb450fdfaa51c3042267a01e381bf3deb6b/dot_for_dbt-0.3.0.tar.gz",
"platform": null,
"description": "# The Data Orchestration Tool for dbt (dot-for-dbt)\n\n`dot` is a lightweight companion CLI for dbt that lets you run any dbt command for an optional named environment and an exact git commit/ref using the shorthand `<environment?>@<ref>`. Adding `@<ref>` builds that historical version into a schema automatically suffixed with the commit\u2019s short hash (e.g. `analytics_a1b2c3d4`) so your current schemas stay untouched. This enables reproducible historical builds, safe experimentation, side\u2011by\u2011side diffing, and confident migration or release validation.\n\n## Installation\n\nRequires Python >= 3.12 (resolved automatically by `uv`).\n\nUsing `uv` (recommended persistent install):\n```bash\nuv tool install dot-for-dbt\ndot --help\n```\n\nUpgrade:\n```bash\nuv tool upgrade dot-for-dbt\n```\n\nEphemeral run (no global install):\n```bash\nuvx dot-for-dbt --help\n```\n\nPin version:\n```bash\nuv tool install dot-for-dbt==0.1.1\nuvx dot-for-dbt==0.1.1 --help\n```\n\nVia `pip` (alternative):\n```bash\npip install dot-for-dbt\n```\n\nUninstall:\n```bash\nuv tool uninstall dot-for-dbt\n```\n\n## Quick Example\n\n```bash\n# Build current project using default environment\ndot build\n\n# Build historical commit (isolated schema)\ndot build @abc1234\n\n# Build specific environment at a ref\ndot run dev@feature/my-branch\n```\n\n## CLI Usage\n\nBasic usage:\n\n```sh\ndot <dbt_command> <environment> [--dry-run] [--no-gitignore-check]\n```\n\n- `<dbt_command>` is any supported dbt command (e.g., build, run, test).\n- `<environment>` (Optional) is the environment which you want to target as defined in your `vars.yml` under the top-level `environment:` key. If you do not specify an environment, the default environment from `vars.yml` will be used.\n\nTo build or run against a specific git commit in an isolated schema, append `@<gitref or commit>` to the environment:\n\n```sh\ndot <dbt_command> <environment>@<gitref or commit>\n```\n\nYou can also build into the default environment at a certain commit:\n\n```sh\ndot <dbt_command> @<gitref or commit>\n```\n\nThis will check out the specified commit in a git worktree, generate a dedicated `profiles.yml`, and build into `yourschema_<short git hash>`. This enables reproducible, isolated builds for any point in your repository history.\n\n## .gitignore Requirement\n\nThe `.dot` directory contains build artifacts and must be ignored by git. By default, the CLI enforces this by checking for a `.dot/` entry in your `.gitignore` file before running any commands. If missing, you will be prompted to add it automatically. The CLI will refuse to run if `.dot/` is not ignored.\n\nTo bypass this enforcement (not recommended for normal use), use the `--no-gitignore-check` flag:\n\n```\ndot build --no-gitignore-check\n```\n\nTo ensure correct setup, add the following line to your `.gitignore`:\n\n```\n.dot/\n```\n\n## vars.yml Behavior\n\nTop-level structure (relevant parts):\n\n```yaml\nvars:\n ...\n\nenvironment:\n default: dev\n all:\n indirect-selection: buildable\n dev:\n target: dev\n vars:\n some_var: true\n prod:\n target: prod\n vars:\n some_var: false\n```\n\nRules:\n\n- `vars.yml` is optional. If it does not exist in your working directory, dot will proceed with default settings and no environment-based variables.\n- If `vars.yml` exists but is malformed (invalid YAML), dot will print an error and exit.\n- If you specify an environment that does not exist in `vars.yml`, dot will print an error and exit.\n- If no environment is specified and no default is set in `vars.yml`, dot will proceed with default settings.\n\n## Isolated Builds\n\nIsolated builds let you execute a dbt command against the exact contents of any git commit (or ref) in a clean, temporary worktree while writing all database objects into a schema that is namespaced by the commit hash. This provides:\n\n- Reproducibility (build exactly what existed at that commit)\n- Confidence to roll forward/back by inspecting isolated artifacts\n\nFuture features are planned to make more extensive use of isolated builds.\n\n### Quick Start\n\nTo build using the default environment specified in `vars.yml` at a particular historical git reference, simply omit the environment and use `@<ref>`:\n\n```sh\ndot build @abc1234\n```\n\nBuild an explicit environment against a ref:\n```sh\ndot run dev@feature/my-branch\ndot test prod@v1.2.0\n```\n\nBuild using a short or symbolic ref (branch, tag, HEAD~N, etc.):\n```sh\ndot run dev@HEAD~1\ndot build prod@main\n```\n\n### Syntax Summary\n\n```\n<environment?>@<gitref>\n```\n- `environment` (optional) \u2014 name defined under `environment` in `vars.yml`\n- `gitref` (optional) \u2014 branch, tag, full/short hash, reflog expression, etc.\n- If `@<gitref>` is supplied with no leading environment, the default environment is used.\n- If no `@` suffix is provided, this is a normal (non\u2011isolated) build against the current state of your project.\n\n### What Happens Internally\n\n1. Resolve `<gitref>` to the full 40\u2011char commit hash and the abbreviated short hash via:\n - `git rev-parse <gitref>`\n - `git rev-parse --short <gitref>`\n2. Construct: `.dot/build/<short_hash>/`\n3. Create (or reuse) a clean git worktree at:\n ```\n .dot/build/<short_hash>/worktree/\n ```\n4. Locate the dbt project inside that worktree matching the original project path.\n5. Detect the active `profiles.yml` location (`dbt debug --config-dir`).\n6. Read the selected profile + target (environment name).\n7. Write an isolated `profiles.yml` to:\n ```\n .dot/build/<short_hash>/env/<environment>/profiles.yml\n ```\n with the target schema updated to `<schema>_<short_hash>`.\n8. Set dbt CLI args so that:\n - `--project-dir` points at the isolated worktree project\n - `--profiles-dir` points at `.dot/build/<short_hash>/env/<environment>`\n - `--target-path` is `.dot/build/<short_hash>/env/<environment>/target`\n - `--log-path` is `.dot/build/<short_hash>/env/<environment>/logs`\n9. Write the full hash to:\n ```\n .dot/build/<short_hash>/commit\n ```\n10. Execute the dbt command.\n\n### Schema Naming\n\nThe target schema becomes:\n\n```\n<original_schema>_<short_hash>\n```\n\nWhere `<short_hash>` is the abbreviated commit hash reported by `git rev-parse --short <ref>` (length chosen automatically by git to avoid ambiguity). For example, if your original target schema is `analytics` and the short hash is `6b777b8c`, the isolated schema is:\n\n```\nanalytics_6b777b8c\n```\n\n### Directory Layout\n\nExample layout for an isolated build:\n\n```\n.dot/\n build/\n <short_hash>/ # Directory keyed by abbreviated hash\n worktree/ # Clean checkout at that commit\n commit # File containing full 40-char commit hash\n env/ # Parent directory for all environment-specific artifacts\n dev/\n profiles.yml # Auto-generated, schema rewritten with _<short_hash>\n target/ # dbt artifacts (manifest, run results, etc.)\n logs/ # dbt logs for this isolated run\n prod/\n profiles.yml\n target/\n logs/\n```\n\nIf you build multiple environments (`dev`, `prod`) for the same commit, each gets its own environment subdirectory under `env/`.\n\n### Examples\n\nDiff models between current development and a feature branch:\n```sh\ndot build dev\ndot build dev@feature/new-metric\n# Compare artifacts or query both schemas: analytics vs analytics_<short_hash>\n```\n\nTest a migration before merging:\n```sh\ndot run prod@migration/rename-columns\ndot test prod@migration/rename-columns\n```\n\nRoll forward validation (red/green):\n```sh\ndot build prod@current_prod_tag\ndot build prod@next_release_candidate\n# Validate row counts, constraints, performance before switching consumers\n```\n\nHistorical investigation:\n```sh\ndot run dev@2024-12-01-tag\n```\n\n### profiles.yml Detection & Rewriting\n\n`dot` invokes `dbt debug --config-dir` (wrapped through its own command builder) to locate the effective `profiles.yml`. It then:\n- Loads the user\u2019s configured profile\n- Extracts the target matching the active environment\n- Updates only the `schema` field (preserving credentials, threads, etc.)\n- Writes a minimal isolated `profiles.yml` containing just that profile + target\n\n### Passing Additional dbt Args\n\nAnything after `--` is passed through untouched:\n```sh\ndot run dev@main -- --select my_model+\n```\n\n### Cleanup\n\nCurrently there is no automatic cleanup. To reclaim space:\n\n- Drop old schemas manually from your warehouse\n- Remove stale directories under `.dot/build/`\n\nAutomatic management of old build artifacts and schemas is planned for a future release. Please let me know if this would be important to you!\n\n### Troubleshooting\n\n| Symptom | Cause | Action |\n|---------|-------|--------|\n| Error: Profile not found | Active environment or profile missing | Verify `profiles.yml` and environment name |\n| Commit not found | Bad ref | Run `git show<ref>` to validate |\n| Schema clutter | Many builds kept | Periodically prune `.dot/build` and drop old schemas |\n| Wrong default environment | `environment.default` unset or unexpected | Set `default` under `environment` in `vars.yml` |\n\n### Reference\n\nFor architectural rationale see: [ADR 0001: Isolated Builds](adr/0001-isolated-builds.md).\n\n## Architectural Decision Records\n\nArchitectural decisions are documented in the [adr/](adr/) directory.\n\n- [ADR 0001: Isolated Builds](adr/0001-isolated-builds.md)\n\n## Changelog\n\nSee [CHANGELOG.md](CHANGELOG.md) for released versions.\n\n## License\n\nThis project is licensed under the MIT License. See the `LICENSE` file for full details.\n\nSPDX-License-Identifier: MIT\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) 2025 Adam MacLeod (adam@macleod.id.au) Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice (including the next paragraph) shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.",
"summary": "dot - the Data Orchestration Tool (for dbt)",
"version": "0.3.0",
"project_urls": {
"Homepage": "https://github.com/adammacleod/dot-for-dbt",
"Issues": "https://github.com/adammacleod/dot-for-dbt/issues",
"Repository": "https://github.com/adammacleod/dot-for-dbt"
},
"split_keywords": [
"analytics",
" cli",
" data",
" dbt",
" orchestration",
" reproducibility"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "26cef412098051f4360208f44b598869ce6928ccd79a9201257a98bd34dca8ec",
"md5": "3840c4200764476098da946f4bd574b2",
"sha256": "a243700d180e29407ada0b243d8d1bc4bc5d51e4446ac05a2e8ac7b2db2fda49"
},
"downloads": -1,
"filename": "dot_for_dbt-0.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3840c4200764476098da946f4bd574b2",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.12",
"size": 16917,
"upload_time": "2025-09-07T10:50:42",
"upload_time_iso_8601": "2025-09-07T10:50:42.881501Z",
"url": "https://files.pythonhosted.org/packages/26/ce/f412098051f4360208f44b598869ce6928ccd79a9201257a98bd34dca8ec/dot_for_dbt-0.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "6a4b41de8ba87eaaf36028f1ab099eb450fdfaa51c3042267a01e381bf3deb6b",
"md5": "3fb7e159b1c0037a13d51274b3a1e0b2",
"sha256": "9c93dd8401f2cb79af07532779ab901bb0f785244a41876a321c953e7645e92e"
},
"downloads": -1,
"filename": "dot_for_dbt-0.3.0.tar.gz",
"has_sig": false,
"md5_digest": "3fb7e159b1c0037a13d51274b3a1e0b2",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.12",
"size": 380637,
"upload_time": "2025-09-07T10:50:45",
"upload_time_iso_8601": "2025-09-07T10:50:45.103390Z",
"url": "https://files.pythonhosted.org/packages/6a/4b/41de8ba87eaaf36028f1ab099eb450fdfaa51c3042267a01e381bf3deb6b/dot_for_dbt-0.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-09-07 10:50:45",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "adammacleod",
"github_project": "dot-for-dbt",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "dot-for-dbt"
}
