# dbt-osmosis
<!---->
[](https://pepy.tech/project/dbt-osmosis)
[](https://z3z1ma-dbt-osmosis-srcdbt-osmosisapp-v2-i0ico9.streamlit.app/)
[](https://github.com/z3z1ma/dbt-osmosis/)
[](https://github.com/z3z1ma/dbt-osmosis/)
## New to dbt-osmosis?
We now have a spiffy [dbt-osmosis documentation site](https://z3z1ma.github.io/dbt-osmosis/)! 🎉
Please check it out for a more in-depth introduction to dbt-osmosis. 👇
[](https://z3z1ma.github.io/dbt-osmosis/)
## What is dbt-osmosis?
Hello and welcome to the project! [dbt-osmosis](https://github.com/z3z1ma/dbt-osmosis) 🌊 serves to enhance the developer experience significantly. We do this through providing 4 core features:
1. Automated schema YAML management.
1a. `dbt-osmosis yaml refactor --project-dir ... --profiles-dir ...`
> Automatically generate documentation based on upstream documented columns, organize yaml files based on configurable rules defined in dbt_project.yml, scaffold new yaml files based on the same rules, inject columns from data warehouse schema if missing in yaml and remove columns no longer present in data warehouse (organize -> document)
1b. `dbt-osmosis yaml organize --project-dir ... --profiles-dir ...`
> Organize yaml files based on configurable rules defined in dbt_project.yml, scaffold new yaml files based on the same rules
1c. `dbt-osmosis yaml document --project-dir ... --profiles-dir ...`
> Automatically generate documentation based on upstream documented columns
2. A highly performant dbt server which integrates with tools such as dbt-power-user for VS Code to enable interactive querying + realtime compilation from your IDE
2a. `dbt-osmosis server serve --project-dir ... --profiles-dir ...`
> Spins up a WSGI server. Can be passed --register-project to automatically register your local project
3. Workbench for dbt Jinja SQL. This workbench is powered by streamlit and the badge at the top of the readme will take you to a demo on streamlit cloud with jaffle_shop loaded (requires extra `pip install "dbt-osmosis[workbench]"`).
3a. `dbt-osmosis workbench --project-dir ... --profiles-dir ...`
> Spins up a streamlit app. This workbench offers similar functionality to the osmosis server + power-user combo without a reliance on VS code. Realtime compilation, query execution, pandas profiling all via copying and pasting whatever you are working on into the workbenchat your leisure. Spin it up and down as needed.
4. Diffs for data model outputs to model outputs across git revisions (🚧 this is in development)
4a. `dbt-osmosis diff -m some_model --project-dir ... --profiles-dir ...`
> Run diffs on models dynamically. This pulls the state of the model before changes from your git history, injects it as a node to the dbt manifest, compiles the old and modified nodes, and diffs their query results optionally writing nodes to temp tables before running the diff query for warehouses with performance or query complexity limits (👀 bigquery)
____
## Pre-commit
You can use dbt-osmosis as a pre-commit hook. This will run the `dbt-osmosis yaml refactor` command on your models directory before each commit. This is one way to ensure that your schema.yml files are always up to date. I would recommend reading the docs for more information on what this command does.
```yaml title=".pre-commit-config.yaml"
repos:
- repo: https://github.com/z3z1ma/dbt-osmosis
rev: v0.11.11 # verify the latest version
hooks:
- id: dbt-osmosis
files: ^models/
# you'd normally run this against your prod target, you can use any target though
args: [--target=prod]
additional_dependencies: [dbt-<adapter>]
```
___
## Workbench
The workbench is a streamlit app that allows you to work on dbt models in a side-by-side editor and query tester. I've kept this portion of the README since users can jump into the streamlit hosted workbench to play around with it via the badge below. Expect the living documentation moving forward to exist at the [dbt-osmosis documentation site](https://z3z1ma.github.io/dbt-osmosis/).
I also expect there is some untapped value in the workbench that is only pending some time from myself. I've seen a path to a truly novel development experience and look forward to exploring it.
Demo the workbench 👇
[](https://z3z1ma-dbt-osmosis-srcdbt-osmosisapp-v2-i0ico9.streamlit.app/)
```sh
# NOTE this requires the workbench extra as you can see
pip install "dbt-osmosis[workbench]"
# Command to start server
dbt-osmosis workbench
```
Press "r" to reload the workbench at any time.
✔️ dbt Editor with instant dbt compilation side-by-side or pivoted
✔️ Full control over model and workbench theme, light and dark mode
✔️ Query Tester, test the model you are working on for instant feedback
✔️ Data Profiler (leverages pandas-profiling)
**Editor**
The editor is able to compile models with control+enter or dynamically as you type. Its speedy! You can choose any target defined in your profiles yml for compilation and execution.
You can pivot the editor for a fuller view while workbenching some dbt SQL.
**Test Query**
Test dbt models as you work against whatever profile you have selected and inspect the results. This allows very fast iterative feedback loops not possible with VS Code alone.
**Profile Model Results**
Profile your datasets on the fly while you develop without switching context. Allows for more refined interactive data modelling when dataset fits in memory.
**Useful Links and RSS Feed**
Some useful links and RSS feeds at the bottom. 🤓
___
Raw data
{
"_id": null,
"home_page": "https://github.com/z3z1ma/dbt-osmosis",
"name": "dbt-osmosis",
"maintainer": null,
"docs_url": null,
"requires_python": "!=2.7.*,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,!=3.7.*,>=3.8",
"maintainer_email": null,
"keywords": "dbt, server, streamlit, git, diff",
"author": "z3z1ma",
"author_email": "butler.alex2010@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/b2/d9/835842ad004f99b319638d8ba0b831849caae1131ecd5c63d2858d9e67fe/dbt_osmosis-0.12.9.tar.gz",
"platform": null,
"description": "# dbt-osmosis\n\n<!---->\n\n\n[](https://pepy.tech/project/dbt-osmosis)\n\n\n[](https://z3z1ma-dbt-osmosis-srcdbt-osmosisapp-v2-i0ico9.streamlit.app/)\n\n\n[](https://github.com/z3z1ma/dbt-osmosis/)\n[](https://github.com/z3z1ma/dbt-osmosis/)\n\n## New to dbt-osmosis?\n\nWe now have a spiffy [dbt-osmosis documentation site](https://z3z1ma.github.io/dbt-osmosis/)! \ud83c\udf89\n\nPlease check it out for a more in-depth introduction to dbt-osmosis. \ud83d\udc47\n\n[](https://z3z1ma.github.io/dbt-osmosis/)\n\n## What is dbt-osmosis?\n\nHello and welcome to the project! [dbt-osmosis](https://github.com/z3z1ma/dbt-osmosis) \ud83c\udf0a serves to enhance the developer experience significantly. We do this through providing 4 core features:\n\n1. Automated schema YAML management.\n\n 1a. `dbt-osmosis yaml refactor --project-dir ... --profiles-dir ...`\n\n > Automatically generate documentation based on upstream documented columns, organize yaml files based on configurable rules defined in dbt_project.yml, scaffold new yaml files based on the same rules, inject columns from data warehouse schema if missing in yaml and remove columns no longer present in data warehouse (organize -> document)\n\n 1b. `dbt-osmosis yaml organize --project-dir ... --profiles-dir ...`\n\n > Organize yaml files based on configurable rules defined in dbt_project.yml, scaffold new yaml files based on the same rules\n\n 1c. `dbt-osmosis yaml document --project-dir ... --profiles-dir ...`\n\n > Automatically generate documentation based on upstream documented columns\n\n2. A highly performant dbt server which integrates with tools such as dbt-power-user for VS Code to enable interactive querying + realtime compilation from your IDE\n\n 2a. `dbt-osmosis server serve --project-dir ... --profiles-dir ...`\n\n > Spins up a WSGI server. Can be passed --register-project to automatically register your local project\n\n3. Workbench for dbt Jinja SQL. This workbench is powered by streamlit and the badge at the top of the readme will take you to a demo on streamlit cloud with jaffle_shop loaded (requires extra `pip install \"dbt-osmosis[workbench]\"`).\n\n 3a. `dbt-osmosis workbench --project-dir ... --profiles-dir ...`\n\n > Spins up a streamlit app. This workbench offers similar functionality to the osmosis server + power-user combo without a reliance on VS code. Realtime compilation, query execution, pandas profiling all via copying and pasting whatever you are working on into the workbenchat your leisure. Spin it up and down as needed.\n\n4. Diffs for data model outputs to model outputs across git revisions (\ud83d\udea7 this is in development)\n\n 4a. `dbt-osmosis diff -m some_model --project-dir ... --profiles-dir ...`\n\n > Run diffs on models dynamically. This pulls the state of the model before changes from your git history, injects it as a node to the dbt manifest, compiles the old and modified nodes, and diffs their query results optionally writing nodes to temp tables before running the diff query for warehouses with performance or query complexity limits (\ud83d\udc40 bigquery)\n\n____\n\n## Pre-commit\n\nYou can use dbt-osmosis as a pre-commit hook. This will run the `dbt-osmosis yaml refactor` command on your models directory before each commit. This is one way to ensure that your schema.yml files are always up to date. I would recommend reading the docs for more information on what this command does.\n\n```yaml title=\".pre-commit-config.yaml\"\nrepos:\n - repo: https://github.com/z3z1ma/dbt-osmosis\n rev: v0.11.11 # verify the latest version\n hooks:\n - id: dbt-osmosis\n files: ^models/\n # you'd normally run this against your prod target, you can use any target though\n args: [--target=prod]\n additional_dependencies: [dbt-<adapter>]\n```\n\n___\n\n## Workbench\n\nThe workbench is a streamlit app that allows you to work on dbt models in a side-by-side editor and query tester. I've kept this portion of the README since users can jump into the streamlit hosted workbench to play around with it via the badge below. Expect the living documentation moving forward to exist at the [dbt-osmosis documentation site](https://z3z1ma.github.io/dbt-osmosis/).\n\nI also expect there is some untapped value in the workbench that is only pending some time from myself. I've seen a path to a truly novel development experience and look forward to exploring it.\n\nDemo the workbench \ud83d\udc47\n\n[](https://z3z1ma-dbt-osmosis-srcdbt-osmosisapp-v2-i0ico9.streamlit.app/)\n\n\n```sh\n# NOTE this requires the workbench extra as you can see\npip install \"dbt-osmosis[workbench]\"\n\n# Command to start server\ndbt-osmosis workbench\n```\n\nPress \"r\" to reload the workbench at any time.\n\n\n\u2714\ufe0f dbt Editor with instant dbt compilation side-by-side or pivoted\n\n\u2714\ufe0f Full control over model and workbench theme, light and dark mode\n\n\u2714\ufe0f Query Tester, test the model you are working on for instant feedback\n\n\u2714\ufe0f Data Profiler (leverages pandas-profiling)\n\n\n**Editor**\n\nThe editor is able to compile models with control+enter or dynamically as you type. Its speedy! You can choose any target defined in your profiles yml for compilation and execution.\n\n![editor](/screenshots/osmosis_editor_main.png?raw=true \"dbt-osmosis Workbench\")\n\nYou can pivot the editor for a fuller view while workbenching some dbt SQL.\n\n![pivot](/screenshots/osmosis_editor_pivot.png?raw=true \"dbt-osmosis Pivot Layout\")\n\n\n**Test Query**\n\nTest dbt models as you work against whatever profile you have selected and inspect the results. This allows very fast iterative feedback loops not possible with VS Code alone.\n\n![test-model](/screenshots/osmosis_tester.png?raw=true \"dbt-osmosis Test Model\")\n\n**Profile Model Results**\n\nProfile your datasets on the fly while you develop without switching context. Allows for more refined interactive data modelling when dataset fits in memory.\n\n![profile-data](/screenshots/osmosis_profile.png?raw=true \"dbt-osmosis Profile Data\")\n\n\n**Useful Links and RSS Feed**\n\nSome useful links and RSS feeds at the bottom. \ud83e\udd13\n\n![profile-data](/screenshots/osmosis_links.png?raw=true \"dbt-osmosis Profile Data\")\n\n\n___\n\n![graph](https://repobeats.axiom.co/api/embed/df37714aa5780fc79871c60e6fc623f8f8e45c35.svg \"Repobeats analytics image\")\n\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "A dbt server and suite of optional developer tools to make developing with dbt delightful.",
"version": "0.12.9",
"project_urls": {
"Documentation": "https://github.com/z3z1ma/dbt-osmosis",
"Homepage": "https://github.com/z3z1ma/dbt-osmosis",
"Repository": "https://github.com/z3z1ma/dbt-osmosis"
},
"split_keywords": [
"dbt",
" server",
" streamlit",
" git",
" diff"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "5e56ce525908b365e00153abbfad299e70026dd2b3061cb04bfd1674444db848",
"md5": "654e8e7cbfe256c4cd7d2082a9ac7eb7",
"sha256": "062b254cb83e6d38bf4c546213cd3af9ca0151c803df31e013e18e59547235dc"
},
"downloads": -1,
"filename": "dbt_osmosis-0.12.9-py3-none-any.whl",
"has_sig": false,
"md5_digest": "654e8e7cbfe256c4cd7d2082a9ac7eb7",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "!=2.7.*,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,!=3.7.*,>=3.8",
"size": 116893,
"upload_time": "2024-04-08T00:13:52",
"upload_time_iso_8601": "2024-04-08T00:13:52.335445Z",
"url": "https://files.pythonhosted.org/packages/5e/56/ce525908b365e00153abbfad299e70026dd2b3061cb04bfd1674444db848/dbt_osmosis-0.12.9-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b2d9835842ad004f99b319638d8ba0b831849caae1131ecd5c63d2858d9e67fe",
"md5": "b4e2e0b5cdc562e9bf8e72fde83e8427",
"sha256": "8739ac620b8eb0cecaa58b6c3e6b347ce48fa4cd776d980b7ead96964dbc4d6e"
},
"downloads": -1,
"filename": "dbt_osmosis-0.12.9.tar.gz",
"has_sig": false,
"md5_digest": "b4e2e0b5cdc562e9bf8e72fde83e8427",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "!=2.7.*,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,!=3.7.*,>=3.8",
"size": 110276,
"upload_time": "2024-04-08T00:13:54",
"upload_time_iso_8601": "2024-04-08T00:13:54.814791Z",
"url": "https://files.pythonhosted.org/packages/b2/d9/835842ad004f99b319638d8ba0b831849caae1131ecd5c63d2858d9e67fe/dbt_osmosis-0.12.9.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-08 00:13:54",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "z3z1ma",
"github_project": "dbt-osmosis",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "dbt-osmosis"
}
JSON
