[](https://pypi.org/project/dbt-af/)
[](https://github.com/Toloka/dbt-af/actions)
[](https://www.apache.org/licenses/LICENSE-2.0.txt)
[](https://pypi.org/project/dbt-af/)
[](https://pypi.org/project/dbt-af/)
[](https://python-poetry.org/)
[](https://github.com/psf/black)
# dbt-af: distributed run of dbt models using Airflow
## Overview
_dbt-af_ is a tool that allows you to run dbt models in a distributed manner using Airflow.
It acts as a wrapper around the Airflow DAG,
allowing you to run the models independently while preserving their dependencies.
### Why?
1. _dbt-af_ is [domain-driven](https://www.datamesh-architecture.com/#what-is-data-mesh).
It is designed to separate models from different domains into different DAGs.
This allows you to run models from different domains in parallel.
2. _dbt-af_ brings scheduling to dbt. You can schedule your dbt models to run at a specific time.
3. _dbt-af_ is an ETL-driven tool.
You can separate your models into tiers or ETL stages
and build graphs showing the dependencies between models within each tier or stage.
4. _dbt-af_ brings additional features to use different dbt targets simultaneously, different tests scenarios, and
maintenance tasks.
## Installation
To install `dbt-af` run `pip install dbt-af`.
To contribute we recommend to use `poetry` to install package dependencies. Run `poetry install --with=dev` to install
all dependencies.
## _dbt-af_ by Example
All tutorials and examples are located in the [examples](examples/README.md) folder.
To get basic Airflow DAGs for your dbt project, you need to put the following code into your `dags` folder:
```python
# LABELS: dag, airflow (it's required for airflow dag-processor)
from dbt_af.dags import compile_dbt_af_dags
from dbt_af.conf import Config, DbtDefaultTargetsConfig, DbtProjectConfig
# specify here all settings for your dbt project
config = Config(
dbt_project=DbtProjectConfig(
dbt_project_name='my_dbt_project',
dbt_project_path='/path/to/my_dbt_project',
dbt_models_path='/path/to/my_dbt_project/models',
dbt_profiles_path='/path/to/my_dbt_project',
dbt_target_path='/path/to/my_dbt_project/target',
dbt_log_path='/path/to/my_dbt_project/logs',
dbt_schema='my_dbt_schema',
),
dbt_default_targets=DbtDefaultTargetsConfig(default_target='dev'),
is_dev=False, # set to True if you want to turn on dry-run mode
)
dags = compile_dbt_af_dags(manifest_path='/path/to/my_dbt_project/target/manifest.json', config=config)
for dag_name, dag in dags.items():
globals()[dag_name] = dag
```
In _dbt_project.yml_ you need to set up default targets for all nodes in your project
(see [example](examples/dags/dbt_project.yml)):
```yaml
sql_cluster: "dev"
daily_sql_cluster: "dev"
py_cluster: "dev"
bf_cluster: "dev"
```
This will create Airflow DAGs for your dbt project.
## Project Information
- [Docs](examples/README.md)
- [PyPI](https://pypi.org/project/dbt-af/)
- Contributing
Raw data
{
"_id": null,
"home_page": "https://github.com/Toloka/dbt-af",
"name": "dbt-af",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.10",
"maintainer_email": null,
"keywords": "python, airflow, dbt",
"author": "Nikita Yurasov",
"author_email": "nikitayurasov@toloka.ai",
"download_url": "https://files.pythonhosted.org/packages/28/bd/30d7e1e051ab722295ca44c8cf537e7a49bfc4c7ff3ddd5a84635c01c4c9/dbt_af-0.4.2.tar.gz",
"platform": null,
"description": "[](https://pypi.org/project/dbt-af/)\n[](https://github.com/Toloka/dbt-af/actions)\n\n[](https://www.apache.org/licenses/LICENSE-2.0.txt)\n[](https://pypi.org/project/dbt-af/)\n[](https://pypi.org/project/dbt-af/)\n\n[](https://python-poetry.org/)\n[](https://github.com/psf/black)\n\n# dbt-af: distributed run of dbt models using Airflow\n\n## Overview\n\n_dbt-af_ is a tool that allows you to run dbt models in a distributed manner using Airflow.\nIt acts as a wrapper around the Airflow DAG,\nallowing you to run the models independently while preserving their dependencies.\n\n\n\n### Why?\n\n1. _dbt-af_ is [domain-driven](https://www.datamesh-architecture.com/#what-is-data-mesh).\n It is designed to separate models from different domains into different DAGs.\n This allows you to run models from different domains in parallel.\n2. _dbt-af_ brings scheduling to dbt. You can schedule your dbt models to run at a specific time.\n3. _dbt-af_ is an ETL-driven tool.\n You can separate your models into tiers or ETL stages\n and build graphs showing the dependencies between models within each tier or stage.\n4. _dbt-af_ brings additional features to use different dbt targets simultaneously, different tests scenarios, and\n maintenance tasks.\n\n## Installation\n\nTo install `dbt-af` run `pip install dbt-af`.\n\nTo contribute we recommend to use `poetry` to install package dependencies. Run `poetry install --with=dev` to install\nall dependencies.\n\n## _dbt-af_ by Example\n\nAll tutorials and examples are located in the [examples](examples/README.md) folder.\n\nTo get basic Airflow DAGs for your dbt project, you need to put the following code into your `dags` folder:\n\n```python\n# LABELS: dag, airflow (it's required for airflow dag-processor)\nfrom dbt_af.dags import compile_dbt_af_dags\nfrom dbt_af.conf import Config, DbtDefaultTargetsConfig, DbtProjectConfig\n\n# specify here all settings for your dbt project\nconfig = Config(\n dbt_project=DbtProjectConfig(\n dbt_project_name='my_dbt_project',\n dbt_project_path='/path/to/my_dbt_project',\n dbt_models_path='/path/to/my_dbt_project/models',\n dbt_profiles_path='/path/to/my_dbt_project',\n dbt_target_path='/path/to/my_dbt_project/target',\n dbt_log_path='/path/to/my_dbt_project/logs',\n dbt_schema='my_dbt_schema',\n ),\n dbt_default_targets=DbtDefaultTargetsConfig(default_target='dev'),\n is_dev=False, # set to True if you want to turn on dry-run mode\n)\n\ndags = compile_dbt_af_dags(manifest_path='/path/to/my_dbt_project/target/manifest.json', config=config)\nfor dag_name, dag in dags.items():\n globals()[dag_name] = dag\n```\n\nIn _dbt_project.yml_ you need to set up default targets for all nodes in your project \n(see [example](examples/dags/dbt_project.yml)):\n\n```yaml\nsql_cluster: \"dev\"\ndaily_sql_cluster: \"dev\"\npy_cluster: \"dev\"\nbf_cluster: \"dev\"\n```\n\nThis will create Airflow DAGs for your dbt project.\n\n## Project Information\n\n- [Docs](examples/README.md)\n- [PyPI](https://pypi.org/project/dbt-af/)\n- Contributing\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Distibuted dbt runs on Apache Airflow",
"version": "0.4.2",
"project_urls": {
"Documentation": "https://github.com/Toloka/dbt-af/blob/main/examples/README.md",
"Homepage": "https://github.com/Toloka/dbt-af",
"Repository": "https://github.com/Toloka/dbt-af"
},
"split_keywords": [
"python",
" airflow",
" dbt"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ff41a2d893177c128029fa5e39d816a61c318cade42c3990ef281618c6e0829e",
"md5": "78640c85033eaec19068370d709d78f7",
"sha256": "87164531293e3802c0c24d434d3957c68911b9b27733e3a3ebfe53dd3d92ac50"
},
"downloads": -1,
"filename": "dbt_af-0.4.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "78640c85033eaec19068370d709d78f7",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.10",
"size": 49349,
"upload_time": "2024-04-22T15:18:24",
"upload_time_iso_8601": "2024-04-22T15:18:24.308949Z",
"url": "https://files.pythonhosted.org/packages/ff/41/a2d893177c128029fa5e39d816a61c318cade42c3990ef281618c6e0829e/dbt_af-0.4.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "28bd30d7e1e051ab722295ca44c8cf537e7a49bfc4c7ff3ddd5a84635c01c4c9",
"md5": "ddd997de2738452c0eebda73078a3a89",
"sha256": "5c395d335a4a9947a8b0c33481fc75a8faa27adaa32a474eb9005fd6509ffa63"
},
"downloads": -1,
"filename": "dbt_af-0.4.2.tar.gz",
"has_sig": false,
"md5_digest": "ddd997de2738452c0eebda73078a3a89",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.10",
"size": 36580,
"upload_time": "2024-04-22T15:18:25",
"upload_time_iso_8601": "2024-04-22T15:18:25.449295Z",
"url": "https://files.pythonhosted.org/packages/28/bd/30d7e1e051ab722295ca44c8cf537e7a49bfc4c7ff3ddd5a84635c01c4c9/dbt_af-0.4.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-22 15:18:25",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Toloka",
"github_project": "dbt-af",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "dbt-af"
}
JSON
