[![PyPI - Version](https://img.shields.io/pypi/v/dbt-af)](https://pypi.org/project/dbt-af/)
[![GitHub Build](https://github.com/Toloka/dbt-af/workflows/Tests/badge.svg)](https://github.com/Toloka/dbt-af/actions)
[![License](https://img.shields.io/:license-Apache%202-blue.svg)](https://www.apache.org/licenses/LICENSE-2.0.txt)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/dbt-af.svg)](https://pypi.org/project/dbt-af/)
[![PyPI - Downloads](https://img.shields.io/pepy/dt/dbt-af)](https://pypi.org/project/dbt-af/)
[![Poetry](https://img.shields.io/endpoint?url=https://python-poetry.org/badge/v0.json)](https://python-poetry.org/)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
# dbt-af: distributed run of dbt models using Airflow
## Overview
_dbt-af_ is a tool that allows you to run dbt models in a distributed manner using Airflow.
It acts as a wrapper around the Airflow DAG,
allowing you to run the models independently while preserving their dependencies.
![dbt-af](docs/static/airflow_dag_layout.png)
### Why?
1. _dbt-af_ is [domain-driven](https://www.datamesh-architecture.com/#what-is-data-mesh).
It is designed to separate models from different domains into different DAGs.
This allows you to run models from different domains in parallel.
2. _dbt-af_ brings scheduling to dbt. You can schedule your dbt models to run at a specific time.
3. _dbt-af_ is an ETL-driven tool.
You can separate your models into tiers or ETL stages
and build graphs showing the dependencies between models within each tier or stage.
4. _dbt-af_ brings additional features to use different dbt targets simultaneously, different tests scenarios, and
maintenance tasks.
## Installation
To install `dbt-af` run `pip install dbt-af`.
To contribute we recommend to use `poetry` to install package dependencies. Run `poetry install --with=dev` to install
all dependencies.
## _dbt-af_ by Example
All tutorials and examples are located in the [examples](examples/README.md) folder.
To get basic Airflow DAGs for your dbt project, you need to put the following code into your `dags` folder:
```python
# LABELS: dag, airflow (it's required for airflow dag-processor)
from dbt_af.dags import compile_dbt_af_dags
from dbt_af.conf import Config, DbtDefaultTargetsConfig, DbtProjectConfig
# specify here all settings for your dbt project
config = Config(
dbt_project=DbtProjectConfig(
dbt_project_name='my_dbt_project',
dbt_project_path='/path/to/my_dbt_project',
dbt_models_path='/path/to/my_dbt_project/models',
dbt_profiles_path='/path/to/my_dbt_project',
dbt_target_path='/path/to/my_dbt_project/target',
dbt_log_path='/path/to/my_dbt_project/logs',
dbt_schema='my_dbt_schema',
),
dbt_default_targets=DbtDefaultTargetsConfig(default_target='dev'),
is_dev=False, # set to True if you want to turn on dry-run mode
)
dags = compile_dbt_af_dags(manifest_path='/path/to/my_dbt_project/target/manifest.json', config=config)
for dag_name, dag in dags.items():
globals()[dag_name] = dag
```
In _dbt_project.yml_ you need to set up default targets for all nodes in your project
(see [example](examples/dags/dbt_project.yml)):
```yaml
sql_cluster: "dev"
daily_sql_cluster: "dev"
py_cluster: "dev"
bf_cluster: "dev"
```
This will create Airflow DAGs for your dbt project.
## Project Information
- [Docs](examples/README.md)
- [PyPI](https://pypi.org/project/dbt-af/)
- Contributing
Raw data
{
"_id": null,
"home_page": "https://github.com/Toloka/dbt-af",
"name": "dbt-af",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.12,>=3.10",
"maintainer_email": null,
"keywords": "python, airflow, dbt",
"author": "Nikita Yurasov",
"author_email": "nikitayurasov@toloka.ai",
"download_url": "https://files.pythonhosted.org/packages/c4/25/7316f97401fbab8341844c10ca644c1ca899f56679e8c2217d4d0ddf8ed1/dbt_af-0.4.3.tar.gz",
"platform": null,
"description": "[![PyPI - Version](https://img.shields.io/pypi/v/dbt-af)](https://pypi.org/project/dbt-af/)\n[![GitHub Build](https://github.com/Toloka/dbt-af/workflows/Tests/badge.svg)](https://github.com/Toloka/dbt-af/actions)\n\n[![License](https://img.shields.io/:license-Apache%202-blue.svg)](https://www.apache.org/licenses/LICENSE-2.0.txt)\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/dbt-af.svg)](https://pypi.org/project/dbt-af/)\n[![PyPI - Downloads](https://img.shields.io/pepy/dt/dbt-af)](https://pypi.org/project/dbt-af/)\n\n[![Poetry](https://img.shields.io/endpoint?url=https://python-poetry.org/badge/v0.json)](https://python-poetry.org/)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n\n# dbt-af: distributed run of dbt models using Airflow\n\n## Overview\n\n_dbt-af_ is a tool that allows you to run dbt models in a distributed manner using Airflow.\nIt acts as a wrapper around the Airflow DAG,\nallowing you to run the models independently while preserving their dependencies.\n\n![dbt-af](docs/static/airflow_dag_layout.png)\n\n### Why?\n\n1. _dbt-af_ is [domain-driven](https://www.datamesh-architecture.com/#what-is-data-mesh).\n It is designed to separate models from different domains into different DAGs.\n This allows you to run models from different domains in parallel.\n2. _dbt-af_ brings scheduling to dbt. You can schedule your dbt models to run at a specific time.\n3. _dbt-af_ is an ETL-driven tool.\n You can separate your models into tiers or ETL stages\n and build graphs showing the dependencies between models within each tier or stage.\n4. _dbt-af_ brings additional features to use different dbt targets simultaneously, different tests scenarios, and\n maintenance tasks.\n\n## Installation\n\nTo install `dbt-af` run `pip install dbt-af`.\n\nTo contribute we recommend to use `poetry` to install package dependencies. Run `poetry install --with=dev` to install\nall dependencies.\n\n## _dbt-af_ by Example\n\nAll tutorials and examples are located in the [examples](examples/README.md) folder.\n\nTo get basic Airflow DAGs for your dbt project, you need to put the following code into your `dags` folder:\n\n```python\n# LABELS: dag, airflow (it's required for airflow dag-processor)\nfrom dbt_af.dags import compile_dbt_af_dags\nfrom dbt_af.conf import Config, DbtDefaultTargetsConfig, DbtProjectConfig\n\n# specify here all settings for your dbt project\nconfig = Config(\n dbt_project=DbtProjectConfig(\n dbt_project_name='my_dbt_project',\n dbt_project_path='/path/to/my_dbt_project',\n dbt_models_path='/path/to/my_dbt_project/models',\n dbt_profiles_path='/path/to/my_dbt_project',\n dbt_target_path='/path/to/my_dbt_project/target',\n dbt_log_path='/path/to/my_dbt_project/logs',\n dbt_schema='my_dbt_schema',\n ),\n dbt_default_targets=DbtDefaultTargetsConfig(default_target='dev'),\n is_dev=False, # set to True if you want to turn on dry-run mode\n)\n\ndags = compile_dbt_af_dags(manifest_path='/path/to/my_dbt_project/target/manifest.json', config=config)\nfor dag_name, dag in dags.items():\n globals()[dag_name] = dag\n```\n\nIn _dbt_project.yml_ you need to set up default targets for all nodes in your project \n(see [example](examples/dags/dbt_project.yml)):\n\n```yaml\nsql_cluster: \"dev\"\ndaily_sql_cluster: \"dev\"\npy_cluster: \"dev\"\nbf_cluster: \"dev\"\n```\n\nThis will create Airflow DAGs for your dbt project.\n\n## Project Information\n\n- [Docs](examples/README.md)\n- [PyPI](https://pypi.org/project/dbt-af/)\n- Contributing\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Distibuted dbt runs on Apache Airflow",
"version": "0.4.3",
"project_urls": {
"Documentation": "https://github.com/Toloka/dbt-af/blob/main/examples/README.md",
"Homepage": "https://github.com/Toloka/dbt-af",
"Repository": "https://github.com/Toloka/dbt-af"
},
"split_keywords": [
"python",
" airflow",
" dbt"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "6c3c26da11ea48b9bb3e9f3ddc39dddd932ca0a0ae9e606ffddf298664ab08e5",
"md5": "03dfa56392f44e21fd578890a9d63010",
"sha256": "b90589f0429978071f43d22e5399466df998e82a4ae11a8e668bb8cd34d1718e"
},
"downloads": -1,
"filename": "dbt_af-0.4.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "03dfa56392f44e21fd578890a9d63010",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.12,>=3.10",
"size": 49336,
"upload_time": "2024-05-16T14:50:58",
"upload_time_iso_8601": "2024-05-16T14:50:58.319386Z",
"url": "https://files.pythonhosted.org/packages/6c/3c/26da11ea48b9bb3e9f3ddc39dddd932ca0a0ae9e606ffddf298664ab08e5/dbt_af-0.4.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c4257316f97401fbab8341844c10ca644c1ca899f56679e8c2217d4d0ddf8ed1",
"md5": "bc14136901b04c75dc2aa571b0853b14",
"sha256": "d9b5f492cdaad8aac1bbd6b2f2d1629828026a69ccf2a8faf95b4dcc4195ae54"
},
"downloads": -1,
"filename": "dbt_af-0.4.3.tar.gz",
"has_sig": false,
"md5_digest": "bc14136901b04c75dc2aa571b0853b14",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.12,>=3.10",
"size": 36588,
"upload_time": "2024-05-16T14:51:04",
"upload_time_iso_8601": "2024-05-16T14:51:04.259464Z",
"url": "https://files.pythonhosted.org/packages/c4/25/7316f97401fbab8341844c10ca644c1ca899f56679e8c2217d4d0ddf8ed1/dbt_af-0.4.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-16 14:51:04",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Toloka",
"github_project": "dbt-af",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "dbt-af"
}