<p align="center">
<a href="https://dataroots.io"><img alt="Maintained by dataroots" src="https://dataroots.io/maintained-rnd.svg" /></a>
<a href="https://pypi.org/project/prefect-dbt-flow/"><img alt="Python versions" src="https://img.shields.io/pypi/pyversions/prefect-dbt-flow" /></a>
<a href="https://pypi.org/project/prefect-dbt-flow/"><img alt="PiPy" src="https://img.shields.io/pypi/v/prefect-dbt-flow" /></a>
<a href="https://pepy.tech/project/prefect-dbt-flow"><img alt="Downloads" src="https://pepy.tech/badge/prefect-dbt-flow" /></a>
<a href="https://github.com/psf/black"><img alt="Code style: black" src="https://img.shields.io/badge/code%20style-black-000000.svg" /></a>
<a href="http://mypy-lang.org/"><img alt="Mypy checked" src="https://img.shields.io/badge/mypy-checked-1f5082.svg" /></a>
<a href="https://codecov.io/gh/datarootsio/prefect-dbt-flow" >
<img src="https://codecov.io/gh/datarootsio/prefect-dbt-flow/graph/badge.svg?token=NQ6RMX6670"/>
</a>
</p>
# prefect-dbt-flow
prefect-dbt-flow is a Python library that enables Prefect to convert dbt workflows into independent tasks within a Prefect flow. This integration simplifies the orchestration and execution of dbt models and tests using Prefect, allowing you to build robust data pipelines and monitor your dbt projects efficiently.
dbt is an immensely popular tool for building and testing data transformation models, and Prefect is a versatile workflow management system. This integration brings together the best of both worlds, empowering data engineers and analysts to create robust data pipelines.
Key features:
- *Simplified Orchestration*: Define and manage your dbt projects and models as Prefect tasks, creating a seamless pipeline for data transformation.
- *Monitoring and Error Handling*: Gain deep insights into the execution of your dbt workflows and take immediate action in case of issues.
- *Workflow Consistency*: Ensure your dbt workflows run consistently by managing them through Prefect. This consistency is crucial for maintaining data quality and reliability.
- *Advanced Configuration*: Customize your dbt workflow by adjusting the dbt project, profile, and DAG options. You can also use Prefect features like scheduling, notifications, and task retries to monitor and manage your dbt flows effectively.
To get started, check out our [getting started guide](https://datarootsio.github.io/prefect-dbt-flow/latest/getting_started/).
**Active Development Notice:** *prefect-dbt-flow is actively under development and may not be ready for production use. We advise users to be aware of potential breaking changes as the library evolves. Please check the changelog for updates.*
## How to Install
You can install prefect-dbt-flow via pip:
```shell
pip install prefect-dbt-flow
```
*Note*: prefect-dbt-flow does not come with dbt as a dependency. You will need to install dbt or a dbt-adapter separately.
## Basic Usage
Here's an example of how to use prefect-dbt-flow to create a Prefect flow for your dbt project:
```python
from prefect_dbt_flow import dbt_flow
from prefect_dbt_flow.dbt import DbtProfile, DbtProject
my_flow = dbt_flow(
project=DbtProject(
name="jaffle_shop",
project_dir="path_to/jaffle_shop",
profiles_dir="path_to/jaffle_shop",
),
profile=DbtProfile(
target="dev",
overrides={
"type": "duckdb",
"path": "path_to/duckdb.db",
},
),
)
if __name__ == "__main__":
my_flow()
```
<img src="https://raw.githubusercontent.com/datarootsio/prefect-dbt-flow/main/docs/images/jaffle_shop_dag.png" alt="jaffle_shop_dag" width="100%">
For more information consult the [docs](https://datarootsio.github.io/prefect-dbt-flow/)
## Inspiration
prefect-dbt-flow draws inspiration from various projects in the data engineering and workflow orchestration space, including:
- [astronomer-cosmos](https://github.com/astronomer/astronomer-cosmos)
- [dbt + Dagster](https://docs.dagster.io/integrations/dbt)
- [prefect-dataplatform - Anna Geller](https://github.com/anna-geller/prefect-dataplatform)
# License
This project is licensed under the MIT License. You are free to use, modify, and distribute this software as per the terms of the license. If you find this project helpful, please consider giving it a star on GitHub.
Raw data
{
"_id": null,
"home_page": "https://datarootsio.github.io/prefect-dbt-flow",
"name": "prefect-dbt-flow",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.9",
"maintainer_email": null,
"keywords": "dbt, prefect",
"author": "David Valdez",
"author_email": "david@dataroots.io",
"download_url": null,
"platform": null,
"description": "<p align=\"center\">\n <a href=\"https://dataroots.io\"><img alt=\"Maintained by dataroots\" src=\"https://dataroots.io/maintained-rnd.svg\" /></a>\n <a href=\"https://pypi.org/project/prefect-dbt-flow/\"><img alt=\"Python versions\" src=\"https://img.shields.io/pypi/pyversions/prefect-dbt-flow\" /></a>\n <a href=\"https://pypi.org/project/prefect-dbt-flow/\"><img alt=\"PiPy\" src=\"https://img.shields.io/pypi/v/prefect-dbt-flow\" /></a>\n <a href=\"https://pepy.tech/project/prefect-dbt-flow\"><img alt=\"Downloads\" src=\"https://pepy.tech/badge/prefect-dbt-flow\" /></a>\n <a href=\"https://github.com/psf/black\"><img alt=\"Code style: black\" src=\"https://img.shields.io/badge/code%20style-black-000000.svg\" /></a>\n <a href=\"http://mypy-lang.org/\"><img alt=\"Mypy checked\" src=\"https://img.shields.io/badge/mypy-checked-1f5082.svg\" /></a>\n <a href=\"https://codecov.io/gh/datarootsio/prefect-dbt-flow\" >\n <img src=\"https://codecov.io/gh/datarootsio/prefect-dbt-flow/graph/badge.svg?token=NQ6RMX6670\"/>\n </a>\n</p>\n\n# prefect-dbt-flow\nprefect-dbt-flow is a Python library that enables Prefect to convert dbt workflows into independent tasks within a Prefect flow. This integration simplifies the orchestration and execution of dbt models and tests using Prefect, allowing you to build robust data pipelines and monitor your dbt projects efficiently.\n\ndbt is an immensely popular tool for building and testing data transformation models, and Prefect is a versatile workflow management system. This integration brings together the best of both worlds, empowering data engineers and analysts to create robust data pipelines.\n\nKey features:\n\n - *Simplified Orchestration*: Define and manage your dbt projects and models as Prefect tasks, creating a seamless pipeline for data transformation.\n - *Monitoring and Error Handling*: Gain deep insights into the execution of your dbt workflows and take immediate action in case of issues.\n - *Workflow Consistency*: Ensure your dbt workflows run consistently by managing them through Prefect. This consistency is crucial for maintaining data quality and reliability.\n - *Advanced Configuration*: Customize your dbt workflow by adjusting the dbt project, profile, and DAG options. You can also use Prefect features like scheduling, notifications, and task retries to monitor and manage your dbt flows effectively.\n\nTo get started, check out our [getting started guide](https://datarootsio.github.io/prefect-dbt-flow/latest/getting_started/).\n\n**Active Development Notice:** *prefect-dbt-flow is actively under development and may not be ready for production use. We advise users to be aware of potential breaking changes as the library evolves. Please check the changelog for updates.*\n\n## How to Install\nYou can install prefect-dbt-flow via pip:\n```shell\npip install prefect-dbt-flow\n```\n\n*Note*: prefect-dbt-flow does not come with dbt as a dependency. You will need to install dbt or a dbt-adapter separately.\n\n## Basic Usage\nHere's an example of how to use prefect-dbt-flow to create a Prefect flow for your dbt project:\n\n```python\nfrom prefect_dbt_flow import dbt_flow\nfrom prefect_dbt_flow.dbt import DbtProfile, DbtProject\n\nmy_flow = dbt_flow(\n project=DbtProject(\n name=\"jaffle_shop\",\n project_dir=\"path_to/jaffle_shop\",\n profiles_dir=\"path_to/jaffle_shop\",\n ),\n profile=DbtProfile(\n target=\"dev\",\n overrides={\n \"type\": \"duckdb\",\n \"path\": \"path_to/duckdb.db\",\n },\n ),\n)\n\nif __name__ == \"__main__\":\n my_flow()\n```\n\n<img src=\"https://raw.githubusercontent.com/datarootsio/prefect-dbt-flow/main/docs/images/jaffle_shop_dag.png\" alt=\"jaffle_shop_dag\" width=\"100%\">\n\nFor more information consult the [docs](https://datarootsio.github.io/prefect-dbt-flow/)\n\n## Inspiration\nprefect-dbt-flow draws inspiration from various projects in the data engineering and workflow orchestration space, including:\n- [astronomer-cosmos](https://github.com/astronomer/astronomer-cosmos)\n- [dbt + Dagster](https://docs.dagster.io/integrations/dbt)\n- [prefect-dataplatform - Anna Geller](https://github.com/anna-geller/prefect-dataplatform)\n\n# License\nThis project is licensed under the MIT License. You are free to use, modify, and distribute this software as per the terms of the license. If you find this project helpful, please consider giving it a star on GitHub.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Prefect - dbt integration",
"version": "0.5.8",
"project_urls": {
"Homepage": "https://datarootsio.github.io/prefect-dbt-flow",
"Repository": "https://github.com/datarootsio/prefect-dbt-flow"
},
"split_keywords": [
"dbt",
" prefect"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ec98d0790cc3ddfb1eee505dcb08b492f91b7ef72c76ab056929d15ca178e4c7",
"md5": "eae6d8324dbcb12111f481ada18af4ba",
"sha256": "016cf39cd53848164dc00af471e58220986a19dae4b8cc0fc1496d50217aea85"
},
"downloads": -1,
"filename": "prefect_dbt_flow-0.5.8-py3-none-any.whl",
"has_sig": false,
"md5_digest": "eae6d8324dbcb12111f481ada18af4ba",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.9",
"size": 11280,
"upload_time": "2024-04-25T09:13:51",
"upload_time_iso_8601": "2024-04-25T09:13:51.227931Z",
"url": "https://files.pythonhosted.org/packages/ec/98/d0790cc3ddfb1eee505dcb08b492f91b7ef72c76ab056929d15ca178e4c7/prefect_dbt_flow-0.5.8-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-25 09:13:51",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "datarootsio",
"github_project": "prefect-dbt-flow",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "prefect-dbt-flow"
}
JSON
