.. |fury| image:: https://badge.fury.io/py/astronomer-cosmos.svg
:target: https://badge.fury.io/py/astronomer-cosmos
.. |ossrank| image:: https://img.shields.io/endpoint?url=https://ossrank.com/shield/2121
:target: https://ossrank.com/p/2121-astronomer-cosmos
.. |downloads| image:: https://img.shields.io/pypi/dm/astronomer-cosmos.svg
:target: https://img.shields.io/pypi/dm/astronomer-cosmos
.. |pre-commit| image:: https://results.pre-commit.ci/badge/github/astronomer/astronomer-cosmos/main.svg
:target: https://results.pre-commit.ci/latest/github/astronomer/astronomer-cosmos/main
:alt: pre-commit.ci status
.. image:: https://raw.githubusercontent.com/astronomer/astronomer-cosmos/main/docs/_static/cosmos-logo.svg
===========================================================
|fury| |ossrank| |downloads| |pre-commit|
Run your dbt Core projects as `Apache Airflow <https://airflow.apache.org/>`_ DAGs and Task Groups with a few lines of code. Benefits include:
- Run dbt projects against Airflow connections instead of dbt profiles
- Native support for installing and running dbt in a virtual environment to avoid dependency conflicts with Airflow
- Run tests immediately after a model is done to catch issues early
- Utilize Airflow's data-aware scheduling to run models immediately after upstream ingestion
- Turn each dbt model into a task/task group complete with retries, alerting, etc.
Quickstart
__________
Check out the Quickstart guide on our `docs <https://astronomer.github.io/astronomer-cosmos/#quickstart>`_.
Example Usage
___________________
You can render an Airflow Task Group using the ``DbtTaskGroup`` class. Here's an example with the jaffle_shop project:
.. code-block:: python
from pendulum import datetime
from airflow import DAG
from airflow.operators.empty import EmptyOperator
from cosmos import DbtTaskGroup, ProfileConfig, ProjectConfig
from cosmos.profiles import PostgresUserPasswordProfileMapping
profile_config = ProfileConfig(
profile_name="default",
target_name="dev",
profile_mapping=PostgresUserPasswordProfileMapping(
conn_id="airflow_db",
profile_args={"schema": "public"},
),
)
with DAG(
dag_id="extract_dag",
start_date=datetime(2022, 11, 27),
schedule="@daily",
):
e1 = EmptyOperator(task_id="pre_dbt")
dbt_tg = DbtTaskGroup(
project_config=ProjectConfig("jaffle_shop"),
profile_config=profile_config,
)
e2 = EmptyOperator(task_id="post_dbt")
e1 >> dbt_tg >> e2
This will generate an Airflow Task Group that looks like this:
.. figure:: /docs/_static/jaffle_shop_task_group.png
Community
_________
- Join us on the Airflow `Slack <https://join.slack.com/t/apache-airflow/shared_invite/zt-1zy8e8h85-es~fn19iMzUmkhPwnyRT6Q>`_ at #airflow-dbt
Changelog
_________
We follow `Semantic Versioning <https://semver.org/>`_ for releases.
Check `CHANGELOG.rst <https://github.com/astronomer/astronomer-cosmos/blob/main/CHANGELOG.rst>`_
for the latest changes.
Contributing Guide
__________________
All contributions, bug reports, bug fixes, documentation improvements, enhancements are welcome.
A detailed overview an how to contribute can be found in the `Contributing Guide <https://astronomer.github.io/astronomer-cosmos/contributing>`_.
As contributors and maintainers to this project, you are expected to abide by the
`Contributor Code of Conduct <https://github.com/astronomer/astronomer-cosmos/blob/main/CODE_OF_CONDUCT.md>`_.
License
_______
`Apache License 2.0 <https://github.com/astronomer/astronomer-cosmos/blob/main/LICENSE>`_
Raw data
{
"_id": null,
"home_page": "",
"name": "astronomer-cosmos",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "airflow,apache-airflow,astronomer,dags,dbt",
"author": "",
"author_email": "Astronomer <humans@astronomer.io>",
"download_url": "https://files.pythonhosted.org/packages/cd/56/d2e09a5420fdf1c10316da1da34356a5ff7c3dffc3c6369706d1e945da8d/astronomer_cosmos-1.1.1.tar.gz",
"platform": null,
"description": ".. |fury| image:: https://badge.fury.io/py/astronomer-cosmos.svg\n :target: https://badge.fury.io/py/astronomer-cosmos\n\n.. |ossrank| image:: https://img.shields.io/endpoint?url=https://ossrank.com/shield/2121\n :target: https://ossrank.com/p/2121-astronomer-cosmos\n\n.. |downloads| image:: https://img.shields.io/pypi/dm/astronomer-cosmos.svg\n :target: https://img.shields.io/pypi/dm/astronomer-cosmos\n\n.. |pre-commit| image:: https://results.pre-commit.ci/badge/github/astronomer/astronomer-cosmos/main.svg\n :target: https://results.pre-commit.ci/latest/github/astronomer/astronomer-cosmos/main\n :alt: pre-commit.ci status\n\n\n\n.. image:: https://raw.githubusercontent.com/astronomer/astronomer-cosmos/main/docs/_static/cosmos-logo.svg\n\n\n===========================================================\n\n|fury| |ossrank| |downloads| |pre-commit|\n\nRun your dbt Core projects as `Apache Airflow <https://airflow.apache.org/>`_ DAGs and Task Groups with a few lines of code. Benefits include:\n\n- Run dbt projects against Airflow connections instead of dbt profiles\n- Native support for installing and running dbt in a virtual environment to avoid dependency conflicts with Airflow\n- Run tests immediately after a model is done to catch issues early\n- Utilize Airflow's data-aware scheduling to run models immediately after upstream ingestion\n- Turn each dbt model into a task/task group complete with retries, alerting, etc.\n\nQuickstart\n__________\n\nCheck out the Quickstart guide on our `docs <https://astronomer.github.io/astronomer-cosmos/#quickstart>`_.\n\n\nExample Usage\n___________________\n\nYou can render an Airflow Task Group using the ``DbtTaskGroup`` class. Here's an example with the jaffle_shop project:\n\n\n.. code-block:: python\n\n from pendulum import datetime\n\n from airflow import DAG\n from airflow.operators.empty import EmptyOperator\n from cosmos import DbtTaskGroup, ProfileConfig, ProjectConfig\n from cosmos.profiles import PostgresUserPasswordProfileMapping\n\n profile_config = ProfileConfig(\n profile_name=\"default\",\n target_name=\"dev\",\n profile_mapping=PostgresUserPasswordProfileMapping(\n conn_id=\"airflow_db\",\n profile_args={\"schema\": \"public\"},\n ),\n )\n\n with DAG(\n dag_id=\"extract_dag\",\n start_date=datetime(2022, 11, 27),\n schedule=\"@daily\",\n ):\n e1 = EmptyOperator(task_id=\"pre_dbt\")\n\n dbt_tg = DbtTaskGroup(\n project_config=ProjectConfig(\"jaffle_shop\"),\n profile_config=profile_config,\n )\n\n e2 = EmptyOperator(task_id=\"post_dbt\")\n\n e1 >> dbt_tg >> e2\n\nThis will generate an Airflow Task Group that looks like this:\n\n.. figure:: /docs/_static/jaffle_shop_task_group.png\n\nCommunity\n_________\n- Join us on the Airflow `Slack <https://join.slack.com/t/apache-airflow/shared_invite/zt-1zy8e8h85-es~fn19iMzUmkhPwnyRT6Q>`_ at #airflow-dbt\n\nChangelog\n_________\n\nWe follow `Semantic Versioning <https://semver.org/>`_ for releases.\nCheck `CHANGELOG.rst <https://github.com/astronomer/astronomer-cosmos/blob/main/CHANGELOG.rst>`_\nfor the latest changes.\n\nContributing Guide\n__________________\n\nAll contributions, bug reports, bug fixes, documentation improvements, enhancements are welcome.\n\nA detailed overview an how to contribute can be found in the `Contributing Guide <https://astronomer.github.io/astronomer-cosmos/contributing>`_.\n\nAs contributors and maintainers to this project, you are expected to abide by the\n`Contributor Code of Conduct <https://github.com/astronomer/astronomer-cosmos/blob/main/CODE_OF_CONDUCT.md>`_.\n\n\nLicense\n_______\n\n`Apache License 2.0 <https://github.com/astronomer/astronomer-cosmos/blob/main/LICENSE>`_\n",
"bugtrack_url": null,
"license": "",
"summary": "Render 3rd party workflows in Airflow",
"version": "1.1.1",
"project_urls": {
"Documentation": "https://astronomer.github.io/astronomer-cosmos",
"Homepage": "https://github.com/astronomer/astronomer-cosmos",
"Source code": "https://github.com/astronomer/astronomer-cosmos"
},
"split_keywords": [
"airflow",
"apache-airflow",
"astronomer",
"dags",
"dbt"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "b16bf0af24b4d48842e4e780f4fb4cbfab5a9b46d5c4cd15e4d2439a02b98bf1",
"md5": "30a9976ba0621b334db0daa8e1eef50b",
"sha256": "745359d71f0017140d59ff41e127d339f8ac5b2d2345d3a0479a441b3cd80eb8"
},
"downloads": -1,
"filename": "astronomer_cosmos-1.1.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "30a9976ba0621b334db0daa8e1eef50b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 67337,
"upload_time": "2023-09-14T15:14:07",
"upload_time_iso_8601": "2023-09-14T15:14:07.973848Z",
"url": "https://files.pythonhosted.org/packages/b1/6b/f0af24b4d48842e4e780f4fb4cbfab5a9b46d5c4cd15e4d2439a02b98bf1/astronomer_cosmos-1.1.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "cd56d2e09a5420fdf1c10316da1da34356a5ff7c3dffc3c6369706d1e945da8d",
"md5": "57806762003772bab7aeff014146f276",
"sha256": "f39898e2a6a90dc42f7226bc442a9564adfdad3d4d4fdc301a90a3ff51dfa022"
},
"downloads": -1,
"filename": "astronomer_cosmos-1.1.1.tar.gz",
"has_sig": false,
"md5_digest": "57806762003772bab7aeff014146f276",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 44681,
"upload_time": "2023-09-14T15:14:09",
"upload_time_iso_8601": "2023-09-14T15:14:09.924872Z",
"url": "https://files.pythonhosted.org/packages/cd/56/d2e09a5420fdf1c10316da1da34356a5ff7c3dffc3c6369706d1e945da8d/astronomer_cosmos-1.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-09-14 15:14:09",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "astronomer",
"github_project": "astronomer-cosmos",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"lcname": "astronomer-cosmos"
}
JSON
