astronomer-cosmos


Nameastronomer-cosmos JSON
Version 1.1.1 PyPI version JSON
download
home_page
SummaryRender 3rd party workflows in Airflow
upload_time2023-09-14 15:14:09
maintainer
docs_urlNone
author
requires_python>=3.8
license
keywords airflow apache-airflow astronomer dags dbt
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage
            .. |fury| image:: https://badge.fury.io/py/astronomer-cosmos.svg
    :target: https://badge.fury.io/py/astronomer-cosmos

.. |ossrank| image:: https://img.shields.io/endpoint?url=https://ossrank.com/shield/2121
    :target: https://ossrank.com/p/2121-astronomer-cosmos

.. |downloads| image:: https://img.shields.io/pypi/dm/astronomer-cosmos.svg
    :target: https://img.shields.io/pypi/dm/astronomer-cosmos

.. |pre-commit| image:: https://results.pre-commit.ci/badge/github/astronomer/astronomer-cosmos/main.svg
   :target: https://results.pre-commit.ci/latest/github/astronomer/astronomer-cosmos/main
   :alt: pre-commit.ci status



.. image:: https://raw.githubusercontent.com/astronomer/astronomer-cosmos/main/docs/_static/cosmos-logo.svg


===========================================================

|fury| |ossrank| |downloads| |pre-commit|

Run your dbt Core projects as `Apache Airflow <https://airflow.apache.org/>`_ DAGs and Task Groups with a few lines of code. Benefits include:

- Run dbt projects against Airflow connections instead of dbt profiles
- Native support for installing and running dbt in a virtual environment to avoid dependency conflicts with Airflow
- Run tests immediately after a model is done to catch issues early
- Utilize Airflow's data-aware scheduling to run models immediately after upstream ingestion
- Turn each dbt model into a task/task group complete with retries, alerting, etc.

Quickstart
__________

Check out the Quickstart guide on our `docs <https://astronomer.github.io/astronomer-cosmos/#quickstart>`_.


Example Usage
___________________

You can render an Airflow Task Group using the ``DbtTaskGroup`` class. Here's an example with the jaffle_shop project:


.. code-block:: python

    from pendulum import datetime

    from airflow import DAG
    from airflow.operators.empty import EmptyOperator
    from cosmos import DbtTaskGroup, ProfileConfig, ProjectConfig
    from cosmos.profiles import PostgresUserPasswordProfileMapping

    profile_config = ProfileConfig(
        profile_name="default",
        target_name="dev",
        profile_mapping=PostgresUserPasswordProfileMapping(
            conn_id="airflow_db",
            profile_args={"schema": "public"},
        ),
    )

    with DAG(
        dag_id="extract_dag",
        start_date=datetime(2022, 11, 27),
        schedule="@daily",
    ):
        e1 = EmptyOperator(task_id="pre_dbt")

        dbt_tg = DbtTaskGroup(
            project_config=ProjectConfig("jaffle_shop"),
            profile_config=profile_config,
        )

        e2 = EmptyOperator(task_id="post_dbt")

        e1 >> dbt_tg >> e2

This will generate an Airflow Task Group that looks like this:

.. figure:: /docs/_static/jaffle_shop_task_group.png

Community
_________
- Join us on the Airflow `Slack <https://join.slack.com/t/apache-airflow/shared_invite/zt-1zy8e8h85-es~fn19iMzUmkhPwnyRT6Q>`_ at #airflow-dbt

Changelog
_________

We follow `Semantic Versioning <https://semver.org/>`_ for releases.
Check `CHANGELOG.rst <https://github.com/astronomer/astronomer-cosmos/blob/main/CHANGELOG.rst>`_
for the latest changes.

Contributing Guide
__________________

All contributions, bug reports, bug fixes, documentation improvements, enhancements are welcome.

A detailed overview an how to contribute can be found in the `Contributing Guide <https://astronomer.github.io/astronomer-cosmos/contributing>`_.

As contributors and maintainers to this project, you are expected to abide by the
`Contributor Code of Conduct <https://github.com/astronomer/astronomer-cosmos/blob/main/CODE_OF_CONDUCT.md>`_.


License
_______

`Apache License 2.0 <https://github.com/astronomer/astronomer-cosmos/blob/main/LICENSE>`_

            

Raw data

            {
    "_id": null,
    "home_page": "",
    "name": "astronomer-cosmos",
    "maintainer": "",
    "docs_url": null,
    "requires_python": ">=3.8",
    "maintainer_email": "",
    "keywords": "airflow,apache-airflow,astronomer,dags,dbt",
    "author": "",
    "author_email": "Astronomer <humans@astronomer.io>",
    "download_url": "https://files.pythonhosted.org/packages/cd/56/d2e09a5420fdf1c10316da1da34356a5ff7c3dffc3c6369706d1e945da8d/astronomer_cosmos-1.1.1.tar.gz",
    "platform": null,
    "description": ".. |fury| image:: https://badge.fury.io/py/astronomer-cosmos.svg\n    :target: https://badge.fury.io/py/astronomer-cosmos\n\n.. |ossrank| image:: https://img.shields.io/endpoint?url=https://ossrank.com/shield/2121\n    :target: https://ossrank.com/p/2121-astronomer-cosmos\n\n.. |downloads| image:: https://img.shields.io/pypi/dm/astronomer-cosmos.svg\n    :target: https://img.shields.io/pypi/dm/astronomer-cosmos\n\n.. |pre-commit| image:: https://results.pre-commit.ci/badge/github/astronomer/astronomer-cosmos/main.svg\n   :target: https://results.pre-commit.ci/latest/github/astronomer/astronomer-cosmos/main\n   :alt: pre-commit.ci status\n\n\n\n.. image:: https://raw.githubusercontent.com/astronomer/astronomer-cosmos/main/docs/_static/cosmos-logo.svg\n\n\n===========================================================\n\n|fury| |ossrank| |downloads| |pre-commit|\n\nRun your dbt Core projects as `Apache Airflow <https://airflow.apache.org/>`_ DAGs and Task Groups with a few lines of code. Benefits include:\n\n- Run dbt projects against Airflow connections instead of dbt profiles\n- Native support for installing and running dbt in a virtual environment to avoid dependency conflicts with Airflow\n- Run tests immediately after a model is done to catch issues early\n- Utilize Airflow's data-aware scheduling to run models immediately after upstream ingestion\n- Turn each dbt model into a task/task group complete with retries, alerting, etc.\n\nQuickstart\n__________\n\nCheck out the Quickstart guide on our `docs <https://astronomer.github.io/astronomer-cosmos/#quickstart>`_.\n\n\nExample Usage\n___________________\n\nYou can render an Airflow Task Group using the ``DbtTaskGroup`` class. Here's an example with the jaffle_shop project:\n\n\n.. code-block:: python\n\n    from pendulum import datetime\n\n    from airflow import DAG\n    from airflow.operators.empty import EmptyOperator\n    from cosmos import DbtTaskGroup, ProfileConfig, ProjectConfig\n    from cosmos.profiles import PostgresUserPasswordProfileMapping\n\n    profile_config = ProfileConfig(\n        profile_name=\"default\",\n        target_name=\"dev\",\n        profile_mapping=PostgresUserPasswordProfileMapping(\n            conn_id=\"airflow_db\",\n            profile_args={\"schema\": \"public\"},\n        ),\n    )\n\n    with DAG(\n        dag_id=\"extract_dag\",\n        start_date=datetime(2022, 11, 27),\n        schedule=\"@daily\",\n    ):\n        e1 = EmptyOperator(task_id=\"pre_dbt\")\n\n        dbt_tg = DbtTaskGroup(\n            project_config=ProjectConfig(\"jaffle_shop\"),\n            profile_config=profile_config,\n        )\n\n        e2 = EmptyOperator(task_id=\"post_dbt\")\n\n        e1 >> dbt_tg >> e2\n\nThis will generate an Airflow Task Group that looks like this:\n\n.. figure:: /docs/_static/jaffle_shop_task_group.png\n\nCommunity\n_________\n- Join us on the Airflow `Slack <https://join.slack.com/t/apache-airflow/shared_invite/zt-1zy8e8h85-es~fn19iMzUmkhPwnyRT6Q>`_ at #airflow-dbt\n\nChangelog\n_________\n\nWe follow `Semantic Versioning <https://semver.org/>`_ for releases.\nCheck `CHANGELOG.rst <https://github.com/astronomer/astronomer-cosmos/blob/main/CHANGELOG.rst>`_\nfor the latest changes.\n\nContributing Guide\n__________________\n\nAll contributions, bug reports, bug fixes, documentation improvements, enhancements are welcome.\n\nA detailed overview an how to contribute can be found in the `Contributing Guide <https://astronomer.github.io/astronomer-cosmos/contributing>`_.\n\nAs contributors and maintainers to this project, you are expected to abide by the\n`Contributor Code of Conduct <https://github.com/astronomer/astronomer-cosmos/blob/main/CODE_OF_CONDUCT.md>`_.\n\n\nLicense\n_______\n\n`Apache License 2.0 <https://github.com/astronomer/astronomer-cosmos/blob/main/LICENSE>`_\n",
    "bugtrack_url": null,
    "license": "",
    "summary": "Render 3rd party workflows in Airflow",
    "version": "1.1.1",
    "project_urls": {
        "Documentation": "https://astronomer.github.io/astronomer-cosmos",
        "Homepage": "https://github.com/astronomer/astronomer-cosmos",
        "Source code": "https://github.com/astronomer/astronomer-cosmos"
    },
    "split_keywords": [
        "airflow",
        "apache-airflow",
        "astronomer",
        "dags",
        "dbt"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "b16bf0af24b4d48842e4e780f4fb4cbfab5a9b46d5c4cd15e4d2439a02b98bf1",
                "md5": "30a9976ba0621b334db0daa8e1eef50b",
                "sha256": "745359d71f0017140d59ff41e127d339f8ac5b2d2345d3a0479a441b3cd80eb8"
            },
            "downloads": -1,
            "filename": "astronomer_cosmos-1.1.1-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "30a9976ba0621b334db0daa8e1eef50b",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.8",
            "size": 67337,
            "upload_time": "2023-09-14T15:14:07",
            "upload_time_iso_8601": "2023-09-14T15:14:07.973848Z",
            "url": "https://files.pythonhosted.org/packages/b1/6b/f0af24b4d48842e4e780f4fb4cbfab5a9b46d5c4cd15e4d2439a02b98bf1/astronomer_cosmos-1.1.1-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "cd56d2e09a5420fdf1c10316da1da34356a5ff7c3dffc3c6369706d1e945da8d",
                "md5": "57806762003772bab7aeff014146f276",
                "sha256": "f39898e2a6a90dc42f7226bc442a9564adfdad3d4d4fdc301a90a3ff51dfa022"
            },
            "downloads": -1,
            "filename": "astronomer_cosmos-1.1.1.tar.gz",
            "has_sig": false,
            "md5_digest": "57806762003772bab7aeff014146f276",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.8",
            "size": 44681,
            "upload_time": "2023-09-14T15:14:09",
            "upload_time_iso_8601": "2023-09-14T15:14:09.924872Z",
            "url": "https://files.pythonhosted.org/packages/cd/56/d2e09a5420fdf1c10316da1da34356a5ff7c3dffc3c6369706d1e945da8d/astronomer_cosmos-1.1.1.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2023-09-14 15:14:09",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "astronomer",
    "github_project": "astronomer-cosmos",
    "travis_ci": false,
    "coveralls": true,
    "github_actions": true,
    "lcname": "astronomer-cosmos"
}
        
Elapsed time: 0.16697s