Astronomer Providers
====================
.. image:: https://badge.fury.io/py/astronomer-providers.svg
:target: https://badge.fury.io/py/astronomer-providers
:alt: PyPI Version
.. image:: https://img.shields.io/pypi/pyversions/astronomer-providers
:target: https://img.shields.io/pypi/pyversions/astronomer-providers
:alt: PyPI - Python Version
.. image:: https://img.shields.io/pypi/l/astronomer-providers?color=blue
:target: https://img.shields.io/pypi/l/astronomer-providers?color=blue
:alt: PyPI - License
.. image:: https://img.shields.io/badge/code%20style-black-000000.svg
:target: https://github.com/psf/black
:alt: Code style: black
.. image:: https://codecov.io/gh/astronomer/astronomer-providers/branch/main/graph/badge.svg?token=LPHFRC3CB3
:target: https://codecov.io/gh/astronomer/astronomer-providers
:alt: CodeCov
.. image:: https://readthedocs.org/projects/astronomer-providers/badge/?version=latest
:target: https://astronomer-providers.readthedocs.io/en/latest/?badge=latest
:alt: Documentation Status
.. image:: https://img.shields.io/badge/security-bandit-green.svg
:target: https://github.com/PyCQA/bandit
:alt: Security: bandit
.. warning::
The majority of operators and sensors within this repository have been deprecated and will not receive further updates.
Read more about the deprecation in the `Deprecation Notice` section below.
Deprecation Notice
------------------
With the release ``1.19.0`` of the astronomer-providers package, most of the operators and sensors are deprecated and will
no longer receive updates. We recommend migrating to the official Apache Airflow Providers for the latest features and support.
For the operators and sensors that are deprecated in this repository, migrating to the official Apache Airflow Providers
is as simple as changing the import path from
.. code-block::
from astronomer.providers.*.*.operator_module import SomeOperatorAsync
to
.. code-block::
from airflow.providers.*.*.operator_module import SomeOperator
and setting the ``deferrable`` argument to ``True`` while using the operator or sensor in your DAG.
Setting the ``deferrable`` argument to ``True`` will ensure that the operator or sensor is using the async version
of the operator or sensor from the official Apache Airflow Providers.
For example, to migrate from
``astronomer.providers.amazon.aws.operators.batch.BatchOperatorAsync`` to
``airflow.providers.amazon.aws.operators.s3.BatchOperator``, simply change the import path and pass
the deferrable argument:
.. code-block:: python
BatchOperator(
task_id="copy_object",
your_arguments,
your_keyword_arguments,
deferrable=True,
)
For more information on using the deferrable operators and sensors from the official Apache Airflow Providers, visit the following links:
- https://airflow.apache.org/docs/apache-airflow-providers/index.html
- https://airflow.apache.org/docs/#providers-packages-docs-apache-airflow-providers-index-html
- https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/deferring.html
.. note::
Although the default value for the `deferrable` argument is `False`, it's possible to configure the default value for the `deferrable` argument across
your deployment by setting the `default_deferrable` flag in the `operators` sections of your Airflow configuration. Once you set the `default_deferrable` flag to `True`,
you can remove the `deferrable` argument from your operators and sensors and they will use the async version of the operator or sensor from the official Apache Airflow Providers
if it exists.
See more at: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#default-deferrable
For troubleshooting of issues with migrations, you are suggested to open up a `GitHub discussion <https://github.com/astronomer/astronomer-providers/discussions>`_
Installation
------------
Install and update using `pip <https://pip.pypa.io/en/stable/getting-started/>`_:
.. code-block:: bash
pip install astronomer-providers
This only installs dependencies for core provider. To install all dependencies, run:
.. code-block:: bash
pip install 'astronomer-providers[all]'
To only install the dependencies for a specific provider, specify the integration name as extra argument, example
to install Kubernetes provider dependencies, run:
.. code-block:: bash
pip install 'astronomer-providers[cncf.kubernetes]'
Extras
^^^^^^
.. EXTRA_DOC_START
.. list-table::
:header-rows: 1
* - Extra Name
- Installation Command
- Dependencies
* - ``all``
- ``pip install 'astronomer-providers[all]'``
- All
* - ``amazon``
- ``pip install 'astronomer-providers[amazon]'``
- Amazon
* - ``apache.hive``
- ``pip install 'astronomer-providers[apache.hive]'``
- Apache Hive
* - ``apache.livy``
- ``pip install 'astronomer-providers[apache.livy]'``
- Apache Livy
* - ``cncf.kubernetes``
- ``pip install 'astronomer-providers[cncf.kubernetes]'``
- Cncf Kubernetes
* - ``databricks``
- ``pip install 'astronomer-providers[databricks]'``
- Databricks
* - ``dbt.cloud``
- ``pip install 'astronomer-providers[dbt.cloud]'``
- Dbt Cloud
* - ``google``
- ``pip install 'astronomer-providers[google]'``
- Google
* - ``http``
- ``pip install 'astronomer-providers[http]'``
- Http
* - ``microsoft.azure``
- ``pip install 'astronomer-providers[microsoft.azure]'``
- Microsoft Azure
* - ``openlineage``
- ``pip install 'astronomer-providers[openlineage]'``
- Openlineage
* - ``sftp``
- ``pip install 'astronomer-providers[sftp]'``
- Sftp
* - ``snowflake``
- ``pip install 'astronomer-providers[snowflake]'``
- Snowflake
.. EXTRA_DOC_END
Example Usage
-------------
This repo is structured same as the Apache Airflow's source code, so for example
if you want to import Async operators, you can import it as follows:
.. code-block:: python
from astronomer.providers.amazon.aws.sensors.s3 import S3KeySensorAsync as S3KeySensor
waiting_for_s3_key = S3KeySensor(
task_id="waiting_for_s3_key",
bucket_key="sample_key.txt",
wildcard_match=False,
bucket_name="sample-bucket",
)
**Example DAGs** for each provider is within the respective provider's folder. For example,
the Kubernetes provider's DAGs are within the
`astronomer/providers/cncf/kubernetes/example_dags <https://github.com/astronomer/astronomer-providers/tree/main/astronomer/providers/cncf/kubernetes/example_dags>`_
folder.
Principle
---------
We will only create Async operators for the "sync-version" of operators that do some level of polling
(take more than a few seconds to complete).
For example, we won’t create an async Operator for a ``BigQueryCreateEmptyTableOperator`` but will create one
for ``BigQueryInsertJobOperator`` that actually runs queries and can take hours in the worst case for task completion.
To create async operators, we need to inherit from the corresponding airflow sync operators.
If sync version isn't available, then inherit from airflow ``BaseOperator``.
To create async sensors, we need to inherit from the corresponding sync sensors.
If sync version isn't available, then inherit from airflow ``BaseSensorOperator``.
Changelog
---------
We follow `Semantic Versioning <https://semver.org/>`_ for releases.
Check `CHANGELOG.rst <https://github.com/astronomer/astronomer-providers/blob/main/CHANGELOG.rst>`_
for the latest changes.
Contributing Guide
------------------
All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.
A detailed overview on how to contribute can be found in the
`Contributing Guide <https://github.com/astronomer/astronomer-providers/blob/main/CONTRIBUTING.rst>`_.
As contributors and maintainers to this project, you are expected to abide by the
`Contributor Code of Conduct <https://github.com/astronomer/astronomer-providers/blob/main/CODE_OF_CONDUCT.md>`_.
Goals for the project
---------------------
- Our focus is on the speed of iteration and development in this stage of the project and so we want to be able to
quickly iterate with our community members and customers and cut releases as necessary
- Airflow Providers are separate packages from the core ``apache-airflow`` package and we would like to avoid
further bloating the Airflow repo
- We want users and the community to be able to easily track features and the roadmap for individual providers
that we develop
- We would love to see the Airflow community members create, maintain and share their providers to build an Ecosystem
of Providers.
Limitations
-----------
- In Airflow sensors have a param ``mode`` which can be ``poke`` and ``reschedule``.
In async sensors, this param has no usage since tasks gets deferred to Triggerer.
License
-------
`Apache License 2.0 <https://github.com/astronomer/astronomer-providers/blob/main/LICENSE>`_
Raw data
{
"_id": null,
"home_page": "https://github.com/astronomer/astronomer-providers/",
"name": "astronomer-providers",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "",
"author": "Astronomer",
"author_email": "humans@astronomer.io",
"download_url": "https://files.pythonhosted.org/packages/77/20/0995743c2597eabf2e06924035548772390feecb61a765bc474f970aea54/astronomer-providers-1.19.0.tar.gz",
"platform": null,
"description": "Astronomer Providers\n====================\n\n.. image:: https://badge.fury.io/py/astronomer-providers.svg\n :target: https://badge.fury.io/py/astronomer-providers\n :alt: PyPI Version\n.. image:: https://img.shields.io/pypi/pyversions/astronomer-providers\n :target: https://img.shields.io/pypi/pyversions/astronomer-providers\n :alt: PyPI - Python Version\n.. image:: https://img.shields.io/pypi/l/astronomer-providers?color=blue\n :target: https://img.shields.io/pypi/l/astronomer-providers?color=blue\n :alt: PyPI - License\n.. image:: https://img.shields.io/badge/code%20style-black-000000.svg\n :target: https://github.com/psf/black\n :alt: Code style: black\n.. image:: https://codecov.io/gh/astronomer/astronomer-providers/branch/main/graph/badge.svg?token=LPHFRC3CB3\n :target: https://codecov.io/gh/astronomer/astronomer-providers\n :alt: CodeCov\n.. image:: https://readthedocs.org/projects/astronomer-providers/badge/?version=latest\n :target: https://astronomer-providers.readthedocs.io/en/latest/?badge=latest\n :alt: Documentation Status\n.. image:: https://img.shields.io/badge/security-bandit-green.svg\n :target: https://github.com/PyCQA/bandit\n :alt: Security: bandit\n\n\n.. warning::\n The majority of operators and sensors within this repository have been deprecated and will not receive further updates.\n Read more about the deprecation in the `Deprecation Notice` section below.\n\nDeprecation Notice\n------------------\n\nWith the release ``1.19.0`` of the astronomer-providers package, most of the operators and sensors are deprecated and will\nno longer receive updates. We recommend migrating to the official Apache Airflow Providers for the latest features and support.\nFor the operators and sensors that are deprecated in this repository, migrating to the official Apache Airflow Providers\nis as simple as changing the import path from\n\n.. code-block::\n\n from astronomer.providers.*.*.operator_module import SomeOperatorAsync\n\nto\n\n.. code-block::\n\n from airflow.providers.*.*.operator_module import SomeOperator\n\nand setting the ``deferrable`` argument to ``True`` while using the operator or sensor in your DAG.\nSetting the ``deferrable`` argument to ``True`` will ensure that the operator or sensor is using the async version\nof the operator or sensor from the official Apache Airflow Providers.\n\nFor example, to migrate from\n``astronomer.providers.amazon.aws.operators.batch.BatchOperatorAsync`` to\n``airflow.providers.amazon.aws.operators.s3.BatchOperator``, simply change the import path and pass\nthe deferrable argument:\n\n.. code-block:: python\n\n BatchOperator(\n task_id=\"copy_object\",\n your_arguments,\n your_keyword_arguments,\n deferrable=True,\n )\n\nFor more information on using the deferrable operators and sensors from the official Apache Airflow Providers, visit the following links:\n\n- https://airflow.apache.org/docs/apache-airflow-providers/index.html\n- https://airflow.apache.org/docs/#providers-packages-docs-apache-airflow-providers-index-html\n- https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/deferring.html\n\n.. note::\n Although the default value for the `deferrable` argument is `False`, it's possible to configure the default value for the `deferrable` argument across\n your deployment by setting the `default_deferrable` flag in the `operators` sections of your Airflow configuration. Once you set the `default_deferrable` flag to `True`,\n you can remove the `deferrable` argument from your operators and sensors and they will use the async version of the operator or sensor from the official Apache Airflow Providers\n if it exists.\n\n See more at: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#default-deferrable\n\nFor troubleshooting of issues with migrations, you are suggested to open up a `GitHub discussion <https://github.com/astronomer/astronomer-providers/discussions>`_\n\n\nInstallation\n------------\n\nInstall and update using `pip <https://pip.pypa.io/en/stable/getting-started/>`_:\n\n.. code-block:: bash\n\n pip install astronomer-providers\n\nThis only installs dependencies for core provider. To install all dependencies, run:\n\n.. code-block:: bash\n\n pip install 'astronomer-providers[all]'\n\nTo only install the dependencies for a specific provider, specify the integration name as extra argument, example\nto install Kubernetes provider dependencies, run:\n\n.. code-block:: bash\n\n pip install 'astronomer-providers[cncf.kubernetes]'\n\nExtras\n^^^^^^\n\n.. EXTRA_DOC_START\n\n.. list-table::\n :header-rows: 1\n\n * - Extra Name\n - Installation Command\n - Dependencies\n\n * - ``all``\n - ``pip install 'astronomer-providers[all]'``\n - All\n\n * - ``amazon``\n - ``pip install 'astronomer-providers[amazon]'``\n - Amazon\n\n * - ``apache.hive``\n - ``pip install 'astronomer-providers[apache.hive]'``\n - Apache Hive\n\n * - ``apache.livy``\n - ``pip install 'astronomer-providers[apache.livy]'``\n - Apache Livy\n\n * - ``cncf.kubernetes``\n - ``pip install 'astronomer-providers[cncf.kubernetes]'``\n - Cncf Kubernetes\n\n * - ``databricks``\n - ``pip install 'astronomer-providers[databricks]'``\n - Databricks\n\n * - ``dbt.cloud``\n - ``pip install 'astronomer-providers[dbt.cloud]'``\n - Dbt Cloud\n\n * - ``google``\n - ``pip install 'astronomer-providers[google]'``\n - Google\n\n * - ``http``\n - ``pip install 'astronomer-providers[http]'``\n - Http\n\n * - ``microsoft.azure``\n - ``pip install 'astronomer-providers[microsoft.azure]'``\n - Microsoft Azure\n\n * - ``openlineage``\n - ``pip install 'astronomer-providers[openlineage]'``\n - Openlineage\n\n * - ``sftp``\n - ``pip install 'astronomer-providers[sftp]'``\n - Sftp\n\n * - ``snowflake``\n - ``pip install 'astronomer-providers[snowflake]'``\n - Snowflake\n\n.. EXTRA_DOC_END\n\nExample Usage\n-------------\n\nThis repo is structured same as the Apache Airflow's source code, so for example\nif you want to import Async operators, you can import it as follows:\n\n.. code-block:: python\n\n from astronomer.providers.amazon.aws.sensors.s3 import S3KeySensorAsync as S3KeySensor\n\n waiting_for_s3_key = S3KeySensor(\n task_id=\"waiting_for_s3_key\",\n bucket_key=\"sample_key.txt\",\n wildcard_match=False,\n bucket_name=\"sample-bucket\",\n )\n\n**Example DAGs** for each provider is within the respective provider's folder. For example,\nthe Kubernetes provider's DAGs are within the\n`astronomer/providers/cncf/kubernetes/example_dags <https://github.com/astronomer/astronomer-providers/tree/main/astronomer/providers/cncf/kubernetes/example_dags>`_\nfolder.\n\nPrinciple\n---------\n\nWe will only create Async operators for the \"sync-version\" of operators that do some level of polling\n(take more than a few seconds to complete).\n\nFor example, we won\u2019t create an async Operator for a ``BigQueryCreateEmptyTableOperator`` but will create one\nfor ``BigQueryInsertJobOperator`` that actually runs queries and can take hours in the worst case for task completion.\n\nTo create async operators, we need to inherit from the corresponding airflow sync operators.\nIf sync version isn't available, then inherit from airflow ``BaseOperator``.\n\nTo create async sensors, we need to inherit from the corresponding sync sensors.\nIf sync version isn't available, then inherit from airflow ``BaseSensorOperator``.\n\nChangelog\n---------\n\nWe follow `Semantic Versioning <https://semver.org/>`_ for releases.\nCheck `CHANGELOG.rst <https://github.com/astronomer/astronomer-providers/blob/main/CHANGELOG.rst>`_\nfor the latest changes.\n\nContributing Guide\n------------------\n\nAll contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.\n\nA detailed overview on how to contribute can be found in the\n`Contributing Guide <https://github.com/astronomer/astronomer-providers/blob/main/CONTRIBUTING.rst>`_.\n\nAs contributors and maintainers to this project, you are expected to abide by the\n`Contributor Code of Conduct <https://github.com/astronomer/astronomer-providers/blob/main/CODE_OF_CONDUCT.md>`_.\n\nGoals for the project\n---------------------\n\n- Our focus is on the speed of iteration and development in this stage of the project and so we want to be able to\n quickly iterate with our community members and customers and cut releases as necessary\n- Airflow Providers are separate packages from the core ``apache-airflow`` package and we would like to avoid\n further bloating the Airflow repo\n- We want users and the community to be able to easily track features and the roadmap for individual providers\n that we develop\n- We would love to see the Airflow community members create, maintain and share their providers to build an Ecosystem\n of Providers.\n\nLimitations\n-----------\n\n- In Airflow sensors have a param ``mode`` which can be ``poke`` and ``reschedule``.\n In async sensors, this param has no usage since tasks gets deferred to Triggerer.\n\nLicense\n-------\n\n`Apache License 2.0 <https://github.com/astronomer/astronomer-providers/blob/main/LICENSE>`_\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "Apache Airflow Providers containing Deferrable Operators & Sensors from Astronomer",
"version": "1.19.0",
"project_urls": {
"Changelog": "https://github.com/astronomer/astronomer-providers/blob/main/CHANGELOG.rst",
"Documentation": "https://astronomer-providers.rtfd.io/",
"Homepage": "https://astronomer.io/",
"Source Code": "https://github.com/astronomer/astronomer-providers/"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4d9221fe836ad390db41ebe16b74a35022931b7462c175753c6598c0b88d1bec",
"md5": "2d5a602a9ac00b134aeff6cea31ff518",
"sha256": "29cf6c27291015a6de34d0cc452e030f83a37484cda87c4d94a4e656f892647d"
},
"downloads": -1,
"filename": "astronomer_providers-1.19.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "2d5a602a9ac00b134aeff6cea31ff518",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 237866,
"upload_time": "2024-02-21T10:46:05",
"upload_time_iso_8601": "2024-02-21T10:46:05.393948Z",
"url": "https://files.pythonhosted.org/packages/4d/92/21fe836ad390db41ebe16b74a35022931b7462c175753c6598c0b88d1bec/astronomer_providers-1.19.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "77200995743c2597eabf2e06924035548772390feecb61a765bc474f970aea54",
"md5": "38490a8bbde28033a526efdf4bd8426c",
"sha256": "2de10855bdba66f807e288d96e1ea1ef3a29db1468cd203e5ef913c0bbe46e4c"
},
"downloads": -1,
"filename": "astronomer-providers-1.19.0.tar.gz",
"has_sig": false,
"md5_digest": "38490a8bbde28033a526efdf4bd8426c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 154523,
"upload_time": "2024-02-21T10:46:08",
"upload_time_iso_8601": "2024-02-21T10:46:08.015146Z",
"url": "https://files.pythonhosted.org/packages/77/20/0995743c2597eabf2e06924035548772390feecb61a765bc474f970aea54/astronomer-providers-1.19.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-21 10:46:08",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "astronomer",
"github_project": "astronomer-providers",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"circle": true,
"lcname": "astronomer-providers"
}
JSON
