# Fivetran Async Provider for Apache Airflow
[](https://github.com/pre-commit/pre-commit)
[](https://github.com/astral-sh/ruff)
This package provides an async operator, sensor and hook that integrates [Fivetran](https://fivetran.com) into Apache Airflow.
`FivetranSensor` allows you to monitor a Fivetran sync job for completion before running downstream processes.
`FivetranOperator` submits a Fivetran sync job and polls for its status on the triggerer.
Since an async sensor or operator frees up worker slot while polling is happening on the triggerer,
they consume less resources when compared to traditional "sync" sensors and operators.
[Fivetran automates your data pipeline, and Airflow automates your data processing.](https://www.youtube.com/watch?v=siSx6L2ckSw&ab_channel=Fivetran)
## Installation
Prerequisites: An environment running `apache-airflow`.
```
pip install airflow-provider-fivetran-async
```
## Configuration
In the Airflow user interface, configure a Connection for Fivetran. Most of the Connection config fields will be left blank. Configure the following fields:
* `Conn Id`: `fivetran`
* `Conn Type`: `Fivetran`
* `Login`: Fivetran API Key
* `Password`: Fivetran API Secret
Find the Fivetran API Key and Secret in the [Fivetran Account Settings](https://fivetran.com/account/settings), under the **API Config** section. See our documentation for more information on [Fivetran API Authentication](https://fivetran.com/docs/rest-api/getting-started#authentication).
The sensor assumes the `Conn Id` is set to `fivetran`, however if you are managing multiple Fivetran accounts, you can set this to anything you like. See the DAG in examples to see how to specify a custom `Conn Id`.
## Modules
### [Fivetran Operator Async](https://github.com/astronomer/airflow-provider-fivetran-async/tree/main/fivetran_provider_async/operators.py)
```python
from fivetran_provider_async.operators import FivetranOperator
```
`FivetranOperator` submits a Fivetran sync job and monitors it on trigger for completion.
`FivetranOperator` requires that you specify the `connector_id` of the Fivetran connector you wish to trigger. You can find `connector_id` in the Settings page of the connector you configured in the [Fivetran dashboard](https://fivetran.com/dashboard/connectors).
The `FivetranOperator` will wait for the sync to complete so long as `wait_for_completion=True` (this is the default). It is recommended that
you run in deferrable mode (this is also the default). If `wait_for_completion=False`, the operator will return the timestamp for the last sync.
Import into your DAG via:
### [Fivetran Sensor Async](https://github.com/astronomer/airflow-provider-fivetran-async/tree/main/fivetran_provider_async/sensors.py)
```python
from fivetran_provider_async.sensors import FivetranSensor
```
`FivetranSensor` monitors a Fivetran sync job for completion.
Monitoring with `FivetranSensor` allows you to trigger downstream processes only when the Fivetran sync jobs have completed, ensuring data consistency.
`FivetranSensor` requires that you specify the `connector_id` of the Fivetran connector you want to wait for. You can find `connector_id` in the Settings page of the connector you configured in the [Fivetran dashboard](https://fivetran.com/dashboard/connectors).
You can use multiple instances of `FivetranSensor` to monitor multiple Fivetran connectors.
`FivetranSensor` is most commonly useful in two scenarios:
1. Fivetran is using a separate scheduler than the Airflow scheduler.
2. You set `wait_for_completion=False` in the `FivetranOperator`, and you need to await the `FivetranOperator` task later. (You may want to do this if you want to arrange your DAG such that some tasks are dependent on _starting_ a sync and other tasks are dependent on _completing_ a sync).
If you are doing the 1st pattern, you may find it useful to set the `completed_after_time` to `data_interval_end`, or `data_interval_end` with some buffer:
```python
fivetran_sensor = FivetranSensor(
task_id="wait_for_fivetran_externally_scheduled_sync",
connector_id="bronzing_largely",
poke_interval=5,
completed_after_time="{{ data_interval_end + macros.timedelta(minutes=1) }}",
)
```
If you are doing the 2nd pattern, you can use XComs to pass the target completed time to the sensor:
```python
fivetran_op = FivetranOperator(
task_id="fivetran_sync_my_db",
connector_id="bronzing_largely",
wait_for_completion=False,
)
fivetran_sensor = FivetranSensor(
task_id="wait_for_fivetran_db_sync",
connector_id="bronzing_largely",
poke_interval=5,
completed_after_time="{{ task_instance.xcom_pull('fivetran_sync_op', key='return_value') }}",
)
fivetran_op >> fivetran_sensor
```
You may also specify the `FivetranSensor` without a `completed_after_time`.
In this case, the sensor will make note of when the last completed time was, and will wait for a new completed time.
## Examples
See the [**examples**](https://github.com/astronomer/airflow-provider-fivetran-async/tree/main/fivetran_provider_async/dev/dags) directory for an example DAG.
## Issues
Please submit [issues](https://github.com/astronomer/airflow-provider-fivetran-async/issues) and [pull requests](https://github.com/astronomer/airflow-provider-fivetran-async/pulls) in our official repo:
[https://github.com/astronomer/airflow-provider-fivetran-async](https://github.com/astronomer/airflow-provider-fivetran-async)
We are happy to hear from you. Please email any feedback to the authors at [humans@astronomer.io](mailto:humans@astronomer.io).
Raw data
{
"_id": null,
"home_page": null,
"name": "airflow-provider-fivetran-async",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "airflow, apache-airflow, astronomer, dag, provider",
"author": null,
"author_email": "Astronomer & Fivetran <humans@astronomer.io>",
"download_url": "https://files.pythonhosted.org/packages/23/50/c793d7c40c071b2d24542f1737be1893016f77d23079f8b5ad2a8ea52090/airflow_provider_fivetran_async-2.2.0.tar.gz",
"platform": null,
"description": "# Fivetran Async Provider for Apache Airflow\n\n[](https://github.com/pre-commit/pre-commit)\n[](https://github.com/astral-sh/ruff)\n\nThis package provides an async operator, sensor and hook that integrates [Fivetran](https://fivetran.com) into Apache Airflow.\n`FivetranSensor` allows you to monitor a Fivetran sync job for completion before running downstream processes.\n`FivetranOperator` submits a Fivetran sync job and polls for its status on the triggerer.\nSince an async sensor or operator frees up worker slot while polling is happening on the triggerer,\nthey consume less resources when compared to traditional \"sync\" sensors and operators.\n\n[Fivetran automates your data pipeline, and Airflow automates your data processing.](https://www.youtube.com/watch?v=siSx6L2ckSw&ab_channel=Fivetran)\n\n## Installation\n\nPrerequisites: An environment running `apache-airflow`.\n\n```\npip install airflow-provider-fivetran-async\n```\n\n## Configuration\n\nIn the Airflow user interface, configure a Connection for Fivetran. Most of the Connection config fields will be left blank. Configure the following fields:\n\n* `Conn Id`: `fivetran`\n* `Conn Type`: `Fivetran`\n* `Login`: Fivetran API Key\n* `Password`: Fivetran API Secret\n\nFind the Fivetran API Key and Secret in the [Fivetran Account Settings](https://fivetran.com/account/settings), under the **API Config** section. See our documentation for more information on [Fivetran API Authentication](https://fivetran.com/docs/rest-api/getting-started#authentication).\n\nThe sensor assumes the `Conn Id` is set to `fivetran`, however if you are managing multiple Fivetran accounts, you can set this to anything you like. See the DAG in examples to see how to specify a custom `Conn Id`.\n\n## Modules\n\n### [Fivetran Operator Async](https://github.com/astronomer/airflow-provider-fivetran-async/tree/main/fivetran_provider_async/operators.py)\n\n```python\nfrom fivetran_provider_async.operators import FivetranOperator\n```\n\n`FivetranOperator` submits a Fivetran sync job and monitors it on trigger for completion.\n\n`FivetranOperator` requires that you specify the `connector_id` of the Fivetran connector you wish to trigger. You can find `connector_id` in the Settings page of the connector you configured in the [Fivetran dashboard](https://fivetran.com/dashboard/connectors).\n\nThe `FivetranOperator` will wait for the sync to complete so long as `wait_for_completion=True` (this is the default). It is recommended that\nyou run in deferrable mode (this is also the default). If `wait_for_completion=False`, the operator will return the timestamp for the last sync.\n\nImport into your DAG via:\n\n### [Fivetran Sensor Async](https://github.com/astronomer/airflow-provider-fivetran-async/tree/main/fivetran_provider_async/sensors.py)\n\n```python\nfrom fivetran_provider_async.sensors import FivetranSensor\n```\n\n`FivetranSensor` monitors a Fivetran sync job for completion.\nMonitoring with `FivetranSensor` allows you to trigger downstream processes only when the Fivetran sync jobs have completed, ensuring data consistency.\n\n`FivetranSensor` requires that you specify the `connector_id` of the Fivetran connector you want to wait for. You can find `connector_id` in the Settings page of the connector you configured in the [Fivetran dashboard](https://fivetran.com/dashboard/connectors).\n\nYou can use multiple instances of `FivetranSensor` to monitor multiple Fivetran connectors.\n\n`FivetranSensor` is most commonly useful in two scenarios:\n\n1. Fivetran is using a separate scheduler than the Airflow scheduler.\n2. You set `wait_for_completion=False` in the `FivetranOperator`, and you need to await the `FivetranOperator` task later. (You may want to do this if you want to arrange your DAG such that some tasks are dependent on _starting_ a sync and other tasks are dependent on _completing_ a sync).\n\nIf you are doing the 1st pattern, you may find it useful to set the `completed_after_time` to `data_interval_end`, or `data_interval_end` with some buffer:\n\n```python\nfivetran_sensor = FivetranSensor(\n task_id=\"wait_for_fivetran_externally_scheduled_sync\",\n connector_id=\"bronzing_largely\",\n poke_interval=5,\n completed_after_time=\"{{ data_interval_end + macros.timedelta(minutes=1) }}\",\n)\n```\n\nIf you are doing the 2nd pattern, you can use XComs to pass the target completed time to the sensor:\n\n```python\nfivetran_op = FivetranOperator(\n task_id=\"fivetran_sync_my_db\",\n connector_id=\"bronzing_largely\",\n wait_for_completion=False,\n)\n\nfivetran_sensor = FivetranSensor(\n task_id=\"wait_for_fivetran_db_sync\",\n connector_id=\"bronzing_largely\",\n poke_interval=5,\n completed_after_time=\"{{ task_instance.xcom_pull('fivetran_sync_op', key='return_value') }}\",\n)\n\nfivetran_op >> fivetran_sensor\n```\n\nYou may also specify the `FivetranSensor` without a `completed_after_time`.\nIn this case, the sensor will make note of when the last completed time was, and will wait for a new completed time.\n\n## Examples\n\nSee the [**examples**](https://github.com/astronomer/airflow-provider-fivetran-async/tree/main/fivetran_provider_async/dev/dags) directory for an example DAG.\n\n## Issues\n\nPlease submit [issues](https://github.com/astronomer/airflow-provider-fivetran-async/issues) and [pull requests](https://github.com/astronomer/airflow-provider-fivetran-async/pulls) in our official repo:\n[https://github.com/astronomer/airflow-provider-fivetran-async](https://github.com/astronomer/airflow-provider-fivetran-async)\n\nWe are happy to hear from you. Please email any feedback to the authors at [humans@astronomer.io](mailto:humans@astronomer.io).\n",
"bugtrack_url": null,
"license": null,
"summary": "A Fivetran async provider for Apache Airflow",
"version": "2.2.0",
"project_urls": {
"Source": "https://github.com/astronomer/airflow-provider-fivetran-async"
},
"split_keywords": [
"airflow",
" apache-airflow",
" astronomer",
" dag",
" provider"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "1fda98d5a539a32553f8cfb320c84560fad86463c7bdd0e5f7201660df5c84de",
"md5": "370898747a52952e797ef6b8656490c5",
"sha256": "776fa97de7870d106ee415a33bdf5eebaac971da98715ae956c768293e05d77a"
},
"downloads": -1,
"filename": "airflow_provider_fivetran_async-2.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "370898747a52952e797ef6b8656490c5",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 25764,
"upload_time": "2025-07-17T12:49:07",
"upload_time_iso_8601": "2025-07-17T12:49:07.851562Z",
"url": "https://files.pythonhosted.org/packages/1f/da/98d5a539a32553f8cfb320c84560fad86463c7bdd0e5f7201660df5c84de/airflow_provider_fivetran_async-2.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "2350c793d7c40c071b2d24542f1737be1893016f77d23079f8b5ad2a8ea52090",
"md5": "41cf135d63fa650a5f73404fd561e414",
"sha256": "e8b44d3548a3813352225fa1b56bce21bdb762dfeddf8fc7ad7e4136e917814f"
},
"downloads": -1,
"filename": "airflow_provider_fivetran_async-2.2.0.tar.gz",
"has_sig": false,
"md5_digest": "41cf135d63fa650a5f73404fd561e414",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 21254,
"upload_time": "2025-07-17T12:49:06",
"upload_time_iso_8601": "2025-07-17T12:49:06.612819Z",
"url": "https://files.pythonhosted.org/packages/23/50/c793d7c40c071b2d24542f1737be1893016f77d23079f8b5ad2a8ea52090/airflow_provider_fivetran_async-2.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-17 12:49:06",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "astronomer",
"github_project": "airflow-provider-fivetran-async",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "airflow-provider-fivetran-async"
}
JSON
