| Name | adoc-airflow-plugin JSON |
| Version |
2.0.0
JSON |
| download |
| home_page | None |
| Summary | Acceldata Airflow Listener Plugin |
| upload_time | 2024-10-03 11:17:58 |
| maintainer | None |
| docs_url | None |
| author | acceldata |
| requires_python | >=3.8 |
| license | None |
| keywords |
acceldata
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
## Overview
The Acceldata Listener plugin integrates Airflow DAGs for automatic observation in ADOC.
## Features
The plugin performs the following actions without requiring additional code in your Airflow DAG, unless you disable instrumentation through environment variables.
- **When the DAG starts:**
- It creates the pipeline if it does not already exist in ADOC.
- It creates a new pipeline run in ADOC.
- **When a TaskInstance starts:**
- It creates jobs in ADOC for each of the Airflow operators used in the task.
- It constructs job input nodes based on the upstream tasks.
- It creates a span and associates it with the jobs.
- It emits span events with metadata.
- **When a TaskInstance is completed:**
- It emits span events with metadata.
- It ends the spans with either success or failure.
- **When the DAG is completed:**
- It updates the pipeline run with success or failure in ADOC.
## Prerequisites
Ensure the following applications are installed on your system:
- Python V3.8.0 and above ([Download Python](https://www.python.org/downloads/))
- Airflow V2.5.0 and above ([Apache Airflow](https://airflow.apache.org/docs/apache-airflow/stable/installation.html))
API keys are essential for authentication when making calls to ADOC. You can generate API keys in the ADOC UI's Admin Central by visiting the [API Keys](https://docs.acceldata.io/documentation/api-keys) section.
## Configuration
### Plugin Environment Variables
The `adoc_airflow_plugin` uses the `acceldata-sdk` to push data to the ADOC backend.
**Mandatory Environment Variables:**
The ADOC client requires the following environment variables:
- `TORCH_CATALOG_URL`: The URL of your ADOC Server instance.
- `TORCH_ACCESS_KEY`: The API access key generated from the ADOC UI.
- `TORCH_SECRET_KEY`: The API secret key generated from the ADOC UI.
**Optional Environment Variables:**
By default, all DAGs are observed. However, the following environment variables can be set to modify this behavior.
**Note:** The variables for ignoring or observing DAGs are mutually exclusive.
- If the following environment variables match specific DAG IDs, those DAGs will be ignored from observation, while all other DAGs will still be observed:
- `DAGIDS_TO_IGNORE`: Comma-separated list of DAG IDs to ignore.
- `DAGIDS_REGEX_TO_IGNORE`: Regular expression pattern for DAG IDs to ignore.
- If the following environment variables match specific DAG IDs, only those DAGs will be observed, and all others will be ignored:
- `DAGIDS_TO_OBSERVE`: Comma-separated list of DAG IDs to observe.
- `DAGIDS_REGEX_TO_OBSERVE`: Regular expression pattern for DAG IDs to observe.
Raw data
{
"_id": null,
"home_page": null,
"name": "adoc-airflow-plugin",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "acceldata",
"author": "acceldata",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/a9/02/4cd853b2a2ae38880377654857a012ac2d697ef6085b8b6e44192ef95344/adoc_airflow_plugin-2.0.0.tar.gz",
"platform": null,
"description": "## Overview\nThe Acceldata Listener plugin integrates Airflow DAGs for automatic observation in ADOC.\n\n## Features\nThe plugin performs the following actions without requiring additional code in your Airflow DAG, unless you disable instrumentation through environment variables.\n\n- **When the DAG starts:**\n - It creates the pipeline if it does not already exist in ADOC.\n - It creates a new pipeline run in ADOC.\n\n- **When a TaskInstance starts:**\n - It creates jobs in ADOC for each of the Airflow operators used in the task.\n - It constructs job input nodes based on the upstream tasks.\n - It creates a span and associates it with the jobs.\n - It emits span events with metadata.\n\n- **When a TaskInstance is completed:**\n - It emits span events with metadata.\n - It ends the spans with either success or failure.\n\n- **When the DAG is completed:**\n - It updates the pipeline run with success or failure in ADOC.\n\n## Prerequisites\nEnsure the following applications are installed on your system:\n- Python V3.8.0 and above ([Download Python](https://www.python.org/downloads/))\n- Airflow V2.5.0 and above ([Apache Airflow](https://airflow.apache.org/docs/apache-airflow/stable/installation.html))\n\nAPI keys are essential for authentication when making calls to ADOC. You can generate API keys in the ADOC UI's Admin Central by visiting the [API Keys](https://docs.acceldata.io/documentation/api-keys) section.\n\n## Configuration\n\n### Plugin Environment Variables\nThe `adoc_airflow_plugin` uses the `acceldata-sdk` to push data to the ADOC backend.\n\n**Mandatory Environment Variables:**\nThe ADOC client requires the following environment variables:\n- `TORCH_CATALOG_URL`: The URL of your ADOC Server instance.\n- `TORCH_ACCESS_KEY`: The API access key generated from the ADOC UI.\n- `TORCH_SECRET_KEY`: The API secret key generated from the ADOC UI.\n\n**Optional Environment Variables:**\nBy default, all DAGs are observed. However, the following environment variables can be set to modify this behavior.\n\n**Note:** The variables for ignoring or observing DAGs are mutually exclusive.\n\n- If the following environment variables match specific DAG IDs, those DAGs will be ignored from observation, while all other DAGs will still be observed:\n - `DAGIDS_TO_IGNORE`: Comma-separated list of DAG IDs to ignore.\n - `DAGIDS_REGEX_TO_IGNORE`: Regular expression pattern for DAG IDs to ignore.\n\n- If the following environment variables match specific DAG IDs, only those DAGs will be observed, and all others will be ignored:\n - `DAGIDS_TO_OBSERVE`: Comma-separated list of DAG IDs to observe.\n - `DAGIDS_REGEX_TO_OBSERVE`: Regular expression pattern for DAG IDs to observe.\n",
"bugtrack_url": null,
"license": null,
"summary": "Acceldata Airflow Listener Plugin",
"version": "2.0.0",
"project_urls": null,
"split_keywords": [
"acceldata"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a9024cd853b2a2ae38880377654857a012ac2d697ef6085b8b6e44192ef95344",
"md5": "cbdf1383119aeee0fab9a1a0772e2f0f",
"sha256": "eecdc98039cad60c95303aaab8674465c40c1cf656714aa096e9f78b9ce08e8a"
},
"downloads": -1,
"filename": "adoc_airflow_plugin-2.0.0.tar.gz",
"has_sig": false,
"md5_digest": "cbdf1383119aeee0fab9a1a0772e2f0f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 10100,
"upload_time": "2024-10-03T11:17:58",
"upload_time_iso_8601": "2024-10-03T11:17:58.810065Z",
"url": "https://files.pythonhosted.org/packages/a9/02/4cd853b2a2ae38880377654857a012ac2d697ef6085b8b6e44192ef95344/adoc_airflow_plugin-2.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-03 11:17:58",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "adoc-airflow-plugin"
}
