| Name | adoc-airflow-plugin JSON |
| Version |
1.1.0
JSON |
| download |
| home_page | None |
| Summary | Acceldata Airflow Listener Plugin |
| upload_time | 2024-09-11 06:17:17 |
| maintainer | None |
| docs_url | None |
| author | acceldata |
| requires_python | >=3.8 |
| license | None |
| keywords |
acceldata
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# ADOC Listener Plugin
A plugin that integrates [Airflow `DAGs`]() for automatic observation in `ADOC`.
## Features
The plugin performs the following actions without requiring any additional code in your Airflow DAG, unless you disable instrumentation through environment variables.
* When the DAG starts:
* It creates the pipeline if it does not already exist in ADOC.
* It creates a new pipeline run in ADOC.
* When a TaskInstance starts:
* It creates jobs in ADOC for each of the airflow operators used in the task.
* It constructs job input nodes based on the upstream tasks.
* It creates a span and associates it with the jobs.
* It emits span events with metadata.
* When a TaskInstance is completed:
* It emits span events with metadata.
* It ends the spans with either success or failure.
* When the DAG is completed:
* It updates the pipeline run with success or failure in ADOC.
## Requirements
Ensure to have the following applications installed in your system:
- [Python 3.6+](https://www.python.org/downloads)
- [Airflow 2.3+](https://pypi.org/project/apache-airflow)
## Configuration
The adoc_listener_plugin utilizes the acceldata-sdk to push data to the ADOC backend.
### Plugin Environment Variables
`adoc_listener_plugin` uses accedlata-sdk [https://pypi.org/project/acceldata-sdk/] to push data to the ADOC backend.
The adoc client depends on environment variables:
* `TORCH_CATALOG_URL` - URL of the torch catalog
* `TORCH_ACCESS_KEY` - API access key generated from torch UI
* `TORCH_SECRET_KEY` - API secret key generated from torch UI
By default, all the dags will be observed. Below set of environment variables could be used to override this behaviour.
Environment variables to either ignore / observe are mutually exclusive .
Below environment variables if matched with the dag ids, will ignore observation of the matched dag ids. All other dag ids will be observed.
* `DAGIDS_TO_IGNORE` = "Comma separated dag ids to ignore observation"
* `DAGIDS_REGEX_TO_IGNORE` = "Regex for dag ids to ignore observation"
Below environment variables if matched with the dag ids, will be observed observation of dag ids. All other dag ids will be ignored.
* `DAGIDS_TO_OBSERVE` = "Comma separated dag ids to observe"
* `DAGIDS_REGEX_TO_OBSERVE` = "Regex for dag ids to observe"
Raw data
{
"_id": null,
"home_page": null,
"name": "adoc-airflow-plugin",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "acceldata",
"author": "acceldata",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/87/80/3452713b769c0ac5418ee8553fcee8946450330983d77082fccfbaa769ef/adoc_airflow_plugin-1.1.0.tar.gz",
"platform": null,
"description": "# ADOC Listener Plugin\nA plugin that integrates [Airflow `DAGs`]() for automatic observation in `ADOC`.\n\n## Features\nThe plugin performs the following actions without requiring any additional code in your Airflow DAG, unless you disable instrumentation through environment variables.\n* When the DAG starts:\n * It creates the pipeline if it does not already exist in ADOC.\n * It creates a new pipeline run in ADOC.\n\n* When a TaskInstance starts:\n * It creates jobs in ADOC for each of the airflow operators used in the task.\n * It constructs job input nodes based on the upstream tasks.\n * It creates a span and associates it with the jobs.\n * It emits span events with metadata.\n\n* When a TaskInstance is completed:\n * It emits span events with metadata.\n * It ends the spans with either success or failure.\n\n* When the DAG is completed:\n * It updates the pipeline run with success or failure in ADOC.\n\n## Requirements\nEnsure to have the following applications installed in your system:\n- [Python 3.6+](https://www.python.org/downloads)\n- [Airflow 2.3+](https://pypi.org/project/apache-airflow)\n\n## Configuration\nThe adoc_listener_plugin utilizes the acceldata-sdk to push data to the ADOC backend.\n\n### Plugin Environment Variables\n\n`adoc_listener_plugin` uses accedlata-sdk [https://pypi.org/project/acceldata-sdk/] to push data to the ADOC backend.\n\nThe adoc client depends on environment variables:\n\n* `TORCH_CATALOG_URL` - URL of the torch catalog\n* `TORCH_ACCESS_KEY` - API access key generated from torch UI\n* `TORCH_SECRET_KEY` - API secret key generated from torch UI\n\nBy default, all the dags will be observed. Below set of environment variables could be used to override this behaviour.\nEnvironment variables to either ignore / observe are mutually exclusive . \n\nBelow environment variables if matched with the dag ids, will ignore observation of the matched dag ids. All other dag ids will be observed.\n* `DAGIDS_TO_IGNORE` = \"Comma separated dag ids to ignore observation\"\n* `DAGIDS_REGEX_TO_IGNORE` = \"Regex for dag ids to ignore observation\"\n\nBelow environment variables if matched with the dag ids, will be observed observation of dag ids. All other dag ids will be ignored.\n* `DAGIDS_TO_OBSERVE` = \"Comma separated dag ids to observe\"\n* `DAGIDS_REGEX_TO_OBSERVE` = \"Regex for dag ids to observe\"\n",
"bugtrack_url": null,
"license": null,
"summary": "Acceldata Airflow Listener Plugin",
"version": "1.1.0",
"project_urls": null,
"split_keywords": [
"acceldata"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "87803452713b769c0ac5418ee8553fcee8946450330983d77082fccfbaa769ef",
"md5": "264f0d214c4283b9d0813763ed0672f3",
"sha256": "efdc74efd5e4a419725d7b847ff9aa1b537d07b2f8d7603e071d1c6beeff6792"
},
"downloads": -1,
"filename": "adoc_airflow_plugin-1.1.0.tar.gz",
"has_sig": false,
"md5_digest": "264f0d214c4283b9d0813763ed0672f3",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 9528,
"upload_time": "2024-09-11T06:17:17",
"upload_time_iso_8601": "2024-09-11T06:17:17.105446Z",
"url": "https://files.pythonhosted.org/packages/87/80/3452713b769c0ac5418ee8553fcee8946450330983d77082fccfbaa769ef/adoc_airflow_plugin-1.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-11 06:17:17",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "adoc-airflow-plugin"
}
