# CanaryPy Airflow Plugin
This is a plugin for Apache Airflow that provides integration with CanaryPy. It modifies the task policies in Airflow, to send signals to CanaryPy on task success or failure. It also adds a custom Jinja template function that can be used to get the latest stable version of an artifact.
## Requirements
- Python 3.8+
- Apache Airflow 2.0+
- [CanaryPy](https://pypi.org/project/canarypy/) Library
## Installation
This plugin is available in Pypi so you can add to your Airflow installation by running the following command:
```
pip install canarypy_airflow_plugin
```
Alternatively, you can add the plugin to Airflow dependencies file.
Also disable lazy loading of plugins by setting `lazy_load_plugins` to `False` in your Airflow configuration file:
```ini
[core]
lazy_load_plugins = False
```
## Configuration
The plugin uses the environment variable `CANARYPY_URL` to set the base URL for CanaryPy. This environment variable must be set before the plugin is loaded by Airflow.
## How It Works
The plugin patches the `task_policy` function in Airflow's settings. If an existing `task_policy` function exists, it is wrapped, so that the original policy is applied first, followed by the policy provided by this plugin.
The plugin's policy adds custom `on_failure_callback` and `on_success_callback` functions to the tasks. These functions send a signal to CanaryPy, containing information about the task's image, run ID, task ID, and state.
If a task is a `MappedOperator`, the policy applies the custom callbacks via the `partial_kwargs` attribute, to work around the fact that these operators do not have `on_failure_callback` and `on_success_callback` properties.
Finally, the plugin adds a `get_latest_stable_version` function to the Jinja environment, which can be used in your DAGs. This function retrieves the latest stable version of an artifact from CanaryPy, given the artifact's URL.
## Usage
With this plugin installed, your tasks in Airflow will automatically send signals to CanaryPy when they start, succeed or fail. This happens without needing any changes to your DAGs.
To use the `get_latest_stable_version` function in a Jinja template, use the following syntax:
```
{{ get_latest_stable_version('artifact_url') }}
```
## Contributing
We welcome contributions! Please feel free to submit a Pull Request.
Raw data
{
"_id": null,
"home_page": "https://github.com/thcidale0808/canarypy",
"name": "canarypy-airflow-plugin",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "",
"author": "Thiago Assumpcao",
"author_email": "thcidale@gmail.com",
"download_url": "",
"platform": null,
"description": "# CanaryPy Airflow Plugin\n\nThis is a plugin for Apache Airflow that provides integration with CanaryPy. It modifies the task policies in Airflow, to send signals to CanaryPy on task success or failure. It also adds a custom Jinja template function that can be used to get the latest stable version of an artifact.\n\n## Requirements\n\n- Python 3.8+\n- Apache Airflow 2.0+\n- [CanaryPy](https://pypi.org/project/canarypy/) Library\n\n## Installation\n\nThis plugin is available in Pypi so you can add to your Airflow installation by running the following command:\n\n```\npip install canarypy_airflow_plugin\n```\n\nAlternatively, you can add the plugin to Airflow dependencies file.\n\nAlso disable lazy loading of plugins by setting `lazy_load_plugins` to `False` in your Airflow configuration file:\n\n```ini\n[core]\nlazy_load_plugins = False\n```\n\n## Configuration\n\nThe plugin uses the environment variable `CANARYPY_URL` to set the base URL for CanaryPy. This environment variable must be set before the plugin is loaded by Airflow.\n\n## How It Works\n\nThe plugin patches the `task_policy` function in Airflow's settings. If an existing `task_policy` function exists, it is wrapped, so that the original policy is applied first, followed by the policy provided by this plugin.\n\nThe plugin's policy adds custom `on_failure_callback` and `on_success_callback` functions to the tasks. These functions send a signal to CanaryPy, containing information about the task's image, run ID, task ID, and state.\n\nIf a task is a `MappedOperator`, the policy applies the custom callbacks via the `partial_kwargs` attribute, to work around the fact that these operators do not have `on_failure_callback` and `on_success_callback` properties.\n\nFinally, the plugin adds a `get_latest_stable_version` function to the Jinja environment, which can be used in your DAGs. This function retrieves the latest stable version of an artifact from CanaryPy, given the artifact's URL.\n\n## Usage\n\nWith this plugin installed, your tasks in Airflow will automatically send signals to CanaryPy when they start, succeed or fail. This happens without needing any changes to your DAGs.\n\nTo use the `get_latest_stable_version` function in a Jinja template, use the following syntax:\n\n```\n{{ get_latest_stable_version('artifact_url') }}\n```\n\n## Contributing\n\nWe welcome contributions! Please feel free to submit a Pull Request.\n\n",
"bugtrack_url": null,
"license": "",
"summary": "CanaryPy Airflow Plugin - Airflow Integration Plugin for CanaryPy",
"version": "0.0.1",
"project_urls": {
"Bug Reports": "https://github.com/thcidale0808/canarypy/issues",
"Homepage": "https://github.com/thcidale0808/canarypy",
"Source": "https://github.com/thcidale0808/canarypy/"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "52a515c908ad22dc3c07975ac8b81a37d39db269fa19ddd1754a1f5db2473a09",
"md5": "14f9c3320806af39caa9a15186d428e6",
"sha256": "f0ab39090d50c6cc8805cc7d23f21b72dc1ef828a1760b9f6b8c350974efd70b"
},
"downloads": -1,
"filename": "canarypy_airflow_plugin-0.0.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "14f9c3320806af39caa9a15186d428e6",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 4206,
"upload_time": "2023-07-02T18:08:16",
"upload_time_iso_8601": "2023-07-02T18:08:16.989882Z",
"url": "https://files.pythonhosted.org/packages/52/a5/15c908ad22dc3c07975ac8b81a37d39db269fa19ddd1754a1f5db2473a09/canarypy_airflow_plugin-0.0.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-07-02 18:08:16",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "thcidale0808",
"github_project": "canarypy",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"test_requirements": [],
"lcname": "canarypy-airflow-plugin"
}
JSON
