# airflow-dagfactory
[](https://github.com/ajbosco/dag-factory/actions?workflow=build)
[](https://codecov.io/github/ajbosco/dag-factory?branch=master)
[](https://pypi.org/project/dag-factory/)
[](https://github.com/ambv/black)
[](https://pepy.tech/project/dag-factory)
*dag-factory* is a library for dynamically generating [Apache Airflow](https://github.com/apache/incubator-airflow) DAGs from YAML configuration files.
- [Installation](#installation)
- [Usage](#usage)
- [Benefits](#benefits)
- [Contributing](#contributing)
## Installation
To install *dag-factory* run `pip install airflow-dagfactory`. It requires Python 3.10.0+ and Apache Airflow 2.9.1+.
## Usage
After installing *airflow-dagfactory* in your Airflow environment, there are two steps to creating DAGs. First, we need to create a YAML configuration file. For example:
```yaml
example_dag1:
default_args:
owner: 'example_owner'
start_date: 2018-01-01 # or '2 days'
end_date: 2018-01-05
retries: 1
retry_delay_sec: 300
schedule_interval: '0 3 * * *'
concurrency: 1
max_active_runs: 1
dagrun_timeout_sec: 60
default_view: 'tree' # or 'graph', 'duration', 'gantt', 'landing_times'
orientation: 'LR' # or 'TB', 'RL', 'BT'
description: 'this is an example dag!'
on_success_callback_name: print_hello
on_success_callback_file: /usr/local/airflow/dags/print_hello.py
on_failure_callback_name: print_hello
on_failure_callback_file: /usr/local/airflow/dags/print_hello.py
tasks:
task_1:
operator: airflow.operators.bash_operator.BashOperator
bash_command: 'echo 1'
task_2:
operator: airflow.operators.bash_operator.BashOperator
bash_command: 'echo 2'
dependencies: [task_1]
task_3:
operator: airflow.operators.bash_operator.BashOperator
bash_command: 'echo 3'
dependencies: [task_1]
```
Then in the DAGs folder in your Airflow environment you need to create a python file like this:
```python
from airflow import DAG
import airflow_dagfactory
dag_factory = airflow_dagfactory.DagFactory("/path/to/dags/config_file.yml")
dag_factory.clean_dags(globals())
dag_factory.generate_dags(globals())
```
And this DAG will be generated and ready to run in Airflow!
If you have several configuration files you can import them like this:
```python
# 'airflow' word is required for the dagbag to parse this file
from airflow_dagfactory import load_yaml_dags
load_yaml_dags(globals_dict=globals(), suffix=['dag.yaml'])
```
## Notes
### HttpSensor (since 0.10.0)
The package `airflow.sensors.http_sensor` works with all supported versions of Airflow. In Airflow 2.0+, the new package name can be used in the operator value: `airflow.providers.http.sensors.http`
The following example shows `response_check` logic in a python file:
```yaml
task_2:
operator: airflow.sensors.http_sensor.HttpSensor
http_conn_id: 'test-http'
method: 'GET'
response_check_name: check_sensor
response_check_file: /path/to/example1/http_conn.py
dependencies: [task_1]
```
The `response_check` logic can also be provided as a lambda:
```yaml
task_2:
operator: airflow.sensors.http_sensor.HttpSensor
http_conn_id: 'test-http'
method: 'GET'
response_check_lambda: 'lambda response: "ok" in reponse.text'
dependencies: [task_1]
```
## Benefits
* Construct DAGs without knowing Python
* Construct DAGs without learning Airflow primitives
* Avoid duplicative code
* Everyone loves YAML! ;)
## Contributing
Contributions are welcome! Just submit a Pull Request or Github Issue.
**Upload Pypi**
`python setup.py sdist bdist_wheel `
`python -m twine upload dist/* --verbose`
Raw data
{
"_id": null,
"home_page": "https://github.com/quydx/dag-factory",
"name": "airflow-dagfactory",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10.0",
"maintainer_email": null,
"keywords": "airflow",
"author": "Quy Do",
"author_email": "quybulu@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/02/5c/29b6fed60955b69306bafe5d4e83b2d5eb426f402e22259bfc2cf2d1b0c9/airflow-dagfactory-0.19.2.tar.gz",
"platform": null,
"description": "\n# airflow-dagfactory\n\n[](https://github.com/ajbosco/dag-factory/actions?workflow=build)\n[](https://codecov.io/github/ajbosco/dag-factory?branch=master)\n[](https://pypi.org/project/dag-factory/)\n[](https://github.com/ambv/black)\n[](https://pepy.tech/project/dag-factory)\n\n*dag-factory* is a library for dynamically generating [Apache Airflow](https://github.com/apache/incubator-airflow) DAGs from YAML configuration files.\n- [Installation](#installation)\n- [Usage](#usage)\n- [Benefits](#benefits)\n- [Contributing](#contributing)\n \n## Installation\n\nTo install *dag-factory* run `pip install airflow-dagfactory`. It requires Python 3.10.0+ and Apache Airflow 2.9.1+.\n\n## Usage\n\nAfter installing *airflow-dagfactory* in your Airflow environment, there are two steps to creating DAGs. First, we need to create a YAML configuration file. For example:\n\n```yaml\nexample_dag1:\n default_args:\n owner: 'example_owner'\n start_date: 2018-01-01 # or '2 days'\n end_date: 2018-01-05\n retries: 1\n retry_delay_sec: 300\n schedule_interval: '0 3 * * *'\n concurrency: 1\n max_active_runs: 1\n dagrun_timeout_sec: 60\n default_view: 'tree' # or 'graph', 'duration', 'gantt', 'landing_times'\n orientation: 'LR' # or 'TB', 'RL', 'BT'\n description: 'this is an example dag!'\n on_success_callback_name: print_hello\n on_success_callback_file: /usr/local/airflow/dags/print_hello.py\n on_failure_callback_name: print_hello\n on_failure_callback_file: /usr/local/airflow/dags/print_hello.py\n tasks:\n task_1:\n operator: airflow.operators.bash_operator.BashOperator\n bash_command: 'echo 1'\n task_2:\n operator: airflow.operators.bash_operator.BashOperator\n bash_command: 'echo 2'\n dependencies: [task_1]\n task_3:\n operator: airflow.operators.bash_operator.BashOperator\n bash_command: 'echo 3'\n dependencies: [task_1]\n```\n\nThen in the DAGs folder in your Airflow environment you need to create a python file like this:\n\n```python\nfrom airflow import DAG\nimport airflow_dagfactory\n\ndag_factory = airflow_dagfactory.DagFactory(\"/path/to/dags/config_file.yml\")\n\ndag_factory.clean_dags(globals())\ndag_factory.generate_dags(globals())\n```\n\nAnd this DAG will be generated and ready to run in Airflow!\n\nIf you have several configuration files you can import them like this:\n\n```python\n# 'airflow' word is required for the dagbag to parse this file\nfrom airflow_dagfactory import load_yaml_dags\n\nload_yaml_dags(globals_dict=globals(), suffix=['dag.yaml'])\n```\n\n\n\n## Notes\n\n### HttpSensor (since 0.10.0)\n\nThe package `airflow.sensors.http_sensor` works with all supported versions of Airflow. In Airflow 2.0+, the new package name can be used in the operator value: `airflow.providers.http.sensors.http`\n\nThe following example shows `response_check` logic in a python file:\n\n```yaml\ntask_2:\n operator: airflow.sensors.http_sensor.HttpSensor\n http_conn_id: 'test-http'\n method: 'GET'\n response_check_name: check_sensor\n response_check_file: /path/to/example1/http_conn.py\n dependencies: [task_1]\n```\n\nThe `response_check` logic can also be provided as a lambda:\n\n```yaml\ntask_2:\n operator: airflow.sensors.http_sensor.HttpSensor\n http_conn_id: 'test-http'\n method: 'GET'\n response_check_lambda: 'lambda response: \"ok\" in reponse.text'\n dependencies: [task_1]\n```\n\n## Benefits\n\n* Construct DAGs without knowing Python\n* Construct DAGs without learning Airflow primitives\n* Avoid duplicative code\n* Everyone loves YAML! ;)\n\n## Contributing\n\nContributions are welcome! Just submit a Pull Request or Github Issue.\n\n\n**Upload Pypi** \n`python setup.py sdist bdist_wheel ` \n`python -m twine upload dist/* --verbose`\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Dynamically build Airflow DAGs from YAML files",
"version": "0.19.2",
"project_urls": {
"Homepage": "https://github.com/quydx/dag-factory"
},
"split_keywords": [
"airflow"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "35cc35c4ee1a23014c80e8271e7f618bbd58cacebbd39b66d571c314f60af816",
"md5": "cb2a608bb7f0065147bed74dafebd486",
"sha256": "7a14b1fea32bb48f7968019bdaeb78aa64fba33be5d069cc7bd22d4d48695c65"
},
"downloads": -1,
"filename": "airflow_dagfactory-0.19.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "cb2a608bb7f0065147bed74dafebd486",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10.0",
"size": 17284,
"upload_time": "2024-05-07T05:22:10",
"upload_time_iso_8601": "2024-05-07T05:22:10.520567Z",
"url": "https://files.pythonhosted.org/packages/35/cc/35c4ee1a23014c80e8271e7f618bbd58cacebbd39b66d571c314f60af816/airflow_dagfactory-0.19.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "025c29b6fed60955b69306bafe5d4e83b2d5eb426f402e22259bfc2cf2d1b0c9",
"md5": "3ab60489f946ccc23fa145b32271f8d1",
"sha256": "aa728638f63abb97dae03777384bd5c5249c95ab0c95118647ad6362d8a5281f"
},
"downloads": -1,
"filename": "airflow-dagfactory-0.19.2.tar.gz",
"has_sig": false,
"md5_digest": "3ab60489f946ccc23fa145b32271f8d1",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10.0",
"size": 18318,
"upload_time": "2024-05-07T05:22:13",
"upload_time_iso_8601": "2024-05-07T05:22:13.416935Z",
"url": "https://files.pythonhosted.org/packages/02/5c/29b6fed60955b69306bafe5d4e83b2d5eb426f402e22259bfc2cf2d1b0c9/airflow-dagfactory-0.19.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-07 05:22:13",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "quydx",
"github_project": "dag-factory",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "airflow-dagfactory"
}
JSON
