# dag-dq-generator
DPaaS Airflow DAG (Dynamic Acyclic Graph) and DQ (Data Quality) generator.
*dag-dq-generator* is a DPaaS [Apache Airflow](https://github.com/apache/incubator-airflow) Airflow DAG (Dynamic Acyclic Graph) and DQ (Data Quality) generator from YAML configuration files.
- [Usage](#usage)
- [YAML Definition](#yaml-definition)
- [Benefits](#benefits)
- [Contributing](#contributing)
## Usage
### Setup
*dag-dq-generator* requires Python 3.6.0+. To set up your environment, you can run `sh build.sh` which installs the required Python packages and run the generator program. Otherwise, you can run `pip install -r requirements.txt` to install the required Python packages and run `python dag_generator.py` with the following parameters:
* `--config-path` defines the path to the configurations folder. Defaults to `./configs/`
* `--dag-storage-path` defines the path to the folder where generated DAGs will be stored. Defaults to `./dags/`
* `--dq-storage-path` defines the path to the folder where DQ SQL files will be stored. Defaults to `./sql/`
## YAML Definition
## Benefits
* Construct DAGs without knowing Python
* Construct DAGs without learning Airflow primitives
* Avoid duplicative code
## Contributing
Contributions are welcome! Just submit a Pull Request or Github Issue. Feel free to join the discussions on the `#dag-dq-generator` Slack channel.
Raw data
{
"_id": null,
"home_page": "https://git.corp.adobe.com/ccea/dag-dq-generator",
"name": "dag-dq-generator",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "airflow dag data-quality",
"author": "CI DMe Data Engineering",
"author_email": "ccea-data-engineering@adobe.com",
"download_url": "https://files.pythonhosted.org/packages/ea/8a/ff3816b7e126a77b0726cb5a426505a78dbf2da35d25d87cef0c2436dde3/dag-dq-generator-1.0.5.tar.gz",
"platform": null,
"description": "# dag-dq-generator\nDPaaS Airflow DAG (Dynamic Acyclic Graph) and DQ (Data Quality) generator.\n\n*dag-dq-generator* is a DPaaS [Apache Airflow](https://github.com/apache/incubator-airflow) Airflow DAG (Dynamic Acyclic Graph) and DQ (Data Quality) generator from YAML configuration files.\n- [Usage](#usage)\n- [YAML Definition](#yaml-definition)\n- [Benefits](#benefits)\n- [Contributing](#contributing)\n\n## Usage\n### Setup\n*dag-dq-generator* requires Python 3.6.0+. To set up your environment, you can run `sh build.sh` which installs the required Python packages and run the generator program. Otherwise, you can run `pip install -r requirements.txt` to install the required Python packages and run `python dag_generator.py` with the following parameters:\n* `--config-path` defines the path to the configurations folder. Defaults to `./configs/`\n* `--dag-storage-path` defines the path to the folder where generated DAGs will be stored. Defaults to `./dags/`\n* `--dq-storage-path` defines the path to the folder where DQ SQL files will be stored. Defaults to `./sql/`\n\n## YAML Definition\n\n## Benefits\n\n* Construct DAGs without knowing Python\n* Construct DAGs without learning Airflow primitives\n* Avoid duplicative code\n\n## Contributing\n\nContributions are welcome! Just submit a Pull Request or Github Issue. Feel free to join the discussions on the `#dag-dq-generator` Slack channel.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "DPaaS Airflow DAG (Dynamic Acyclic Graph) and DQ (Data Quality) generator",
"version": "1.0.5",
"project_urls": {
"Homepage": "https://git.corp.adobe.com/ccea/dag-dq-generator"
},
"split_keywords": [
"airflow",
"dag",
"data-quality"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ea8aff3816b7e126a77b0726cb5a426505a78dbf2da35d25d87cef0c2436dde3",
"md5": "50f78b4e2195a7a2ee7a2b58cd36fb6c",
"sha256": "52d9f8d6ef1bc4a895dc8e3834e3aee4dc54e120a4631c4a9b925362b70178dc"
},
"downloads": -1,
"filename": "dag-dq-generator-1.0.5.tar.gz",
"has_sig": false,
"md5_digest": "50f78b4e2195a7a2ee7a2b58cd36fb6c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 28541,
"upload_time": "2023-12-06T16:52:20",
"upload_time_iso_8601": "2023-12-06T16:52:20.390359Z",
"url": "https://files.pythonhosted.org/packages/ea/8a/ff3816b7e126a77b0726cb5a426505a78dbf2da35d25d87cef0c2436dde3/dag-dq-generator-1.0.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-12-06 16:52:20",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "dag-dq-generator"
}
JSON