| Name | dlt-meta JSON |
| Version |
0.0.8
JSON |
| download |
| home_page | None |
| Summary | DLT-META Framework |
| upload_time | 2024-08-06 23:22:37 |
| maintainer | None |
| docs_url | None |
| author | Ravi Gawai |
| requires_python | >=3.8 |
| license | Databricks License |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# DLT-META
<!-- Top bar will be removed from PyPi packaged versions -->
<!-- Dont remove: exclude package -->
[Documentation](https://databrickslabs.github.io/dlt-meta/) |
[Release Notes](CHANGELOG.md) |
[Examples](https://github.com/databrickslabs/dlt-meta/tree/main/examples)
<!-- Dont remove: end exclude package -->
---
<p align="left">
<a href="https://databrickslabs.github.io/dlt-meta/">
<img src="https://img.shields.io/badge/DOCS-PASSING-green?style=for-the-badge" alt="Documentation Status"/>
</a>
<a href="https://pypi.org/project/dlt-meta/">
<img src="https://img.shields.io/badge/PYPI-v%200.0.7-green?style=for-the-badge" alt="Latest Python Release"/>
</a>
<a href="https://github.com/databrickslabs/dlt-meta/actions/workflows/onpush.yml">
<img src="https://img.shields.io/github/workflow/status/databrickslabs/dlt-meta/build/main?style=for-the-badge"
alt="GitHub Workflow Status (branch)"/>
</a>
<a href="https://codecov.io/gh/databrickslabs/dlt-meta">
<img src="https://img.shields.io/codecov/c/github/databrickslabs/dlt-meta?style=for-the-badge&token=2CxLj3YBam"
alt="codecov"/>
</a>
<a href="https://pypistats.org/packages/dl-meta">
<img src="https://img.shields.io/pypi/dm/dlt-meta?style=for-the-badge" alt="downloads"/>
</a>
<a href="https://github.com/PyCQA/flake8">
<img src="https://img.shields.io/badge/FLAKE8-FLAKE8-lightgrey?style=for-the-badge"
alt="We use flake8 for formatting"/>
</a>
</p>
[](<[https://codecov.io/github/databrickslabs/dlt-meta](https://github.com/databrickslabs/dlt-meta)>)
---
# Project Overview
`DLT-META` is a metadata-driven framework designed to work with [Delta Live Tables](https://www.databricks.com/product/delta-live-tables). This framework enables the automation of bronze and silver data pipelines by leveraging metadata recorded in an onboarding JSON file. This file, known as the Dataflowspec, serves as the data flow specification, detailing the source and target metadata required for the pipelines.
In practice, a single generic DLT pipeline reads the Dataflowspec and uses it to orchestrate and run the necessary data processing workloads. This approach streamlines the development and management of data pipelines, allowing for a more efficient and scalable data processing workflow
### Components:
#### Metadata Interface
- Capture input/output metadata in [onboarding file](https://github.com/databrickslabs/dlt-meta/blob/main/examples/onboarding.json)
- Capture [Data Quality Rules](https://github.com/databrickslabs/dlt-meta/tree/main/examples/dqe/customers/bronze_data_quality_expectations.json)
- Capture processing logic as sql in [Silver transformation file](https://github.com/databrickslabs/dlt-meta/blob/main/examples/silver_transformations.json)
#### Generic DLT pipeline
- Apply appropriate readers based on input metadata
- Apply data quality rules with DLT expectations
- Apply CDC apply changes if specified in metadata
- Builds DLT graph based on input/output metadata
- Launch DLT pipeline
## High-Level Process Flow:
## Steps
## Getting Started
Refer to the [Getting Started](https://databrickslabs.github.io/dlt-meta/getting_started)
### Databricks Labs DLT-META CLI lets you run onboard and deploy in interactive python terminal
#### pre-requisites:
- Python 3.8.0 +
- Databricks CLI v0.213 or later. See [instructions](https://docs.databricks.com/en/dev-tools/cli/tutorial.html)
- Install Databricks CLI on macOS:
- 
- Install Databricks CLI on Windows:
- 
Once you install Databricks CLI, authenticate your current machine to a Databricks Workspace:
```commandline
databricks auth login --host WORKSPACE_HOST
```
To enable debug logs, simply add `--debug` flag to any command.
### Installing dlt-meta:
- Install dlt-meta via Databricks CLI:
```commandline
databricks labs install dlt-meta
```
### Onboard using dlt-meta CLI:
If you want to run existing demo files please follow these steps before running onboard command:
```commandline
git clone https://github.com/databrickslabs/dlt-meta.git
```
```commandline
cd dlt-meta
```
```commandline
python -m venv .venv
```
```commandline
source .venv/bin/activate
```
```commandline
pip install databricks-sdk
```
```commandline
dlt_meta_home=$(pwd)
```
```commandline
export PYTHONPATH=$dlt_meta_home
```
```commandline
databricks labs dlt-meta onboard
```
Above commands will prompt you to provide onboarding details. If you have cloned dlt-meta git repo then accept defaults which will launch config from demo folder.
- Goto your databricks workspace and located onboarding job under: Workflow->Jobs runs
### depoly using dlt-meta CLI:
- Once onboarding jobs is finished deploy `bronze` and `silver` DLT using below command
- ```commandline
databricks labs dlt-meta deploy
```
- - Above command will prompt you to provide dlt details. Please provide respective details for schema which you provided in above steps
- - Bronze DLT
- Silver DLT
- - ```commandline
databricks labs dlt-meta deploy
```
- - Above command will prompt you to provide dlt details. Please provide respective details for schema which you provided in above steps
## More questions
Refer to the [FAQ](https://databrickslabs.github.io/dlt-meta/faq)
and DLT-META [documentation](https://databrickslabs.github.io/dlt-meta/)
# Project Support
Please note that all projects released under [`Databricks Labs`](https://www.databricks.com/learn/labs)
are provided for your exploration only, and are not formally supported by Databricks with Service Level Agreements
(SLAs). They are provided AS-IS and we do not make any guarantees of any kind. Please do not submit a support ticket
relating to any issues arising from the use of these projects.
Any issues discovered through the use of this project should be filed as issues on the Github Repo.
They will be reviewed as time permits, but there are no formal SLAs for support.
Raw data
{
"_id": null,
"home_page": null,
"name": "dlt-meta",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": null,
"author": "Ravi Gawai",
"author_email": "databrickslabs@databricks.com",
"download_url": null,
"platform": null,
"description": "# DLT-META\n\n<!-- Top bar will be removed from PyPi packaged versions -->\n<!-- Dont remove: exclude package -->\n\n[Documentation](https://databrickslabs.github.io/dlt-meta/) |\n[Release Notes](CHANGELOG.md) |\n[Examples](https://github.com/databrickslabs/dlt-meta/tree/main/examples)\n\n<!-- Dont remove: end exclude package -->\n\n---\n\n<p align=\"left\">\n <a href=\"https://databrickslabs.github.io/dlt-meta/\">\n <img src=\"https://img.shields.io/badge/DOCS-PASSING-green?style=for-the-badge\" alt=\"Documentation Status\"/>\n </a>\n <a href=\"https://pypi.org/project/dlt-meta/\">\n <img src=\"https://img.shields.io/badge/PYPI-v%200.0.7-green?style=for-the-badge\" alt=\"Latest Python Release\"/>\n </a>\n <a href=\"https://github.com/databrickslabs/dlt-meta/actions/workflows/onpush.yml\">\n <img src=\"https://img.shields.io/github/workflow/status/databrickslabs/dlt-meta/build/main?style=for-the-badge\"\n alt=\"GitHub Workflow Status (branch)\"/>\n </a>\n <a href=\"https://codecov.io/gh/databrickslabs/dlt-meta\">\n <img src=\"https://img.shields.io/codecov/c/github/databrickslabs/dlt-meta?style=for-the-badge&token=2CxLj3YBam\"\n alt=\"codecov\"/>\n </a>\n <a href=\"https://pypistats.org/packages/dl-meta\">\n <img src=\"https://img.shields.io/pypi/dm/dlt-meta?style=for-the-badge\" alt=\"downloads\"/>\n </a>\n <a href=\"https://github.com/PyCQA/flake8\">\n <img src=\"https://img.shields.io/badge/FLAKE8-FLAKE8-lightgrey?style=for-the-badge\"\n alt=\"We use flake8 for formatting\"/>\n </a>\n</p>\n\n[](<[https://codecov.io/github/databrickslabs/dlt-meta](https://github.com/databrickslabs/dlt-meta)>)\n\n---\n\n# Project Overview\n`DLT-META` is a metadata-driven framework designed to work with [Delta Live Tables](https://www.databricks.com/product/delta-live-tables). This framework enables the automation of bronze and silver data pipelines by leveraging metadata recorded in an onboarding JSON file. This file, known as the Dataflowspec, serves as the data flow specification, detailing the source and target metadata required for the pipelines.\n\nIn practice, a single generic DLT pipeline reads the Dataflowspec and uses it to orchestrate and run the necessary data processing workloads. This approach streamlines the development and management of data pipelines, allowing for a more efficient and scalable data processing workflow\n\n### Components:\n\n#### Metadata Interface\n\n- Capture input/output metadata in [onboarding file](https://github.com/databrickslabs/dlt-meta/blob/main/examples/onboarding.json)\n- Capture [Data Quality Rules](https://github.com/databrickslabs/dlt-meta/tree/main/examples/dqe/customers/bronze_data_quality_expectations.json)\n- Capture processing logic as sql in [Silver transformation file](https://github.com/databrickslabs/dlt-meta/blob/main/examples/silver_transformations.json)\n\n#### Generic DLT pipeline\n\n- Apply appropriate readers based on input metadata\n- Apply data quality rules with DLT expectations\n- Apply CDC apply changes if specified in metadata\n- Builds DLT graph based on input/output metadata\n- Launch DLT pipeline\n\n## High-Level Process Flow:\n\n\n\n## Steps\n\n\n\n## Getting Started\n\nRefer to the [Getting Started](https://databrickslabs.github.io/dlt-meta/getting_started)\n\n### Databricks Labs DLT-META CLI lets you run onboard and deploy in interactive python terminal\n\n#### pre-requisites:\n\n- Python 3.8.0 +\n\n- Databricks CLI v0.213 or later. See [instructions](https://docs.databricks.com/en/dev-tools/cli/tutorial.html)\n\n- Install Databricks CLI on macOS:\n- \n\n- Install Databricks CLI on Windows:\n- \n\nOnce you install Databricks CLI, authenticate your current machine to a Databricks Workspace:\n\n```commandline\ndatabricks auth login --host WORKSPACE_HOST\n```\n\n To enable debug logs, simply add `--debug` flag to any command.\n\n### Installing dlt-meta:\n\n- Install dlt-meta via Databricks CLI:\n\n```commandline\n databricks labs install dlt-meta\n```\n\n### Onboard using dlt-meta CLI:\n\nIf you want to run existing demo files please follow these steps before running onboard command:\n\n```commandline\n git clone https://github.com/databrickslabs/dlt-meta.git\n```\n\n```commandline\n cd dlt-meta\n```\n\n```commandline\n python -m venv .venv\n```\n\n```commandline\n source .venv/bin/activate\n```\n\n```commandline\n pip install databricks-sdk\n```\n\n```commandline\n dlt_meta_home=$(pwd)\n```\n\n```commandline\n export PYTHONPATH=$dlt_meta_home\n```\n```commandline\n databricks labs dlt-meta onboard\n```\n\n\n\nAbove commands will prompt you to provide onboarding details. If you have cloned dlt-meta git repo then accept defaults which will launch config from demo folder.\n\n\n\n- Goto your databricks workspace and located onboarding job under: Workflow->Jobs runs\n\n### depoly using dlt-meta CLI:\n\n- Once onboarding jobs is finished deploy `bronze` and `silver` DLT using below command\n- ```commandline\n databricks labs dlt-meta deploy\n ```\n- - Above command will prompt you to provide dlt details. Please provide respective details for schema which you provided in above steps\n- - Bronze DLT\n\n\n\n\n- Silver DLT\n- - ```commandline\n databricks labs dlt-meta deploy\n ```\n- - Above command will prompt you to provide dlt details. Please provide respective details for schema which you provided in above steps\n\n\n\n\n## More questions\n\nRefer to the [FAQ](https://databrickslabs.github.io/dlt-meta/faq)\nand DLT-META [documentation](https://databrickslabs.github.io/dlt-meta/)\n\n# Project Support\n\nPlease note that all projects released under [`Databricks Labs`](https://www.databricks.com/learn/labs)\nare provided for your exploration only, and are not formally supported by Databricks with Service Level Agreements\n(SLAs). They are provided AS-IS and we do not make any guarantees of any kind. Please do not submit a support ticket\nrelating to any issues arising from the use of these projects.\n\nAny issues discovered through the use of this project should be filed as issues on the Github Repo. \nThey will be reviewed as time permits, but there are no formal SLAs for support.\n",
"bugtrack_url": null,
"license": "Databricks License",
"summary": "DLT-META Framework",
"version": "0.0.8",
"project_urls": null,
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "34d4fe556c8d2e1abc1d4fb71302ccb39508b8a22560f543114741b229d80f03",
"md5": "994902c8f87e43b6a3356e31cf2e8d10",
"sha256": "ff65bff3e99a23461e4f77526cd307b44793ceb047e8f819f8e75f0e7a875ca9"
},
"downloads": -1,
"filename": "dlt_meta-0.0.8-py3-none-any.whl",
"has_sig": false,
"md5_digest": "994902c8f87e43b6a3356e31cf2e8d10",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 42709,
"upload_time": "2024-08-06T23:22:37",
"upload_time_iso_8601": "2024-08-06T23:22:37.689173Z",
"url": "https://files.pythonhosted.org/packages/34/d4/fe556c8d2e1abc1d4fb71302ccb39508b8a22560f543114741b229d80f03/dlt_meta-0.0.8-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-06 23:22:37",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "dlt-meta"
}
