# spark_dql_mvp_tools
[](https://opensource.org/licenses/Apache-2.0)
[](pyup)
[](pyup)
[](codecov)
spark_dql_mvp_tools is a Python library that implements quality rules in sandbox
## Installation
The code is packaged for PyPI, so that the installation consists in running:
## Usage
wrapper create hammurabies MVP
## Sandbox
## Installation
```sh
!yes| pip uninstall spark-dql-mvp-tools
```
```sh
pip install spark-dql-mvptools --user --upgrade
```
## IMPORTS
```sh
import os
import pyspark
from pyspark.sql import functions as func
from spark_generated_rules_tools import dq_path_workspace
from spark_generated_rules_tools import dq_generated_mvp
import spark_dataframe_tools
```
## Variables
```sh
user_sandbox="P030772"
```
## Creating Workspace
```sh
dq_path_workspace(user_sandbox=user_sandbox)
```
## Run
```sh
table_raw_name = 't_klau_moe_adj_id_mthly_info'
table_master_name = 't_pmfi_moe_adj_id_mthly_info'
periodicity = 'Daily'
target_staging_path = '/in/staging/datax/klau/my_file_{?YEAR_MONTH}.csv'
is_uuaa_tag = False
dq_generated_mvp(table_master_name=table_master_name,
table_raw_name=table_raw_name,
periodicity=periodicity,
target_staging_path=target_staging_path,
is_uuaa_tag=is_uuaa_tag)
```
## License
[Apache License 2.0](https://www.dropbox.com/s/8t6xtgk06o3ij61/LICENSE?dl=0).
## New features v1.0
## BugFix
- choco install visualcpp-build-tools
## Reference
- Jonathan Quiza [github](https://github.com/jonaqp).
- Jonathan Quiza [RumiMLSpark](http://rumi-ml.herokuapp.com/).
Raw data
{
"_id": null,
"home_page": "https://github.com/jonaqp/spark_dql_mvp_tools/",
"name": "spark-dql-mvp-tools",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "spark,dql,rules,hammurabies,haas,mvp",
"author": "Jonathan Quiza",
"author_email": "jony327@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/69/81/a2903022079ad5ae2e03b1fd448440f1e30285fadf2a69407cb4363d6483/spark_dql_mvp_tools-0.9.8.tar.gz",
"platform": null,
"description": "# spark_dql_mvp_tools\r\n\r\n[](https://opensource.org/licenses/Apache-2.0)\r\n[](pyup)\r\n[](pyup)\r\n[](codecov)\r\n\r\nspark_dql_mvp_tools is a Python library that implements quality rules in sandbox\r\n\r\n## Installation\r\n\r\nThe code is packaged for PyPI, so that the installation consists in running:\r\n\r\n## Usage\r\n\r\nwrapper create hammurabies MVP\r\n\r\n## Sandbox\r\n\r\n## Installation\r\n\r\n```sh\r\n!yes| pip uninstall spark-dql-mvp-tools\r\n```\r\n\r\n```sh\r\npip install spark-dql-mvptools --user --upgrade\r\n```\r\n\r\n## IMPORTS\r\n\r\n```sh\r\nimport os\r\nimport pyspark\r\nfrom pyspark.sql import functions as func\r\nfrom spark_generated_rules_tools import dq_path_workspace\r\nfrom spark_generated_rules_tools import dq_generated_mvp\r\nimport spark_dataframe_tools \r\n\r\n```\r\n\r\n## Variables\r\n\r\n```sh\r\nuser_sandbox=\"P030772\"\r\n```\r\n\r\n## Creating Workspace\r\n\r\n```sh\r\ndq_path_workspace(user_sandbox=user_sandbox)\r\n```\r\n\r\n## Run\r\n\r\n```sh\r\ntable_raw_name = 't_klau_moe_adj_id_mthly_info'\r\ntable_master_name = 't_pmfi_moe_adj_id_mthly_info'\r\nperiodicity = 'Daily'\r\ntarget_staging_path = '/in/staging/datax/klau/my_file_{?YEAR_MONTH}.csv'\r\nis_uuaa_tag = False\r\n\r\ndq_generated_mvp(table_master_name=table_master_name,\r\n table_raw_name=table_raw_name,\r\n periodicity=periodicity,\r\n target_staging_path=target_staging_path,\r\n is_uuaa_tag=is_uuaa_tag)\r\n \r\n```\r\n\r\n## License\r\n\r\n[Apache License 2.0](https://www.dropbox.com/s/8t6xtgk06o3ij61/LICENSE?dl=0).\r\n\r\n## New features v1.0\r\n\r\n## BugFix\r\n\r\n- choco install visualcpp-build-tools\r\n\r\n## Reference\r\n\r\n- Jonathan Quiza [github](https://github.com/jonaqp).\r\n- Jonathan Quiza [RumiMLSpark](http://rumi-ml.herokuapp.com/).\r\n\r\n\r\n",
"bugtrack_url": null,
"license": "",
"summary": "spark_dql_mvp_tools",
"version": "0.9.8",
"project_urls": {
"Download": "https://github.com/jonaqp/spark_dql_mvp_tools/archive/main.zip",
"Homepage": "https://github.com/jonaqp/spark_dql_mvp_tools/"
},
"split_keywords": [
"spark",
"dql",
"rules",
"hammurabies",
"haas",
"mvp"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "258f54c4804b4fcf4eb9f521bd9a8978b80ab87cf3f289ab563c5c7eb3519574",
"md5": "432a83cfdb3b2784f0cb29ac85750652",
"sha256": "c076f807335bd8f9824a700c6317ca33ae5119b0f7fd751cb30dab4334d5a8c5"
},
"downloads": -1,
"filename": "spark_dql_mvp_tools-0.9.8-py3-none-any.whl",
"has_sig": false,
"md5_digest": "432a83cfdb3b2784f0cb29ac85750652",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 12062,
"upload_time": "2023-09-20T12:10:07",
"upload_time_iso_8601": "2023-09-20T12:10:07.216580Z",
"url": "https://files.pythonhosted.org/packages/25/8f/54c4804b4fcf4eb9f521bd9a8978b80ab87cf3f289ab563c5c7eb3519574/spark_dql_mvp_tools-0.9.8-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6981a2903022079ad5ae2e03b1fd448440f1e30285fadf2a69407cb4363d6483",
"md5": "b3bbbb8b528afb3355830ef7835807d7",
"sha256": "bc0781752e6516cbfe1f09cba693795d955e5c2fee8b2583c0e96f598d5a057a"
},
"downloads": -1,
"filename": "spark_dql_mvp_tools-0.9.8.tar.gz",
"has_sig": false,
"md5_digest": "b3bbbb8b528afb3355830ef7835807d7",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 12396,
"upload_time": "2023-09-20T12:10:09",
"upload_time_iso_8601": "2023-09-20T12:10:09.362989Z",
"url": "https://files.pythonhosted.org/packages/69/81/a2903022079ad5ae2e03b1fd448440f1e30285fadf2a69407cb4363d6483/spark_dql_mvp_tools-0.9.8.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-09-20 12:10:09",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "jonaqp",
"github_project": "spark_dql_mvp_tools",
"github_not_found": true,
"lcname": "spark-dql-mvp-tools"
}
JSON
