# spark_quality_rules_tools
[](https://opensource.org/licenses/Apache-2.0)
[](pyup)
[](pyup)
[](codecov)
spark_quality_rules_tools is a Python library that implements quality rules in sandbox
## Installation
The code is packaged for PyPI, so that the installation consists in running:
## Usage
wrapper run hammurabies
## Sandbox
## Installation
```sh
!yes| pip uninstall spark-quality-rules-tools
```
```sh
pip install spark-quality-rules-tools --user --upgrade
```
## IMPORTS
```sh
import os
import pyspark
from spark_quality_rules_tools import dq_path_workspace
from spark_quality_rules_tools import dq_download_jar
from spark_quality_rules_tools import dq_spark_session
from spark_quality_rules_tools import dq_extract_parameters
from spark_quality_rules_tools import dq_run_sandbox
from spark_quality_rules_tools import dq_validate_conf
from spark_quality_rules_tools import dq_validate_rules
from spark_quality_rules_tools import show_spark_df
pyspark.sql.dataframe.DataFrame.show2 = show_spark_df
```
## Variables
```sh
project_sda="SDA_37036"
url_conf = "http://artifactory-gdt.central-02.nextgen.igrupobbva/artifactory/gl-datio-spark-libs-maven-local/com/datiobd/cdd-hammurabi/4.0.9/DQ_LOCAL_CONFS/KCOG/KCOG_branch_MRField.conf"
```
## Creating Workspace
```sh
dq_path_workspace(project_sda=project_sda)
```
## Download haas jar
```sh
dq_download_jar(haas_version="4.8.0", force=True)
```
## Spark Session
```sh
spark, sc = dq_spark_session()
```
## Validate Conf
```sh
dq_validate_conf(url_conf=url_conf)
```
## Extract Params
```sh
dq_extract_parameters(url_conf=url_conf)
```
## Json params
```sh
parameter_conf_list = [
{
"ARTIFACTORY_UNIQUE_CACHE": "http://artifactory-gdt.central-02.nextgen.igrupobbva",
"ODATE_DATE": "2022-11-11",
"COUNTRY_ID": "PE",
"SCHEMA_PATH": "t_kcog_branch.output.schema",
"CUTOFF_DATE": "2022-11-11",
"SCHEMAS_REPOSITORY": "gl-datio-da-generic-local/schemas/pe/kcog/master/t_kcog_branch/latest/"
}
]
```
## Run
```sh
dq_run_sandbox(spark=spark,
sc=sc,
parameter_conf_list=parameter_conf_list,
url_conf=url_conf)
```
```sh
df = spark.read.csv("file:/var/sds/homes/P030772/workspace/data_quality_rules/data_reports/KCOG/KCOG_BRANCH_MRFIELD_202304120046_20221111.csv",
header=True)
df.show2(100)
```
## Run
```sh
dq_validate_rules(url_conf=url_conf)
```
## License
[Apache License 2.0](https://www.dropbox.com/s/8t6xtgk06o3ij61/LICENSE?dl=0).
## New features v1.0
## BugFix
- choco install visualcpp-build-tools
## Reference
- Jonathan Quiza [github](https://github.com/jonaqp).
- Jonathan Quiza [RumiMLSpark](http://rumi-ml.herokuapp.com/).
Raw data
{
"_id": null,
"home_page": "https://github.com/jonaqp/spark_quality_rules_tools/",
"name": "spark-quality-rules-tools",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "spark, dq, rules, hammurabies",
"author": "Jonathan Quiza",
"author_email": "jony327@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/ab/58/943254bcac19a40934d4d5886d57d3a1eda05a18ed09f04be244fb395dbe/spark_quality_rules_tools-0.9.10.tar.gz",
"platform": null,
"description": "# spark_quality_rules_tools\r\n\r\n[](https://opensource.org/licenses/Apache-2.0)\r\n[](pyup)\r\n[](pyup)\r\n[](codecov)\r\n\r\nspark_quality_rules_tools is a Python library that implements quality rules in sandbox\r\n\r\n## Installation\r\n\r\nThe code is packaged for PyPI, so that the installation consists in running:\r\n\r\n\r\n## Usage\r\n\r\nwrapper run hammurabies\r\n\r\n## Sandbox\r\n## Installation\r\n```sh\r\n!yes| pip uninstall spark-quality-rules-tools\r\n```\r\n\r\n```sh\r\npip install spark-quality-rules-tools --user --upgrade\r\n```\r\n\r\n## IMPORTS\r\n```sh\r\nimport os\r\nimport pyspark\r\nfrom spark_quality_rules_tools import dq_path_workspace\r\nfrom spark_quality_rules_tools import dq_download_jar\r\nfrom spark_quality_rules_tools import dq_spark_session\r\nfrom spark_quality_rules_tools import dq_extract_parameters\r\nfrom spark_quality_rules_tools import dq_run_sandbox\r\nfrom spark_quality_rules_tools import dq_validate_conf\r\nfrom spark_quality_rules_tools import dq_validate_rules\r\nfrom spark_quality_rules_tools import show_spark_df\r\npyspark.sql.dataframe.DataFrame.show2 = show_spark_df\r\n```\r\n\r\n## Variables\r\n```sh\r\nproject_sda=\"SDA_37036\"\r\nurl_conf = \"http://artifactory-gdt.central-02.nextgen.igrupobbva/artifactory/gl-datio-spark-libs-maven-local/com/datiobd/cdd-hammurabi/4.0.9/DQ_LOCAL_CONFS/KCOG/KCOG_branch_MRField.conf\"\r\n```\r\n\r\n\r\n## Creating Workspace\r\n```sh\r\ndq_path_workspace(project_sda=project_sda)\r\n```\r\n\r\n\r\n## Download haas jar\r\n```sh\r\ndq_download_jar(haas_version=\"4.8.0\", force=True)\r\n```\r\n\r\n\r\n## Spark Session\r\n```sh\r\nspark, sc = dq_spark_session()\r\n```\r\n\r\n\r\n## Validate Conf\r\n```sh\r\ndq_validate_conf(url_conf=url_conf)\r\n```\r\n\r\n\r\n## Extract Params\r\n```sh\r\ndq_extract_parameters(url_conf=url_conf)\r\n```\r\n\r\n\r\n## Json params\r\n```sh\r\nparameter_conf_list = [\r\n { \r\n \"ARTIFACTORY_UNIQUE_CACHE\": \"http://artifactory-gdt.central-02.nextgen.igrupobbva\",\r\n \"ODATE_DATE\": \"2022-11-11\",\r\n \"COUNTRY_ID\": \"PE\",\r\n \"SCHEMA_PATH\": \"t_kcog_branch.output.schema\",\r\n \"CUTOFF_DATE\": \"2022-11-11\",\r\n \"SCHEMAS_REPOSITORY\": \"gl-datio-da-generic-local/schemas/pe/kcog/master/t_kcog_branch/latest/\"\r\n }\r\n]\r\n```\r\n\r\n\r\n## Run \r\n```sh\r\ndq_run_sandbox(spark=spark,\r\n sc=sc,\r\n parameter_conf_list=parameter_conf_list,\r\n url_conf=url_conf)\r\n```\r\n\r\n \r\n```sh \r\ndf = spark.read.csv(\"file:/var/sds/homes/P030772/workspace/data_quality_rules/data_reports/KCOG/KCOG_BRANCH_MRFIELD_202304120046_20221111.csv\", \r\n header=True) \r\ndf.show2(100)\r\n```\r\n\r\n\r\n## Run \r\n```sh\r\ndq_validate_rules(url_conf=url_conf)\r\n```\r\n\r\n\r\n## License\r\n\r\n[Apache License 2.0](https://www.dropbox.com/s/8t6xtgk06o3ij61/LICENSE?dl=0).\r\n\r\n## New features v1.0\r\n\r\n## BugFix\r\n\r\n- choco install visualcpp-build-tools\r\n\r\n## Reference\r\n\r\n- Jonathan Quiza [github](https://github.com/jonaqp).\r\n- Jonathan Quiza [RumiMLSpark](http://rumi-ml.herokuapp.com/).\r\n\r\n\r\n",
"bugtrack_url": null,
"license": null,
"summary": "spark_quality_rules_tools",
"version": "0.9.10",
"project_urls": {
"Download": "https://github.com/jonaqp/spark_quality_rules_tools/archive/main.zip",
"Homepage": "https://github.com/jonaqp/spark_quality_rules_tools/"
},
"split_keywords": [
"spark",
" dq",
" rules",
" hammurabies"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4199b0d5f008070d4fa673ac6686b8b8d6a78f2b89f4feb3504ae3d82baeab99",
"md5": "c0bb2db9402216e3d7556a2ea77cbf4c",
"sha256": "d97f08d98d4a33b83f32900617f8a168ea1ff7c39432103e76887398718d83e2"
},
"downloads": -1,
"filename": "spark_quality_rules_tools-0.9.10-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c0bb2db9402216e3d7556a2ea77cbf4c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 17998,
"upload_time": "2024-04-08T23:05:04",
"upload_time_iso_8601": "2024-04-08T23:05:04.961271Z",
"url": "https://files.pythonhosted.org/packages/41/99/b0d5f008070d4fa673ac6686b8b8d6a78f2b89f4feb3504ae3d82baeab99/spark_quality_rules_tools-0.9.10-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ab58943254bcac19a40934d4d5886d57d3a1eda05a18ed09f04be244fb395dbe",
"md5": "acbf8de4f7603d5241cbe54e107f9ffa",
"sha256": "256f51025919983bd1be597312607e7f2fe9fd254c39854b4275212fd9875ffe"
},
"downloads": -1,
"filename": "spark_quality_rules_tools-0.9.10.tar.gz",
"has_sig": false,
"md5_digest": "acbf8de4f7603d5241cbe54e107f9ffa",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 17223,
"upload_time": "2024-04-08T23:05:07",
"upload_time_iso_8601": "2024-04-08T23:05:07.063724Z",
"url": "https://files.pythonhosted.org/packages/ab/58/943254bcac19a40934d4d5886d57d3a1eda05a18ed09f04be244fb395dbe/spark_quality_rules_tools-0.9.10.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-08 23:05:07",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "jonaqp",
"github_project": "spark_quality_rules_tools",
"github_not_found": true,
"lcname": "spark-quality-rules-tools"
}
JSON
