# spark_dql_mvp_tools
[](https://opensource.org/licenses/Apache-2.0)
[](pyup)
[](pyup)
[](codecov)
spark_dql_mvp_tools is a Python library that implements quality rules in sandbox
## Installation
The code is packaged for PyPI, so that the installation consists in running:
## Usage
wrapper create hammurabies MVP
## Sandbox
## Installation
```sh
!yes| pip uninstall spark-dql-mvp-tools
```
```sh
pip install spark-dql-mvptools --user --upgrade
```
## IMPORTS
```sh
import os
import pyspark
from pyspark.sql import functions as func
from spark_generated_rules_tools import dq_path_workspace
from spark_generated_rules_tools import dq_generated_mvp
import spark_dataframe_tools
```
## Variables
```sh
user_sandbox="P030772"
```
## Creating Workspace
```sh
dq_path_workspace(user_sandbox=user_sandbox)
```
## Run
```sh
table_raw_name = 't_klau_moe_adj_id_mthly_info'
table_master_name = 't_pmfi_moe_adj_id_mthly_info'
periodicity = 'Daily'
target_staging_path = '/in/staging/datax/klau/my_file_{?YEAR_MONTH}.csv'
is_uuaa_tag = False
dq_generated_mvp(table_master_name=table_master_name,
table_raw_name=table_raw_name,
periodicity=periodicity,
target_staging_path=target_staging_path,
is_uuaa_tag=is_uuaa_tag)
```
## License
[Apache License 2.0](https://www.dropbox.com/s/8t6xtgk06o3ij61/LICENSE?dl=0).
## New features v1.0
## BugFix
- choco install visualcpp-build-tools
## Reference
- Jonathan Quiza [github](https://github.com/jonaqp).
- Jonathan Quiza [RumiMLSpark](http://rumi-ml.herokuapp.com/).
Raw data
{
"_id": null,
"home_page": "https://github.com/jonaqp/spark_dql_mvp_tools/",
"name": "spark-dql-tools",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "spark, dql, rules, hammurabies, haas, mvp",
"author": "Jonathan Quiza",
"author_email": "jony327@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/5a/3e/6e3c69104d6c08fc2e8423746eb693f180d84fb2c63cb08350c8380afcf3/spark_dql_tools-0.7.2.tar.gz",
"platform": null,
"description": "# spark_dql_mvp_tools\r\n\r\n[](https://opensource.org/licenses/Apache-2.0)\r\n[](pyup)\r\n[](pyup)\r\n[](codecov)\r\n\r\nspark_dql_mvp_tools is a Python library that implements quality rules in sandbox\r\n\r\n## Installation\r\n\r\nThe code is packaged for PyPI, so that the installation consists in running:\r\n\r\n## Usage\r\n\r\nwrapper create hammurabies MVP\r\n\r\n## Sandbox\r\n\r\n## Installation\r\n\r\n```sh\r\n!yes| pip uninstall spark-dql-mvp-tools\r\n```\r\n\r\n```sh\r\npip install spark-dql-mvptools --user --upgrade\r\n```\r\n\r\n## IMPORTS\r\n\r\n```sh\r\nimport os\r\nimport pyspark\r\nfrom pyspark.sql import functions as func\r\nfrom spark_generated_rules_tools import dq_path_workspace\r\nfrom spark_generated_rules_tools import dq_generated_mvp\r\nimport spark_dataframe_tools \r\n\r\n```\r\n\r\n## Variables\r\n\r\n```sh\r\nuser_sandbox=\"P030772\"\r\n```\r\n\r\n## Creating Workspace\r\n\r\n```sh\r\ndq_path_workspace(user_sandbox=user_sandbox)\r\n```\r\n\r\n## Run\r\n\r\n```sh\r\ntable_raw_name = 't_klau_moe_adj_id_mthly_info'\r\ntable_master_name = 't_pmfi_moe_adj_id_mthly_info'\r\nperiodicity = 'Daily'\r\ntarget_staging_path = '/in/staging/datax/klau/my_file_{?YEAR_MONTH}.csv'\r\nis_uuaa_tag = False\r\n\r\ndq_generated_mvp(table_master_name=table_master_name,\r\n table_raw_name=table_raw_name,\r\n periodicity=periodicity,\r\n target_staging_path=target_staging_path,\r\n is_uuaa_tag=is_uuaa_tag)\r\n \r\n```\r\n\r\n## License\r\n\r\n[Apache License 2.0](https://www.dropbox.com/s/8t6xtgk06o3ij61/LICENSE?dl=0).\r\n\r\n## New features v1.0\r\n\r\n## BugFix\r\n\r\n- choco install visualcpp-build-tools\r\n\r\n## Reference\r\n\r\n- Jonathan Quiza [github](https://github.com/jonaqp).\r\n- Jonathan Quiza [RumiMLSpark](http://rumi-ml.herokuapp.com/).\r\n\r\n\r\n",
"bugtrack_url": null,
"license": null,
"summary": "spark_dql_tools",
"version": "0.7.2",
"project_urls": {
"Download": "https://github.com/jonaqp/spark_dql_mvp_tools/archive/main.zip",
"Homepage": "https://github.com/jonaqp/spark_dql_mvp_tools/"
},
"split_keywords": [
"spark",
" dql",
" rules",
" hammurabies",
" haas",
" mvp"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "fc7a8281a3bd47c35ab55e6f09f1766257225b6bf428029fbb155df03eda0f74",
"md5": "9ece4f6e29d97f5ebd085e2e3f5acfa9",
"sha256": "dc6c9c8cf288cbe2d1a644f72ec082a43953f25ad04c5435cf536aa69344dda0"
},
"downloads": -1,
"filename": "spark_dql_tools-0.7.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "9ece4f6e29d97f5ebd085e2e3f5acfa9",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 16402,
"upload_time": "2024-04-02T00:03:11",
"upload_time_iso_8601": "2024-04-02T00:03:11.089149Z",
"url": "https://files.pythonhosted.org/packages/fc/7a/8281a3bd47c35ab55e6f09f1766257225b6bf428029fbb155df03eda0f74/spark_dql_tools-0.7.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "5a3e6e3c69104d6c08fc2e8423746eb693f180d84fb2c63cb08350c8380afcf3",
"md5": "383749675c7d8f1a27ac16822d9ea69f",
"sha256": "39723336a030fec92682ffa3920ec082b251b85a69b875dd81bdea1230a4c4c2"
},
"downloads": -1,
"filename": "spark_dql_tools-0.7.2.tar.gz",
"has_sig": false,
"md5_digest": "383749675c7d8f1a27ac16822d9ea69f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 16818,
"upload_time": "2024-04-02T00:03:13",
"upload_time_iso_8601": "2024-04-02T00:03:13.047415Z",
"url": "https://files.pythonhosted.org/packages/5a/3e/6e3c69104d6c08fc2e8423746eb693f180d84fb2c63cb08350c8380afcf3/spark_dql_tools-0.7.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-02 00:03:13",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "jonaqp",
"github_project": "spark_dql_mvp_tools",
"github_not_found": true,
"lcname": "spark-dql-tools"
}
JSON
