# spetlr-tools
## Table of Contents
- [Description](#description)
- [Purpose of spetlr-tools](#purpose-of-spetlr-tools)
- [Installation](#installation)
- [Development Notes](#development-notes)
- [Testing](#testing)
- [General Project Info](#general-project-info)
- [Contributing](#contributing)
- [Build Status](#build-status)
- [Releases](#releases)
- [Contact](#contact)
## Description
SPETLR-tools is a library that provides a set of tools for working with Databricks Lakehouses. These tools include test fixtures and development utilities that are not part of the runtime tools in [SPETLR](https://github.com/spetlr-org/spetlr).
Visit the official SPETLR webpage: [https://spetlr.com/](https://spetlr.com/)
## Purpose of SPETLR-tools
SPETLR-tools is designed to support SPETLR in various scenarios, including:
* Test tools in pytest:
* Examples: Dataframe validation checks, Data format checking, ...
* Helpers for investigating data:
* Examples: Extract schema from binary encoded columns, Get the difference between two dataframes , ...
* SPETLR-tools CLI:
* Examples: Submit pytests to Databricks cluster, Automated Azure Token extraction, ...
## SPETLR-tools vs. SPETLR
* SPETLR-tools: Tested in a Python interpreter and per january 2024 also integration tested using on-cluster job tests.
* SPETLR-tools: Github workflow have an very simple Azure Deployment
* SPETLR: Fully unit and integration tested - library ready for production use
* SPETLR-tools: Supports deployment and testing
* Use only in `test_requirements.txt`
# Installation
Install SPETLR from PyPI:
[](https://pypi.org/project/spetlr-tools/)
[](https://pypi.org/project/spetlr-tools/)
```
pip install spetlr-tools
```
# Development Notes
To prepare for development, please install following additional requirements:
- Java 8
- `pip install -r test_requirements.txt`
Then install the package locally:
python setup.py develop
## Testing
### Local tests
After installing the dev-requirements, execute tests by running:
pytest tests
These tests are located in the `./tests/unit` folder and only require a Python interpreter. Pull requests will not be accepted if these tests do not pass. If you add new features, please include corresponding tests.
### CLI and Cluster tests
During the pre-integration workflow (`.gitub/workflows/pre-integration.yml`) spetlr-tool supported CLI are (should) be tested.
# General Project Info
[](https://github.com/spetlr-org/spetlr-tools)
[](https://github.com/spetlr-org/spetlr-tools)
[](https://github.com/spetlr-org/spetlr-tools)
[](https://github.com/spetlr-org/spetlr-tools)
[](https://github.com/spetlr-org/spetlr-tools/issues)
[](https://pypi.org/project/spetlr-tools/)
# Contributing
Feel free to contribute to SPETLR-tools. Any contributions are appreciated - not only new features, but also if you find a way to improve SPETLR-tools.
If you have a suggestion that can enhance SPETLR-tools, please fork the repository and create a pull request. Alternatively, you can open an issue with the "enhancement" tag.
1. Fork the Project
2. Create your Feature Branch (`git checkout -b feature/NewSPETLRToolsFeature`)
3. Commit your Changes (`git commit -m 'Add some SEPTLRToolsFeature'`)
4. Push to the Branch (`git push origin feature/NewSPETLRToolsFeature`)
5. Open a Pull Request
# Build Status
[](https://github.com/spetlr-org/spetlr-tools/actions/workflows/post-integration.yml)
# Releases
Releases to PyPI is an Github Action which needs to be manually triggered.
[](https://github.com/spetlr-org/spetlr-tools/actions/workflows/release.yml)
[](https://pypi.org/project/spetlr-tools/)
# Contact
For any inquiries, please use the [SPETLR Discord Server](https://discord.gg/p9bzqGybVW).
Raw data
{
"_id": null,
"home_page": "https://github.com/spetlr-org/spetlr-tools",
"name": "spetlr-tools",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "databricks,pyspark,spetlr",
"author": "Spetlr.Org",
"author_email": "spetlr.org@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/a9/7b/66f7c261c6b0dbd345b36c0f31518fe26a8dfbc1e0bac147544fbf68ec33/spetlr-tools-0.1.65.tar.gz",
"platform": null,
"description": "# spetlr-tools\n\n## Table of Contents\n\n- [Description](#description)\n- [Purpose of spetlr-tools](#purpose-of-spetlr-tools)\n- [Installation](#installation)\n- [Development Notes](#development-notes)\n- [Testing](#testing)\n- [General Project Info](#general-project-info)\n- [Contributing](#contributing)\n- [Build Status](#build-status)\n- [Releases](#releases)\n- [Contact](#contact)\n\n## Description\nSPETLR-tools is a library that provides a set of tools for working with Databricks Lakehouses. These tools include test fixtures and development utilities that are not part of the runtime tools in [SPETLR](https://github.com/spetlr-org/spetlr).\n\n\nVisit the official SPETLR webpage: [https://spetlr.com/](https://spetlr.com/)\n\n\n## Purpose of SPETLR-tools\n\nSPETLR-tools is designed to support SPETLR in various scenarios, including:\n\n* Test tools in pytest:\n * Examples: Dataframe validation checks, Data format checking, ...\n* Helpers for investigating data:\n * Examples: Extract schema from binary encoded columns, Get the difference between two dataframes , ...\n* SPETLR-tools CLI:\n * Examples: Submit pytests to Databricks cluster, Automated Azure Token extraction, ... \n\n## SPETLR-tools vs. SPETLR\n\n* SPETLR-tools: Tested in a Python interpreter and per january 2024 also integration tested using on-cluster job tests.\n* SPETLR-tools: Github workflow have an very simple Azure Deployment\n* SPETLR: Fully unit and integration tested - library ready for production use\n* SPETLR-tools: Supports deployment and testing\n * Use only in `test_requirements.txt`\n\n# Installation\n\nInstall SPETLR from PyPI: \n\n[](https://pypi.org/project/spetlr-tools/)\n[](https://pypi.org/project/spetlr-tools/)\n``` \npip install spetlr-tools\n```\n\n# Development Notes\n\nTo prepare for development, please install following additional requirements:\n - Java 8\n - `pip install -r test_requirements.txt`\n\nThen install the package locally:\n\n python setup.py develop\n\n## Testing\n\n### Local tests\nAfter installing the dev-requirements, execute tests by running:\n\n pytest tests\n\nThese tests are located in the `./tests/unit` folder and only require a Python interpreter. Pull requests will not be accepted if these tests do not pass. If you add new features, please include corresponding tests.\n\n### CLI and Cluster tests\nDuring the pre-integration workflow (`.gitub/workflows/pre-integration.yml`) spetlr-tool supported CLI are (should) be tested. \n\n# General Project Info\n[](https://github.com/spetlr-org/spetlr-tools)\n[](https://github.com/spetlr-org/spetlr-tools)\n[](https://github.com/spetlr-org/spetlr-tools)\n[](https://github.com/spetlr-org/spetlr-tools)\n[](https://github.com/spetlr-org/spetlr-tools/issues)\n[](https://pypi.org/project/spetlr-tools/)\n\n\n# Contributing\n\nFeel free to contribute to SPETLR-tools. Any contributions are appreciated - not only new features, but also if you find a way to improve SPETLR-tools. \n\nIf you have a suggestion that can enhance SPETLR-tools, please fork the repository and create a pull request. Alternatively, you can open an issue with the \"enhancement\" tag.\n\n1. Fork the Project\n2. Create your Feature Branch (`git checkout -b feature/NewSPETLRToolsFeature`)\n3. Commit your Changes (`git commit -m 'Add some SEPTLRToolsFeature'`)\n4. Push to the Branch (`git push origin feature/NewSPETLRToolsFeature`)\n5. Open a Pull Request\n\n\n# Build Status\n\n[](https://github.com/spetlr-org/spetlr-tools/actions/workflows/post-integration.yml)\n\n\n# Releases\nReleases to PyPI is an Github Action which needs to be manually triggered. \n\n[](https://github.com/spetlr-org/spetlr-tools/actions/workflows/release.yml)\n[](https://pypi.org/project/spetlr-tools/)\n\n# Contact\n\nFor any inquiries, please use the [SPETLR Discord Server](https://discord.gg/p9bzqGybVW).\n\n",
"bugtrack_url": null,
"license": "",
"summary": "Supplements to the python SPark ETL libRary (SPETLR) for Databricks.",
"version": "0.1.65",
"project_urls": {
"Bug Reports": "https://github.com/spetlr-org/spetlr-tools/issues",
"Documentation": "https://github.com/spetlr-org/spetlr-tools",
"Homepage": "https://github.com/spetlr-org/spetlr-tools",
"Source Code": "https://github.com/spetlr-org/spetlr-tools"
},
"split_keywords": [
"databricks",
"pyspark",
"spetlr"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "6495731ec8bee758946bac676b58a5d14241ab38d81f01ebd3f27c452498e3ce",
"md5": "79fdcf3e9a10738424cf9140cf5f1aca",
"sha256": "bf41a2a910cd93d1fb6c13b3013436896192fcfad6536593745ad348821add47"
},
"downloads": -1,
"filename": "spetlr_tools-0.1.65-py3-none-any.whl",
"has_sig": false,
"md5_digest": "79fdcf3e9a10738424cf9140cf5f1aca",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 45498,
"upload_time": "2024-02-19T19:44:16",
"upload_time_iso_8601": "2024-02-19T19:44:16.128053Z",
"url": "https://files.pythonhosted.org/packages/64/95/731ec8bee758946bac676b58a5d14241ab38d81f01ebd3f27c452498e3ce/spetlr_tools-0.1.65-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a97b66f7c261c6b0dbd345b36c0f31518fe26a8dfbc1e0bac147544fbf68ec33",
"md5": "006c71e0b9ab870da268524abfcd3db4",
"sha256": "86a006e0720f7025782241d15a55fdd6e9f255ad40766271774bb60633c32b71"
},
"downloads": -1,
"filename": "spetlr-tools-0.1.65.tar.gz",
"has_sig": false,
"md5_digest": "006c71e0b9ab870da268524abfcd3db4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 33265,
"upload_time": "2024-02-19T19:44:17",
"upload_time_iso_8601": "2024-02-19T19:44:17.587587Z",
"url": "https://files.pythonhosted.org/packages/a9/7b/66f7c261c6b0dbd345b36c0f31518fe26a8dfbc1e0bac147544fbf68ec33/spetlr-tools-0.1.65.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-19 19:44:17",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "spetlr-org",
"github_project": "spetlr-tools",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"test_requirements": [],
"lcname": "spetlr-tools"
}
JSON
