# spetlr
A python ETL libRary (SPETLR) for Databricks powered by Apache SPark.
Visit SPETLR official webpage: [https://spetlr.com/](https://spetlr.com/)
# NEWS
Start Supporting DBR LTS14.3: [Follow the PR.](https://github.com/spetlr-org/spetlr/pull/177).
- Cluster test submission with spetlr-tools
- Upgrade to Python 3.10
- The spetlr library (probably except SQL connection with ODBC) still supports older LTS versions between 9.1 and 13.3, but only 14.3 is tested.
- SQL ODBC driver version 18 is suppoetd (this is a breaking change if you haven't upgraded your ODBC driver).
- Neweset CosmosDB connector is tested for compatibility with DBR LTS14.3.
# Table of Contents
- [Description](#description)
- [Important Notes](#important-notes)
- [Installation](#installation)
- [Development Notes](#development-notes)
- [Testing](#testing)
- [General Project Info](#general-project-info)
- [Contributing](#contributing)
- [Build Status](#build-status)
- [Releases](#releases)
- [Requirements](#requirements-and-dependencies)
- [Contact](#contact)
# Description
SPETLR has a lot of great tools for working with ETL in Databricks. But to make it easy for you to consider why you need
SPETLR here is a list of the core features:
* ETL framework: A common ETL framework that enables reusable transformations in an object-oriented manner. Standardized
structures facilitate cooperation in large teams.
* Integration testing: A framework for creating test databases and tables before deploying to production in order to
ensure reliable and stable data platforms. An additional layer of data abstraction allows full integration testing.
* Handlers: Standard connectors with commonly used options reduce boilerplate.
For more information, visit SPETLR official webpage: [https://spetlr.com/](https://spetlr.com/)
# Important Notes
This package can not be run or tested without access to `pyspark`.
However, installing `pyspark` as part of our installer gave issues when
other versions of `pyspark` were needed. Hence we took out the dependency
from our installer.
# Installation
Install SPETLR from PyPI:
[](https://pypi.org/project/spetlr/)
[](https://pypi.org/project/spetlr/)
```
pip install spetlr
```
# Development Notes
To prepare for development, please install these additional requirements:
- Java 8
- `pip install -r test_requirements.txt`
Then install the package locally
python setup.py develop
## Testing
### Local tests
After installing the dev-requirements, execute tests by running:
pytest tests
These tests are located in the `./tests/local` folder and only require a Python interpreter. Pull requests will not be
accepted if these tests do not pass. If you add new features, please include corresponding tests.
### Cluster tests
Tests in the `./tests/cluster` folder are designed to run on a Databricks cluster.
The [Pre-integration Test](https://github.com/spetlr-org/spetlr/blob/main/.github/workflows/pre-integration.yml)
utilizes Azure Resource deployment - and can only be run by the spetlr-org admins.
To deploy the necessary Azure resources to your own Azure Tenant, run the following command:
```powershell
.\.github\deploy\deploy.ps1 -uniqueRunId "yourUniqueId"
```
Be aware that the applied name for *uniqueRunId* should only contain lower case and numbers, and its length should not
exceed 12 characters.
Afterward, execute the following commands:
```powershell
.\.github\submit\build.ps1
.\.github\submit\submit_test_job.ps1
```
# General Project Info
[](https://github.com/spetlr-org/spetlr)
[](https://github.com/spetlr-org/spetlr)
[](https://github.com/spetlr-org/spetlr)
[](https://github.com/spetlr-org/spetlr)
[](https://github.com/spetlr-org/spetlr/issues)
[](https://pypi.org/project/spetlr/)
# Contributing
Feel free to contribute to SPETLR. Any contributions are appreciated - not only new features, but also if you find a way
to improve SPETLR.
If you have a suggestion that can enhance SPETLR, please fork the repository and create a pull request. Alternatively,
you can open an issue with the "enhancement" tag.
1. Fork the Project
2. Create your Feature Branch (`git checkout -b feature/NewSPETLRFeature`)
3. Commit your Changes (`git commit -m 'Add some SEPTLRFeature'`)
4. Push to the Branch (`git push origin feature/NewSPETLRFeature`)
5. Open a Pull Request
# Build Status
[](https://github.com/spetlr-org/spetlr/actions/workflows/post-integration.yml)
# Releases
Releases to PyPI is an Github Action which needs to be manually triggered.
[](https://github.com/spetlr-org/spetlr/actions/workflows/release.yml)
[](https://pypi.org/project/spetlr/)
# Requirements and dependencies
The library has three txt-files at the root of the repo. These files defines three levels of requirements:
- `requirements_install.txt` - this file contains the required libraries to be able to install spetlr.
- `requirements_test.txt` - libraries required to run unit- and integration tests
- `requirements_dev.txt` - libraries required in the development process in order to contribute to the repo
All libraries and their dependencies are added with a fixed version to the configuration file `setup.cfg` using the defined requirements from `requirements_install.txt`.
To __upgrade__ the the dependencies in the `setup.cfg` file do the following:
1. Create a new branch
2. Run `upgrade_requirements.ps1` in your terminal
3. Commit the changes the script has made to the cfg file. If there are no changes, everything is up to date.
4. The PR runs all tests and ensure that the library is compliant with any updates
Note that if it is desired to upgrade a dependency, but not to its newest version, it is possible to set the desired version in the `requirements_install.txt`, then this will be respected by the upgrade script.
# Contact
For any inquiries, please use the [SPETLR Discord Server](https://discord.gg/p9bzqGybVW).
Raw data
{
"_id": null,
"home_page": "https://github.com/spetlr-org/spetlr",
"name": "spetlr",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "databricks, pyspark",
"author": "Spetlr.Org",
"author_email": "spetlr.org@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/7b/1d/55a3326575f5c8b8be65d7a969b965a3312a3c8a62008afb3b944041c8e2/spetlr-14.3.5.tar.gz",
"platform": null,
"description": "# spetlr\n\nA python ETL libRary (SPETLR) for Databricks powered by Apache SPark.\n\nVisit SPETLR official webpage: [https://spetlr.com/](https://spetlr.com/)\n\n# NEWS\n\nStart Supporting DBR LTS14.3: [Follow the PR.](https://github.com/spetlr-org/spetlr/pull/177).\n- Cluster test submission with spetlr-tools\n- Upgrade to Python 3.10\n- The spetlr library (probably except SQL connection with ODBC) still supports older LTS versions between 9.1 and 13.3, but only 14.3 is tested.\n- SQL ODBC driver version 18 is suppoetd (this is a breaking change if you haven't upgraded your ODBC driver).\n- Neweset CosmosDB connector is tested for compatibility with DBR LTS14.3.\n\n# Table of Contents\n\n- [Description](#description)\n- [Important Notes](#important-notes)\n- [Installation](#installation)\n- [Development Notes](#development-notes)\n- [Testing](#testing)\n- [General Project Info](#general-project-info)\n- [Contributing](#contributing)\n- [Build Status](#build-status)\n- [Releases](#releases)\n- [Requirements](#requirements-and-dependencies)\n- [Contact](#contact)\n\n# Description\n\nSPETLR has a lot of great tools for working with ETL in Databricks. But to make it easy for you to consider why you need\nSPETLR here is a list of the core features:\n\n* ETL framework: A common ETL framework that enables reusable transformations in an object-oriented manner. Standardized\n structures facilitate cooperation in large teams.\n\n* Integration testing: A framework for creating test databases and tables before deploying to production in order to\n ensure reliable and stable data platforms. An additional layer of data abstraction allows full integration testing.\n\n* Handlers: Standard connectors with commonly used options reduce boilerplate.\n\nFor more information, visit SPETLR official webpage: [https://spetlr.com/](https://spetlr.com/)\n\n# Important Notes\n\nThis package can not be run or tested without access to `pyspark`.\nHowever, installing `pyspark` as part of our installer gave issues when\nother versions of `pyspark` were needed. Hence we took out the dependency\nfrom our installer.\n\n# Installation\n\nInstall SPETLR from PyPI:\n[](https://pypi.org/project/spetlr/)\n[](https://pypi.org/project/spetlr/)\n\n``` \npip install spetlr\n```\n\n# Development Notes\n\nTo prepare for development, please install these additional requirements:\n\n- Java 8\n- `pip install -r test_requirements.txt`\n\nThen install the package locally\n\n python setup.py develop\n\n## Testing\n\n### Local tests\n\nAfter installing the dev-requirements, execute tests by running:\n\n pytest tests\n\nThese tests are located in the `./tests/local` folder and only require a Python interpreter. Pull requests will not be\naccepted if these tests do not pass. If you add new features, please include corresponding tests.\n\n### Cluster tests\n\nTests in the `./tests/cluster` folder are designed to run on a Databricks cluster.\nThe [Pre-integration Test](https://github.com/spetlr-org/spetlr/blob/main/.github/workflows/pre-integration.yml)\nutilizes Azure Resource deployment - and can only be run by the spetlr-org admins.\n\nTo deploy the necessary Azure resources to your own Azure Tenant, run the following command:\n\n```powershell\n.\\.github\\deploy\\deploy.ps1 -uniqueRunId \"yourUniqueId\"\n```\nBe aware that the applied name for *uniqueRunId* should only contain lower case and numbers, and its length should not\nexceed 12 characters.\n\nAfterward, execute the following commands:\n\n```powershell \n.\\.github\\submit\\build.ps1\n.\\.github\\submit\\submit_test_job.ps1\n```\n\n# General Project Info\n\n[](https://github.com/spetlr-org/spetlr)\n[](https://github.com/spetlr-org/spetlr)\n[](https://github.com/spetlr-org/spetlr)\n[](https://github.com/spetlr-org/spetlr)\n[](https://github.com/spetlr-org/spetlr/issues)\n[](https://pypi.org/project/spetlr/)\n\n# Contributing\n\nFeel free to contribute to SPETLR. Any contributions are appreciated - not only new features, but also if you find a way\nto improve SPETLR.\n\nIf you have a suggestion that can enhance SPETLR, please fork the repository and create a pull request. Alternatively,\nyou can open an issue with the \"enhancement\" tag.\n\n1. Fork the Project\n2. Create your Feature Branch (`git checkout -b feature/NewSPETLRFeature`)\n3. Commit your Changes (`git commit -m 'Add some SEPTLRFeature'`)\n4. Push to the Branch (`git push origin feature/NewSPETLRFeature`)\n5. Open a Pull Request\n\n# Build Status\n\n[](https://github.com/spetlr-org/spetlr/actions/workflows/post-integration.yml)\n\n# Releases\n\nReleases to PyPI is an Github Action which needs to be manually triggered.\n\n[](https://github.com/spetlr-org/spetlr/actions/workflows/release.yml)\n[](https://pypi.org/project/spetlr/)\n\n\n# Requirements and dependencies\nThe library has three txt-files at the root of the repo. These files defines three levels of requirements:\n- `requirements_install.txt` - this file contains the required libraries to be able to install spetlr.\n- `requirements_test.txt` - libraries required to run unit- and integration tests\n- `requirements_dev.txt` - libraries required in the development process in order to contribute to the repo\n\nAll libraries and their dependencies are added with a fixed version to the configuration file `setup.cfg` using the defined requirements from `requirements_install.txt`.\n\nTo __upgrade__ the the dependencies in the `setup.cfg` file do the following:\n\n1. Create a new branch\n2. Run `upgrade_requirements.ps1` in your terminal\n3. Commit the changes the script has made to the cfg file. If there are no changes, everything is up to date.\n4. The PR runs all tests and ensure that the library is compliant with any updates\n\nNote that if it is desired to upgrade a dependency, but not to its newest version, it is possible to set the desired version in the `requirements_install.txt`, then this will be respected by the upgrade script.\n\n# Contact\n\nFor any inquiries, please use the [SPETLR Discord Server](https://discord.gg/p9bzqGybVW).\n",
"bugtrack_url": null,
"license": null,
"summary": "A python ETL libRary (SPETLR) for Databricks powered by Apache SPark.",
"version": "14.3.5",
"project_urls": {
"Bug Reports": "https://github.com/spetlr-org/spetlr/issues",
"Documentation": "https://github.com/spetlr-org/spetlr",
"Homepage": "https://github.com/spetlr-org/spetlr",
"Source Code": "https://github.com/spetlr-org/spetlr"
},
"split_keywords": [
"databricks",
" pyspark"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ff03df0024b1bacdcc7277ab01235eb9e6022e06fa1d9012f86ca71cd0e4283c",
"md5": "f2f7bd5541d818599326f35fb0c18e4c",
"sha256": "020d139c68be5fb3e5a1327840a45795d7e75ef87212ff281354b5ac04195a1a"
},
"downloads": -1,
"filename": "spetlr-14.3.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f2f7bd5541d818599326f35fb0c18e4c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 164927,
"upload_time": "2024-11-23T09:01:23",
"upload_time_iso_8601": "2024-11-23T09:01:23.583169Z",
"url": "https://files.pythonhosted.org/packages/ff/03/df0024b1bacdcc7277ab01235eb9e6022e06fa1d9012f86ca71cd0e4283c/spetlr-14.3.5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "7b1d55a3326575f5c8b8be65d7a969b965a3312a3c8a62008afb3b944041c8e2",
"md5": "aa805a82312c25f7e1d292a6bd6f7c2c",
"sha256": "7addc603e3d98ee5ed5b3b78f5288639c641b8d08007696db7f1fb3cad117c99"
},
"downloads": -1,
"filename": "spetlr-14.3.5.tar.gz",
"has_sig": false,
"md5_digest": "aa805a82312c25f7e1d292a6bd6f7c2c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 183244,
"upload_time": "2024-11-23T09:01:26",
"upload_time_iso_8601": "2024-11-23T09:01:26.341823Z",
"url": "https://files.pythonhosted.org/packages/7b/1d/55a3326575f5c8b8be65d7a969b965a3312a3c8a62008afb3b944041c8e2/spetlr-14.3.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-23 09:01:26",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "spetlr-org",
"github_project": "spetlr",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "spetlr"
}
JSON
