# Sparta
Library to help ETL using Pyspark.
Sparta is a simple library to help you work on ETL builds using PySpark.
## Important Sources
- <a href="https://spark.apache.org/">Apache Spark</a>
- <a href="https://pypi.org/project/smart-open/">Smart Open</a>
- <a href="https://github.com/MrPowers/chispa">Chispa</a>
## Installation
Install the latest version with ```pip install pysparta```
## Documentation
<a href="https://jcpsantos.github.io/sparta/">Sparta</a>
## Modules
### Extract
This is a module with functions for extracting and reading data.
**Example**
```python
from sparta.extract import read_with_schema
schema = 'epidemiological_week LONG, date DATE, order_for_place INT, state STRING, city STRING, city_ibge_code LONG, place_type STRING, last_available_confirmed INT'
path = '/content/sample_data/covid19-e0534be4ad17411e81305aba2d9194d9.csv'
df = read_with_schema(path, schema, {'header': 'true'}, 'csv')
```
### Transformation
This is a module with data transformation functions
**Example**
```python
from sparta.transformation import drop_duplicates
cols = ['longitude','latitude']
df = drop_duplicates(df, 'population', cols)
```
### Load
This is a module with load and write functions.
**Example**
```python
from sparta.load import create_hive_table
create_hive_table(df, "table_name", 5, "col1", "col2", "col3")
```
### Others
This is a module with several functions that can help in ETL work.
**Example**
```python
from sparta.secret import get_secret_aws
get_secret_aws('Nome_Secret', 'sa-east-1')
```
## Supported PySpark / Python versions
Sparta currently supports PySpark 3.0+ and Python 3.7+.
Raw data
{
"_id": null,
"home_page": "https://github.com/jcpsantos/sparta",
"name": "pysparta",
"maintainer": "",
"docs_url": null,
"requires_python": ">= 3.7",
"maintainer_email": "",
"keywords": "spark etl data sparta",
"author": "Juan Caio",
"author_email": "juancaiops@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/7a/07/edfae5d6a1efd769070528d3028c91850c8f10ee37503df92d695b82e848/pysparta-0.4.2.tar.gz",
"platform": null,
"description": "# Sparta\r\n\r\nLibrary to help ETL using Pyspark.\r\n\r\nSparta is a simple library to help you work on ETL builds using PySpark.\r\n\r\n## Important Sources\r\n\r\n- <a href=\"https://spark.apache.org/\">Apache Spark</a>\r\n- <a href=\"https://pypi.org/project/smart-open/\">Smart Open</a>\r\n- <a href=\"https://github.com/MrPowers/chispa\">Chispa</a>\r\n\r\n## Installation\r\n\r\nInstall the latest version with ```pip install pysparta```\r\n\r\n## Documentation\r\n\r\n<a href=\"https://jcpsantos.github.io/sparta/\">Sparta</a>\r\n\r\n## Modules\r\n\r\n### Extract\r\n\r\nThis is a module with functions for extracting and reading data.\r\n\r\n**Example**\r\n\r\n```python\r\nfrom sparta.extract import read_with_schema\r\n\r\nschema = 'epidemiological_week LONG, date DATE, order_for_place INT, state STRING, city STRING, city_ibge_code LONG, place_type STRING, last_available_confirmed INT'\r\npath = '/content/sample_data/covid19-e0534be4ad17411e81305aba2d9194d9.csv'\r\ndf = read_with_schema(path, schema, {'header': 'true'}, 'csv')\r\n```\r\n\r\n### Transformation\r\n\r\nThis is a module with data transformation functions\r\n\r\n**Example**\r\n\r\n```python\r\nfrom sparta.transformation import drop_duplicates\r\n\r\ncols = ['longitude','latitude']\r\ndf = drop_duplicates(df, 'population', cols)\r\n```\r\n\r\n### Load\r\n\r\nThis is a module with load and write functions.\r\n\r\n**Example**\r\n\r\n```python\r\nfrom sparta.load import create_hive_table\r\n\r\ncreate_hive_table(df, \"table_name\", 5, \"col1\", \"col2\", \"col3\")\r\n```\r\n\r\n### Others\r\n\r\nThis is a module with several functions that can help in ETL work.\r\n\r\n**Example**\r\n\r\n```python\r\nfrom sparta.secret import get_secret_aws\r\n\r\nget_secret_aws('Nome_Secret', 'sa-east-1')\r\n```\r\n\r\n## Supported PySpark / Python versions\r\n\r\nSparta currently supports PySpark 3.0+ and Python 3.7+.\r\n",
"bugtrack_url": null,
"license": "GNU General Public License v2.0",
"summary": "Library to help ETL using pyspark",
"version": "0.4.2",
"split_keywords": [
"spark",
"etl",
"data",
"sparta"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "069225bc677be942445f3767d1b3412f6836fe1bfedbc25e85b5f0f00c711eeb",
"md5": "f96108465af5f01d705070482d39c334",
"sha256": "bd22fb5a737fe5415b3ad0670eab724ed06675a75685ab1ae4aaa8edd6fcdd0e"
},
"downloads": -1,
"filename": "pysparta-0.4.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f96108465af5f01d705070482d39c334",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">= 3.7",
"size": 16250,
"upload_time": "2023-03-11T20:04:52",
"upload_time_iso_8601": "2023-03-11T20:04:52.063669Z",
"url": "https://files.pythonhosted.org/packages/06/92/25bc677be942445f3767d1b3412f6836fe1bfedbc25e85b5f0f00c711eeb/pysparta-0.4.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "7a07edfae5d6a1efd769070528d3028c91850c8f10ee37503df92d695b82e848",
"md5": "fabfb9dec0e7a4400a6c791f0fcb9b29",
"sha256": "0f05f7c6351b023a60b4a550a353b7a3e8fdb6549d61d36bcff50a78bbde3dd5"
},
"downloads": -1,
"filename": "pysparta-0.4.2.tar.gz",
"has_sig": false,
"md5_digest": "fabfb9dec0e7a4400a6c791f0fcb9b29",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">= 3.7",
"size": 14247,
"upload_time": "2023-03-11T20:05:01",
"upload_time_iso_8601": "2023-03-11T20:05:01.083062Z",
"url": "https://files.pythonhosted.org/packages/7a/07/edfae5d6a1efd769070528d3028c91850c8f10ee37503df92d695b82e848/pysparta-0.4.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-03-11 20:05:01",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "jcpsantos",
"github_project": "sparta",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "pysparta"
}
JSON
