# Sparta
Library to help ETL using Pyspark.
Sparta is a simple library to help you work on ETL builds using PySpark.
## Important Sources
- <a href="https://spark.apache.org/">Apache Spark</a>
- <a href="https://pypi.org/project/smart-open/">Smart Open</a>
- <a href="https://github.com/MrPowers/chispa">Chispa</a>
## Installation
Install the latest version with ```pip install pysparta```
## Documentation
<a href="https://jcpsantos.github.io/sparta/">Sparta</a>
## Modules
### Extract
This is a module with functions for extracting and reading data.
**Example**
```python
from sparta.extract import read_with_schema
schema = 'epidemiological_week LONG, date DATE, order_for_place INT, state STRING, city STRING, city_ibge_code LONG, place_type STRING, last_available_confirmed INT'
path = '/content/sample_data/covid19-e0534be4ad17411e81305aba2d9194d9.csv'
df = read_with_schema(path, schema, {'header': 'true'}, 'csv')
```
### Transformation
This is a module with data transformation functions
**Example**
```python
from sparta.transformation import drop_duplicates
cols = ['longitude','latitude']
df = drop_duplicates(df, 'population', cols)
```
### Load
This is a module with load and write functions.
**Example**
```python
from sparta.load import create_hive_table
create_hive_table(df, "table_name", 5, "col1", "col2", "col3")
```
### Others
This is a module with several functions that can help in ETL work.
**Example**
```python
from sparta.secret import get_secret_aws
get_secret_aws('Nome_Secret', 'sa-east-1')
```
## Supported PySpark / Python versions
Sparta currently supports PySpark 3.0+ and Python 3.7+.
Raw data
{
"_id": null,
"home_page": "https://github.com/jcpsantos/sparta",
"name": "pysparta",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": null,
"keywords": "spark etl data sparta",
"author": "Juan Caio",
"author_email": "juancaiops@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/8f/00/2d790036703224fdbe9aaab6cd43bb4d91a055923a4b28fa1f5bac3cd17e/pysparta-0.5.4.tar.gz",
"platform": null,
"description": "# Sparta\r\n\r\nLibrary to help ETL using Pyspark.\r\n\r\nSparta is a simple library to help you work on ETL builds using PySpark.\r\n\r\n## Important Sources\r\n\r\n- <a href=\"https://spark.apache.org/\">Apache Spark</a>\r\n- <a href=\"https://pypi.org/project/smart-open/\">Smart Open</a>\r\n- <a href=\"https://github.com/MrPowers/chispa\">Chispa</a>\r\n\r\n## Installation\r\n\r\nInstall the latest version with ```pip install pysparta```\r\n\r\n## Documentation\r\n\r\n<a href=\"https://jcpsantos.github.io/sparta/\">Sparta</a>\r\n\r\n## Modules\r\n\r\n### Extract\r\n\r\nThis is a module with functions for extracting and reading data.\r\n\r\n**Example**\r\n\r\n```python\r\nfrom sparta.extract import read_with_schema\r\n\r\nschema = 'epidemiological_week LONG, date DATE, order_for_place INT, state STRING, city STRING, city_ibge_code LONG, place_type STRING, last_available_confirmed INT'\r\npath = '/content/sample_data/covid19-e0534be4ad17411e81305aba2d9194d9.csv'\r\ndf = read_with_schema(path, schema, {'header': 'true'}, 'csv')\r\n```\r\n\r\n### Transformation\r\n\r\nThis is a module with data transformation functions\r\n\r\n**Example**\r\n\r\n```python\r\nfrom sparta.transformation import drop_duplicates\r\n\r\ncols = ['longitude','latitude']\r\ndf = drop_duplicates(df, 'population', cols)\r\n```\r\n\r\n### Load\r\n\r\nThis is a module with load and write functions.\r\n\r\n**Example**\r\n\r\n```python\r\nfrom sparta.load import create_hive_table\r\n\r\ncreate_hive_table(df, \"table_name\", 5, \"col1\", \"col2\", \"col3\")\r\n```\r\n\r\n### Others\r\n\r\nThis is a module with several functions that can help in ETL work.\r\n\r\n**Example**\r\n\r\n```python\r\nfrom sparta.secret import get_secret_aws\r\n\r\nget_secret_aws('Nome_Secret', 'sa-east-1')\r\n```\r\n\r\n## Supported PySpark / Python versions\r\n\r\nSparta currently supports PySpark 3.0+ and Python 3.7+.\r\n",
"bugtrack_url": null,
"license": "GNU General Public License v2.0",
"summary": "Library to help ETL using pyspark",
"version": "0.5.4",
"project_urls": {
"Documentation": "https://jcpsantos.github.io/sparta/",
"Homepage": "https://github.com/jcpsantos/sparta",
"Source code": "https://github.com/jcpsantos/sparta"
},
"split_keywords": [
"spark",
"etl",
"data",
"sparta"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "5b23bd8d1fdf201ea683a1f2812a8aabbc1b41dc275505f5b0e75ccf98f5c466",
"md5": "e1b9efd373a1a983e2f7108ef2c553d0",
"sha256": "f829b1d1698114f34771d3ca5fd20bfa72461bd518694d2099a4393d0c3fd27f"
},
"downloads": -1,
"filename": "pysparta-0.5.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "e1b9efd373a1a983e2f7108ef2c553d0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 22373,
"upload_time": "2024-10-29T17:43:57",
"upload_time_iso_8601": "2024-10-29T17:43:57.862360Z",
"url": "https://files.pythonhosted.org/packages/5b/23/bd8d1fdf201ea683a1f2812a8aabbc1b41dc275505f5b0e75ccf98f5c466/pysparta-0.5.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "8f002d790036703224fdbe9aaab6cd43bb4d91a055923a4b28fa1f5bac3cd17e",
"md5": "736f84c70ee2e8bbba2faf837022589f",
"sha256": "0ce9c05d6d6ee53f51f3e8c6b713d30e846b713bda838da695af6e564b97d42f"
},
"downloads": -1,
"filename": "pysparta-0.5.4.tar.gz",
"has_sig": false,
"md5_digest": "736f84c70ee2e8bbba2faf837022589f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 19505,
"upload_time": "2024-10-29T17:43:59",
"upload_time_iso_8601": "2024-10-29T17:43:59.242382Z",
"url": "https://files.pythonhosted.org/packages/8f/00/2d790036703224fdbe9aaab6cd43bb4d91a055923a4b28fa1f5bac3cd17e/pysparta-0.5.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-29 17:43:59",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "jcpsantos",
"github_project": "sparta",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "pysparta"
}
JSON
