# Pyspark IOMETE Library
This library is providing a set of utility functions to speed up the development of pyspark applications.
## Installation
```bash
pip install pyspark-iomete
```
## Utility functions
### get_spark_logger
This function is returning a spark logger instance.
As you may know, spark is using log4j as a logging framework. This function is returning a spark logger instance that is using the log4j logger. Standard python logging is not working with pyspark. The following function get the spark logger instance and returns it.
Usage:
```python
from pyspark_iomete.utils import get_spark_logger
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
# spark session and name will be used to create the logger
# both are optional
logger = get_spark_logger(spark=spark, name="my_custom_logger")
# spark session will be retrieved using SparkSession.getActiveSession() and name will be set to the current file name
logger = get_spark_logger()
```
## Test utility functions
### table_name_with_random_suffix
This function is returning a table name with a random suffix. This is useful for testing purposes.
Usage:
```python
from pyspark_iomete.test_utils import table_name_with_random_suffix
table_name = table_name_with_random_suffix("my_table")
```
Raw data
{
"_id": null,
"home_page": "https://github.com/iomete/pyspark-iomete",
"name": "pyspark-iomete",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "iomete,pyspark,utils",
"author": "IOMETE",
"author_email": "vusal@iomete.com",
"download_url": "https://files.pythonhosted.org/packages/b8/25/1857521401a105e744100e7417fb79f1d6592b63f7cdfec9c8e06e50f0c5/pyspark_iomete-0.0.3.tar.gz",
"platform": null,
"description": "# Pyspark IOMETE Library\n\nThis library is providing a set of utility functions to speed up the development of pyspark applications.\n\n\n## Installation\n\n```bash\npip install pyspark-iomete\n```\n\n\n## Utility functions\n\n### get_spark_logger\n\nThis function is returning a spark logger instance.\n\nAs you may know, spark is using log4j as a logging framework. This function is returning a spark logger instance that is using the log4j logger. Standard python logging is not working with pyspark. The following function get the spark logger instance and returns it.\n\nUsage:\n```python\nfrom pyspark_iomete.utils import get_spark_logger\nfrom pyspark.sql import SparkSession\n\nspark = SparkSession.builder.getOrCreate()\n\n# spark session and name will be used to create the logger\n# both are optional\nlogger = get_spark_logger(spark=spark, name=\"my_custom_logger\")\n\n# spark session will be retrieved using SparkSession.getActiveSession() and name will be set to the current file name\nlogger = get_spark_logger()\n```\n\n## Test utility functions\n\n### table_name_with_random_suffix\n\nThis function is returning a table name with a random suffix. This is useful for testing purposes.\n\nUsage:\n```python\nfrom pyspark_iomete.test_utils import table_name_with_random_suffix\n\ntable_name = table_name_with_random_suffix(\"my_table\")\n```\n \n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "IOMETE's PySpark library that contains useful utilities for working with PySpark",
"version": "0.0.3",
"project_urls": {
"Homepage": "https://github.com/iomete/pyspark-iomete"
},
"split_keywords": [
"iomete",
"pyspark",
"utils"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e00f532da655af96a602449754e6778fc9b72055ca6365894250233696e0a561",
"md5": "65ec9ffede954826088fa1e7f8cb8568",
"sha256": "410c11953bab712ce696d8f78df80cfedcee048277a4a1769b13c196526d7e14"
},
"downloads": -1,
"filename": "pyspark_iomete-0.0.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "65ec9ffede954826088fa1e7f8cb8568",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 2783,
"upload_time": "2023-07-11T15:36:49",
"upload_time_iso_8601": "2023-07-11T15:36:49.128453Z",
"url": "https://files.pythonhosted.org/packages/e0/0f/532da655af96a602449754e6778fc9b72055ca6365894250233696e0a561/pyspark_iomete-0.0.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b8251857521401a105e744100e7417fb79f1d6592b63f7cdfec9c8e06e50f0c5",
"md5": "0346cc2626193b07d9cfe5a3f30eeead",
"sha256": "ab58af5d41bea82a97219c0ff8578dd6c84581fd4a8a97a4eff2ab2129a57b00"
},
"downloads": -1,
"filename": "pyspark_iomete-0.0.3.tar.gz",
"has_sig": false,
"md5_digest": "0346cc2626193b07d9cfe5a3f30eeead",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 2462,
"upload_time": "2023-07-11T15:36:50",
"upload_time_iso_8601": "2023-07-11T15:36:50.566786Z",
"url": "https://files.pythonhosted.org/packages/b8/25/1857521401a105e744100e7417fb79f1d6592b63f7cdfec9c8e06e50f0c5/pyspark_iomete-0.0.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-07-11 15:36:50",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "iomete",
"github_project": "pyspark-iomete",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "pyspark-iomete"
}
JSON
