[](https://github.com/godatadriven/pydantic-spark/actions/workflows/python-package.yml)
[](https://codecov.io/gh/godatadriven/pydantic-spark)
[](https://badge.fury.io/py/pydantic-spark)
[](https://github.com/godatadriven/pydantic-spark/actions/workflows/codeql-analysis.yml)
# pydantic-spark
This library can convert a pydantic class to a spark schema or generate python code from a spark schema.
### Install
```bash
pip install pydantic-spark
```
### Pydantic class to spark schema
```python
import json
from typing import Optional
from pydantic_spark.base import SparkBase
class TestModel(SparkBase):
key1: str
key2: int
key2: Optional[str]
schema_dict: dict = TestModel.spark_schema()
print(json.dumps(schema_dict))
```
#### Coerce type
Pydantic-spark provides a `coerce_type` option that allows type coercion.
When applied to a field, pydantic-spark converts the column's data type to the specified coercion type.
```python
import json
from pydantic import Field
from pydantic_spark.base import SparkBase, CoerceType
class TestModel(SparkBase):
key1: str = Field(extra_json_schema={"coerce_type": CoerceType.integer})
schema_dict: dict = TestModel.spark_schema()
print(json.dumps(schema_dict))
```
### Install for developers
###### Install package
- Requirement: Poetry 1.*
```shell
poetry install
```
###### Run unit tests
```shell
pytest
coverage run -m pytest # with coverage
# or (depends on your local env)
poetry run pytest
poetry run coverage run -m pytest # with coverage
```
##### Run linting
The linting is checked in the github workflow. To fix and review issues run this:
```shell
black . # Auto fix all issues
isort . # Auto fix all issues
pflake . # Only display issues, fixing is manual
```
Raw data
{
"_id": null,
"home_page": "https://github.com/godatadriven/pydantic-spark",
"name": "pydantic-spark",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8,<4.0",
"maintainer_email": "",
"keywords": "pydantic,spark",
"author": "Peter van 't Hof'",
"author_email": "peter.vanthof@godatadriven.com",
"download_url": "https://files.pythonhosted.org/packages/a3/46/6c0e67c3b6fdd3cb9c11b2bca9644cf57ddfa1f1bd20a1687b7a9cf8e8f3/pydantic_spark-1.0.1.tar.gz",
"platform": null,
"description": "[](https://github.com/godatadriven/pydantic-spark/actions/workflows/python-package.yml)\n[](https://codecov.io/gh/godatadriven/pydantic-spark)\n[](https://badge.fury.io/py/pydantic-spark)\n[](https://github.com/godatadriven/pydantic-spark/actions/workflows/codeql-analysis.yml)\n\n# pydantic-spark\n\nThis library can convert a pydantic class to a spark schema or generate python code from a spark schema.\n\n### Install\n\n```bash\npip install pydantic-spark\n```\n\n### Pydantic class to spark schema\n\n```python\nimport json\nfrom typing import Optional\n\nfrom pydantic_spark.base import SparkBase\n\nclass TestModel(SparkBase):\n key1: str\n key2: int\n key2: Optional[str]\n\nschema_dict: dict = TestModel.spark_schema()\nprint(json.dumps(schema_dict))\n\n```\n#### Coerce type\nPydantic-spark provides a `coerce_type` option that allows type coercion. \nWhen applied to a field, pydantic-spark converts the column's data type to the specified coercion type. \n\n```python\nimport json\nfrom pydantic import Field\nfrom pydantic_spark.base import SparkBase, CoerceType\n\nclass TestModel(SparkBase):\n key1: str = Field(extra_json_schema={\"coerce_type\": CoerceType.integer})\n\nschema_dict: dict = TestModel.spark_schema()\nprint(json.dumps(schema_dict))\n\n```\n\n\n### Install for developers\n\n###### Install package\n\n- Requirement: Poetry 1.*\n\n```shell\npoetry install\n```\n\n###### Run unit tests\n```shell\npytest\ncoverage run -m pytest # with coverage\n# or (depends on your local env) \npoetry run pytest\npoetry run coverage run -m pytest # with coverage\n```\n\n##### Run linting\n\nThe linting is checked in the github workflow. To fix and review issues run this:\n```shell\nblack . # Auto fix all issues\nisort . # Auto fix all issues\npflake . # Only display issues, fixing is manual\n```\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Converting pydantic classes to spark schemas",
"version": "1.0.1",
"project_urls": {
"Homepage": "https://github.com/godatadriven/pydantic-spark",
"Repository": "https://github.com/godatadriven/pydantic-spark"
},
"split_keywords": [
"pydantic",
"spark"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a223b993e9b3901a9b2e143bf8c5da4419a671dfa921cd07202accf351be2bea",
"md5": "3a14f8131f30cc4ee42b47ee7c48e843",
"sha256": "51900d2e273b1be232fc1e0fb7c2d259f179daa1edd25754e07e4f7653def32e"
},
"downloads": -1,
"filename": "pydantic_spark-1.0.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3a14f8131f30cc4ee42b47ee7c48e843",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8,<4.0",
"size": 6338,
"upload_time": "2023-11-24T15:12:14",
"upload_time_iso_8601": "2023-11-24T15:12:14.897329Z",
"url": "https://files.pythonhosted.org/packages/a2/23/b993e9b3901a9b2e143bf8c5da4419a671dfa921cd07202accf351be2bea/pydantic_spark-1.0.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a3466c0e67c3b6fdd3cb9c11b2bca9644cf57ddfa1f1bd20a1687b7a9cf8e8f3",
"md5": "7be185afb8f29035993159d1a131b92d",
"sha256": "ddfa8ad977de941e240310b5c0a2aefbc33f92dd303f5a6d06e9bb4ba90959e8"
},
"downloads": -1,
"filename": "pydantic_spark-1.0.1.tar.gz",
"has_sig": false,
"md5_digest": "7be185afb8f29035993159d1a131b92d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8,<4.0",
"size": 4833,
"upload_time": "2023-11-24T15:12:16",
"upload_time_iso_8601": "2023-11-24T15:12:16.508294Z",
"url": "https://files.pythonhosted.org/packages/a3/46/6c0e67c3b6fdd3cb9c11b2bca9644cf57ddfa1f1bd20a1687b7a9cf8e8f3/pydantic_spark-1.0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-24 15:12:16",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "godatadriven",
"github_project": "pydantic-spark",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "pydantic-spark"
}
JSON
