| Name | pyspark-testframework JSON |
| Version |
2.0.0
JSON |
| download |
| home_page | None |
| Summary | Testframework for PySpark DataFrames |
| upload_time | 2024-06-22 17:07:44 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.9.5 |
| license | MIT License Copyright (c) 2024 Woonstad Rotterdam Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
pyspark
dataframe
test
testframework
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
[](https://pypi.org/project/pyspark-testframework/)
[](https://github.com/astral-sh/ruff)
# pyspark-testframework
⏳ **Work in progress**
&width=145>)
The goal of the `pyspark-testframework` is to provide a simple way to create tests for PySpark DataFrames. The test results are returned in DataFrame format as well.
# Tutorial
**Let's first create an example pyspark DataFrame**
The data will contain the primary keys, street names and house numbers of some addresses.
```python
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, IntegerType, StringType
from pyspark.sql import functions as F
```
```python
# Initialize Spark session
spark = SparkSession.builder.appName("CreateDataFrame").getOrCreate()
# Define the schema
schema = StructType(
[
StructField("primary_key", IntegerType(), True),
StructField("street", StringType(), True),
StructField("house_number", IntegerType(), True),
]
)
# Define the data
data = [
(1, "Rochussenstraat", 27),
(2, "Coolsingel", 31),
(3, "%Witte de Withstraat", 27),
(4, "Lijnbaan", -3),
(5, None, 13),
]
df = spark.createDataFrame(data, schema)
df.show(truncate=False)
```
+-----------+--------------------+------------+
|primary_key|street |house_number|
+-----------+--------------------+------------+
|1 |Rochussenstraat |27 |
|2 |Coolsingel |31 |
|3 |%Witte de Withstraat|27 |
|4 |Lijnbaan |-3 |
|5 |null |13 |
+-----------+--------------------+------------+
**Import and initialize the `DataFrameTester`**
```python
from testframework.dataquality import DataFrameTester
```
```python
df_tester = DataFrameTester(
df=df,
primary_key="primary_key",
spark=spark,
)
```
**Import configurable tests**
```python
from testframework.dataquality.tests import ValidNumericRange, RegexTest
```
**Initialize the `RegexTest` to test for valid street names**
```python
valid_street_name = RegexTest(
name="ValidStreetName",
pattern=r"^[A-Z][a-zéèáàëï]*([ -][A-Z]?[a-zéèáàëï]*)*$",
)
```
**Run `valid_street_name` on the _street_ column using the `.test()` method of `DataFrameTester`.**
```python
df_tester.test(
col="street",
test=valid_street_name,
nullable=False, # nullable, hence null values are converted to True
description="street contains valid Dutch street name.",
).show(truncate=False)
```
+-----------+--------------------+-----------------------+
|primary_key|street |street__ValidStreetName|
+-----------+--------------------+-----------------------+
|1 |Rochussenstraat |true |
|2 |Coolsingel |true |
|3 |%Witte de Withstraat|false |
|4 |Lijnbaan |true |
|5 |null |false |
+-----------+--------------------+-----------------------+
**Run the `IntegerString` test on the _number_ column**
```python
df_tester.test(
col="house_number",
test=ValidNumericRange(
min_value=0,
),
nullable=True, # nullable, hence null values are converted to True
# description is optional, let's not define it for illustration purposes
).show()
```
+-----------+------------+-------------------------------+
|primary_key|house_number|house_number__ValidNumericRange|
+-----------+------------+-------------------------------+
| 1| 27| true|
| 2| 31| true|
| 3| 27| true|
| 4| -3| false|
| 5| 13| true|
+-----------+------------+-------------------------------+
**Let's take a look at the test results of the DataFrame using the `.results` attribute.**
```python
df_tester.results.show(truncate=False)
```
+-----------+-----------------------+-------------------------------+
|primary_key|street__ValidStreetName|house_number__ValidNumericRange|
+-----------+-----------------------+-------------------------------+
|1 |true |true |
|2 |true |true |
|3 |false |true |
|4 |true |false |
|5 |false |true |
+-----------+-----------------------+-------------------------------+
**We can use `.descriptions` or `.descriptions_df` to get the descriptions of the tests.**
<br>
This can be useful for reporting purposes.
For example to create reports for the business with more detailed information than just the column name and the test name.
```python
df_tester.descriptions
```
{'street__ValidStreetName': 'street contains valid Dutch street name.',
'house_number__ValidNumericRange': 'house_number__ValidNumericRange(min_value=0.0, max_value=inf)'}
```python
df_tester.description_df.show(truncate=False)
```
+-------------------------------+-------------------------------------------------------------+
|test |description |
+-------------------------------+-------------------------------------------------------------+
|street__ValidStreetName |street contains valid Dutch street name. |
|house_number__ValidNumericRange|house_number__ValidNumericRange(min_value=0.0, max_value=inf)|
+-------------------------------+-------------------------------------------------------------+
### Custom tests
Sometimes tests are too specific or complex to be covered by the configurable tests. That's why we can create custom tests and add them to the `DataFrameTester` object.
Let's do this using a custom test which should tests that every house has a bath room. We'll start by creating a new DataFrame with rooms rather than houses.
```python
rooms = [
(1, "living room"),
(1, "bath room"),
(1, "kitchen"),
(1, "bed room"),
(2, "living room"),
(2, "bed room"),
(2, "kitchen"),
]
schema_rooms = StructType(
[
StructField("primary_key", IntegerType(), True),
StructField("room", StringType(), True),
]
)
room_df = spark.createDataFrame(rooms, schema=schema_rooms)
room_df.show(truncate=False)
```
+-----------+-----------+
|primary_key|room |
+-----------+-----------+
|1 |living room|
|1 |bath room |
|1 |kitchen |
|1 |bed room |
|2 |living room|
|2 |bed room |
|2 |kitchen |
+-----------+-----------+
To create a custom test, we should create a pyspark DataFrame which contains the same primary_key column as the DataFrame to be tested using the `DataFrameTester`.
Let's create a boolean column that indicates whether the house has a bath room or not.
```python
house_has_bath_room = room_df.groupBy("primary_key").agg(
F.max(F.when(F.col("room") == "bath room", 1).otherwise(0)).alias("has_bath_room")
)
house_has_bath_room.show(truncate=False)
```
+-----------+-------------+
|primary_key|has_bath_room|
+-----------+-------------+
|1 |1 |
|2 |0 |
+-----------+-------------+
**We can add this 'custom test' to the `DataFrameTester` using `add_custom_test_result`.**
In the background, all kinds of data validation checks are done by `DataFrameTester` to make sure that it fits the requirements to be added to the other test results.
```python
df_tester.add_custom_test_result(
result=house_has_bath_room,
name="has_bath_room",
description="House has a bath room",
# fillna_value=0, # optional; by default null.
).show(truncate=False)
```
+-----------+-------------+
|primary_key|has_bath_room|
+-----------+-------------+
|1 |1 |
|2 |0 |
|3 |null |
|4 |null |
|5 |null |
+-----------+-------------+
**Despite that the data whether a house has a bath room is not available in the house DataFrame; we can still add the custom test to the `DataFrameTester` object.**
```python
df_tester.results.show(truncate=False)
```
+-----------+-----------------------+-------------------------------+-------------+
|primary_key|street__ValidStreetName|house_number__ValidNumericRange|has_bath_room|
+-----------+-----------------------+-------------------------------+-------------+
|1 |true |true |1 |
|2 |true |true |0 |
|3 |false |true |null |
|4 |true |false |null |
|5 |false |true |null |
+-----------+-----------------------+-------------------------------+-------------+
```python
df_tester.descriptions
```
{'street__ValidStreetName': 'street contains valid Dutch street name.',
'house_number__ValidNumericRange': 'house_number__ValidNumericRange(min_value=0.0, max_value=inf)',
'has_bath_room': 'House has a bath room'}
Raw data
{
"_id": null,
"home_page": null,
"name": "pyspark-testframework",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9.5",
"maintainer_email": null,
"keywords": "pyspark, dataframe, test, testframework",
"author": null,
"author_email": "Woonstad Rotterdam <info@woonstadrotterdam.nl>, Tomer Gabay <tomer.gabay@woonstadrotterdam.nl>, Vincent van der Meij <vincent.van.der.meij@woonstadrotterdam.nl>, Tiddo Loos <tiddo.loos@woonstadrotterdam.nl>, Ben Verhees <ben.verhees@woonstadrotterdam.nl>",
"download_url": "https://files.pythonhosted.org/packages/68/76/a0e329d3c073d19d31423aaad90de16de0e829c45db558e9b158af4fa442/pyspark_testframework-2.0.0.tar.gz",
"platform": null,
"description": "\n\n[](https://pypi.org/project/pyspark-testframework/)\n\n[](https://github.com/astral-sh/ruff)\n\n# pyspark-testframework\n\n\u23f3 **Work in progress**\n\n \n \n \n \n \n \n \n \n&width=145>)\n\nThe goal of the `pyspark-testframework` is to provide a simple way to create tests for PySpark DataFrames. The test results are returned in DataFrame format as well.\n\n# Tutorial\n\n**Let's first create an example pyspark DataFrame**\n\nThe data will contain the primary keys, street names and house numbers of some addresses.\n\n```python\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql.types import StructType, StructField, IntegerType, StringType\nfrom pyspark.sql import functions as F\n```\n\n```python\n# Initialize Spark session\nspark = SparkSession.builder.appName(\"CreateDataFrame\").getOrCreate()\n\n# Define the schema\nschema = StructType(\n [\n StructField(\"primary_key\", IntegerType(), True),\n StructField(\"street\", StringType(), True),\n StructField(\"house_number\", IntegerType(), True),\n ]\n)\n\n# Define the data\ndata = [\n (1, \"Rochussenstraat\", 27),\n (2, \"Coolsingel\", 31),\n (3, \"%Witte de Withstraat\", 27),\n (4, \"Lijnbaan\", -3),\n (5, None, 13),\n]\n\ndf = spark.createDataFrame(data, schema)\n\ndf.show(truncate=False)\n```\n\n +-----------+--------------------+------------+\n |primary_key|street |house_number|\n +-----------+--------------------+------------+\n |1 |Rochussenstraat |27 |\n |2 |Coolsingel |31 |\n |3 |%Witte de Withstraat|27 |\n |4 |Lijnbaan |-3 |\n |5 |null |13 |\n +-----------+--------------------+------------+\n\n**Import and initialize the `DataFrameTester`**\n\n```python\nfrom testframework.dataquality import DataFrameTester\n```\n\n```python\ndf_tester = DataFrameTester(\n df=df,\n primary_key=\"primary_key\",\n spark=spark,\n)\n```\n\n**Import configurable tests**\n\n```python\nfrom testframework.dataquality.tests import ValidNumericRange, RegexTest\n```\n\n**Initialize the `RegexTest` to test for valid street names**\n\n```python\nvalid_street_name = RegexTest(\n name=\"ValidStreetName\",\n pattern=r\"^[A-Z][a-z\u00e9\u00e8\u00e1\u00e0\u00eb\u00ef]*([ -][A-Z]?[a-z\u00e9\u00e8\u00e1\u00e0\u00eb\u00ef]*)*$\",\n)\n```\n\n**Run `valid_street_name` on the _street_ column using the `.test()` method of `DataFrameTester`.**\n\n```python\ndf_tester.test(\n col=\"street\",\n test=valid_street_name,\n nullable=False, # nullable, hence null values are converted to True\n description=\"street contains valid Dutch street name.\",\n).show(truncate=False)\n```\n\n +-----------+--------------------+-----------------------+\n |primary_key|street |street__ValidStreetName|\n +-----------+--------------------+-----------------------+\n |1 |Rochussenstraat |true |\n |2 |Coolsingel |true |\n |3 |%Witte de Withstraat|false |\n |4 |Lijnbaan |true |\n |5 |null |false |\n +-----------+--------------------+-----------------------+\n\n**Run the `IntegerString` test on the _number_ column**\n\n```python\ndf_tester.test(\n col=\"house_number\",\n test=ValidNumericRange(\n min_value=0,\n ),\n nullable=True, # nullable, hence null values are converted to True\n # description is optional, let's not define it for illustration purposes\n).show()\n```\n\n +-----------+------------+-------------------------------+\n |primary_key|house_number|house_number__ValidNumericRange|\n +-----------+------------+-------------------------------+\n | 1| 27| true|\n | 2| 31| true|\n | 3| 27| true|\n | 4| -3| false|\n | 5| 13| true|\n +-----------+------------+-------------------------------+\n\n**Let's take a look at the test results of the DataFrame using the `.results` attribute.**\n\n```python\ndf_tester.results.show(truncate=False)\n```\n\n +-----------+-----------------------+-------------------------------+\n |primary_key|street__ValidStreetName|house_number__ValidNumericRange|\n +-----------+-----------------------+-------------------------------+\n |1 |true |true |\n |2 |true |true |\n |3 |false |true |\n |4 |true |false |\n |5 |false |true |\n +-----------+-----------------------+-------------------------------+\n\n**We can use `.descriptions` or `.descriptions_df` to get the descriptions of the tests.**\n\n<br>\nThis can be useful for reporting purposes. \nFor example to create reports for the business with more detailed information than just the column name and the test name.\n\n```python\ndf_tester.descriptions\n```\n\n {'street__ValidStreetName': 'street contains valid Dutch street name.',\n 'house_number__ValidNumericRange': 'house_number__ValidNumericRange(min_value=0.0, max_value=inf)'}\n\n```python\ndf_tester.description_df.show(truncate=False)\n```\n\n +-------------------------------+-------------------------------------------------------------+\n |test |description |\n +-------------------------------+-------------------------------------------------------------+\n |street__ValidStreetName |street contains valid Dutch street name. |\n |house_number__ValidNumericRange|house_number__ValidNumericRange(min_value=0.0, max_value=inf)|\n +-------------------------------+-------------------------------------------------------------+\n\n### Custom tests\n\nSometimes tests are too specific or complex to be covered by the configurable tests. That's why we can create custom tests and add them to the `DataFrameTester` object.\n\nLet's do this using a custom test which should tests that every house has a bath room. We'll start by creating a new DataFrame with rooms rather than houses.\n\n```python\nrooms = [\n (1, \"living room\"),\n (1, \"bath room\"),\n (1, \"kitchen\"),\n (1, \"bed room\"),\n (2, \"living room\"),\n (2, \"bed room\"),\n (2, \"kitchen\"),\n]\n\nschema_rooms = StructType(\n [\n StructField(\"primary_key\", IntegerType(), True),\n StructField(\"room\", StringType(), True),\n ]\n)\n\nroom_df = spark.createDataFrame(rooms, schema=schema_rooms)\n\nroom_df.show(truncate=False)\n```\n\n +-----------+-----------+\n |primary_key|room |\n +-----------+-----------+\n |1 |living room|\n |1 |bath room |\n |1 |kitchen |\n |1 |bed room |\n |2 |living room|\n |2 |bed room |\n |2 |kitchen |\n +-----------+-----------+\n\nTo create a custom test, we should create a pyspark DataFrame which contains the same primary_key column as the DataFrame to be tested using the `DataFrameTester`.\n\nLet's create a boolean column that indicates whether the house has a bath room or not.\n\n```python\nhouse_has_bath_room = room_df.groupBy(\"primary_key\").agg(\n F.max(F.when(F.col(\"room\") == \"bath room\", 1).otherwise(0)).alias(\"has_bath_room\")\n)\n\nhouse_has_bath_room.show(truncate=False)\n```\n\n +-----------+-------------+\n |primary_key|has_bath_room|\n +-----------+-------------+\n |1 |1 |\n |2 |0 |\n +-----------+-------------+\n\n**We can add this 'custom test' to the `DataFrameTester` using `add_custom_test_result`.**\n\nIn the background, all kinds of data validation checks are done by `DataFrameTester` to make sure that it fits the requirements to be added to the other test results.\n\n```python\ndf_tester.add_custom_test_result(\n result=house_has_bath_room,\n name=\"has_bath_room\",\n description=\"House has a bath room\",\n # fillna_value=0, # optional; by default null.\n).show(truncate=False)\n```\n\n +-----------+-------------+\n |primary_key|has_bath_room|\n +-----------+-------------+\n |1 |1 |\n |2 |0 |\n |3 |null |\n |4 |null |\n |5 |null |\n +-----------+-------------+\n\n**Despite that the data whether a house has a bath room is not available in the house DataFrame; we can still add the custom test to the `DataFrameTester` object.**\n\n```python\ndf_tester.results.show(truncate=False)\n```\n\n +-----------+-----------------------+-------------------------------+-------------+\n |primary_key|street__ValidStreetName|house_number__ValidNumericRange|has_bath_room|\n +-----------+-----------------------+-------------------------------+-------------+\n |1 |true |true |1 |\n |2 |true |true |0 |\n |3 |false |true |null |\n |4 |true |false |null |\n |5 |false |true |null |\n +-----------+-----------------------+-------------------------------+-------------+\n\n```python\ndf_tester.descriptions\n```\n\n {'street__ValidStreetName': 'street contains valid Dutch street name.',\n 'house_number__ValidNumericRange': 'house_number__ValidNumericRange(min_value=0.0, max_value=inf)',\n 'has_bath_room': 'House has a bath room'}\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) 2024 Woonstad Rotterdam Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
"summary": "Testframework for PySpark DataFrames",
"version": "2.0.0",
"project_urls": {
"Homepage": "https://github.com/woonstadrotterdam/pyspark-testframework",
"Issues": "https://github.com/woonstadrotterdam/pyspark-testframework/issues"
},
"split_keywords": [
"pyspark",
" dataframe",
" test",
" testframework"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2882d5a5c28c09f1b10ba08e091ca778a3f084f0c483ce908badde618e87a9e2",
"md5": "3d9734c8b601c17a4f0d677237163d13",
"sha256": "c1faeafa66908bb26a2dc68cde910dca8792de4175ce0b6c2340e74a7b5f0fce"
},
"downloads": -1,
"filename": "pyspark_testframework-2.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3d9734c8b601c17a4f0d677237163d13",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9.5",
"size": 16184,
"upload_time": "2024-06-22T17:07:42",
"upload_time_iso_8601": "2024-06-22T17:07:42.825620Z",
"url": "https://files.pythonhosted.org/packages/28/82/d5a5c28c09f1b10ba08e091ca778a3f084f0c483ce908badde618e87a9e2/pyspark_testframework-2.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6876a0e329d3c073d19d31423aaad90de16de0e829c45db558e9b158af4fa442",
"md5": "9d32badfee8ed1ce6dc045a1b5b6aa9f",
"sha256": "8f2ef03b5febd74c5f4615c7e7cb38fc945b18b7f561e174e00c6fb0d92d1c02"
},
"downloads": -1,
"filename": "pyspark_testframework-2.0.0.tar.gz",
"has_sig": false,
"md5_digest": "9d32badfee8ed1ce6dc045a1b5b6aa9f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9.5",
"size": 25374,
"upload_time": "2024-06-22T17:07:44",
"upload_time_iso_8601": "2024-06-22T17:07:44.003857Z",
"url": "https://files.pythonhosted.org/packages/68/76/a0e329d3c073d19d31423aaad90de16de0e829c45db558e9b158af4fa442/pyspark_testframework-2.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-06-22 17:07:44",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "woonstadrotterdam",
"github_project": "pyspark-testframework",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "pyspark-testframework"
}
