# YAML Testing Framework
A simple, low-code framework for unit testing in Python with tests are defined in YAML files.
## Features
- Standardized data/tests defined in YAML files
- App functions as a pytest plugin
- Support functional programming
- Can be used to test synchronous and asynchronous logic
- Easy to use API
## Requirements
Python 3.7+
## Installation
```bash
pip install yaml-testing-framework
```
## Example
Create the files
- `test_entrypoint.py` - uses pytest to collect and run tests
```python
from types import SimpleNamespace as sns
import pytest
@pytest.mark.parametrize(argnames='test', argvalues=pytest.yaml_tests)
def test_(test: sns) -> None:
assert test.expected == test.output
```
- `assertions.py` - contains logic for verifying the output from a function
```python
from types import SimpleNamespace as sns
from typing import Any
equals(expected: Any, output: Any) -> sns:
passed = expected == output
return sns(**locals())
```
- `add.py` - contains the function to test
```python
def main(a: int, b: int) -> int:
return a + b
```
- `add_test.yaml` - contains tests for the function
```yaml
configurations:
resources:
- assertions.py
tests:
- function: main
tests:
- arguments:
a: 1
b: 1
assertions:
- method: assertions.equals
expected: 2
- arguments:
a: 1
b: 2
assertions:
- method: assertions.equals
expected: 3
- arguments:
a: 1
b: '1'
assertions:
- method: assertions.equals
field: 2
```
Execute the following command in your command line to run the tests.
```bash
pytest --project-directory=add.py
```
## Configuration
The app can be configured within the pytest settings of a configuration file,
such as a `pytest.ini`, or in the console when invoking pytest. The
configurations are
| Field | Type | Description | Default |
| - | - | - | - |
| project-directory | str | Location of a directory containing files or an an individual module or YAML file to test. | . |
| exclude-files | str or list| A list of patterns. Exclude files from testing that match a specified pattern . | [] |
| resources | str or list | The locations of modules to use as resources during tests | [] |
| resources_folder_name | str | Name of folders containing resources to use for tests| _resources |
| yaml-suffix | str | Suffix in the names of YAML files containing tests | _test |
#### Configure pytest.ini
```ini
[pytest]
project-directory = .
exclude_files =
matching
patterns
to
exclude
resources =
resource_location_a
resource_location_b
resources_folder_name = _resources
yaml_suffix = _test
```
#### Configure command line command
```console
pytest \
--project-directory=.app.py \
--exclude_files matching patterns to exclude \
--resources resource_location_a resource_location_b \
--resource-folder-name _resources \
--yaml-suffix _test
```
## YAML Test Files
Tests are defined in YAML files with the top level keys picked up by the app
being:
- `configurations` - Configurations to be used locally for each test in the YAML files
- `tests` - Configurations used for multiple of individual tests.
### Expanding and Collating Tests
Using the app we can define configurations for tests at various levels
(configurations, tests, nested tests), expand those configurations to lower
configurations, and collate individual tests. This allows us to reuse
configurations and reduce the duplication of content across a YAML file. This is
similar to [anchors](https://yaml.org/spec/1.2.2/#anchors-and-aliases) in YAML,
which we can take advantage, along with the other features available in YAML.
#### Example
This is an abstract example of the expanding/collating configurations done by
the app, where the configurations for tests are comprised of:
- `config_a` - a list
- `config_b` - an object
- `config_c` - a string
- `config_d` - null
In this example, we set these configurations at various levels, globally, tests,
and nested tests; and the expanded/collated results are three individual tests
containing various values for each configuration.
```yaml
# Defined
configurations:
config_a:
- A
config_b:
b: B
config_c: C
tests:
- config_a:
- B
- config_b:
c: C
tests:
- config_a:
- C
config_c: C0
- config_d: D
tests:
- config_a:
- B
config_b:
b: B0
```
```yaml
# Expanded
tests:
- config_a: # test 1
- A
- B # Appended item
config_b:
b: B
config_c: C
config_d: null # Standard test config not defined
- config_a: # test 2
- A
- C # Appended item
config_b:
b: B
c: C # Added key/value
config_c: C0 # Replace string
config_d: null
- config_a: # test 3
- A
config_b:
b: B0 # Updated key/value pair
c: C
config_c: C
config_d: D # Standard test config defined
```
### Schema
Details for configurations or fields of an actual test are defined below. These
fields can be defined globally or at different test levels.
| Field | Type | Description | Expand Action |
| - | - | - | - |
| function | str | Name of function to test | replace |
| environment | dict | Environment variables used by functions in a module | update |
| description | str or list | Additional details about the module, function, or test | append |
| resources | str or list | Resources or modules to use during test | append |
| patches | dict or list | Objects in a module to patch for tests | append |
| cast_arguments | dict or list | Convert function arguments to other data types | append |
| cast_output | dict or list | Convert function output to other data types | append |
| assertions | dict or list | Verifies the output of functions | append |
| tests | dict or list | Nested configurations that get expanded into individual tests | append |
## Resources
Resources represent the location of modules to import and use during tests. Resources can be defined globally when configuring the app, or at the module or test levels under the key `resources` in a YAML file.
```yaml
configurations:
resources: # module level definition
- resource_a.py
tests:
- resources: # test level definition
- resource_b.py
```
Resources are defined at various levels are aggregated into a single list for each test. Each resource listed is imported into the module to test, and is accessible from the module using dot notation based on the locations of the resource and module: `[module_name].[resource_name]`.
**Note**: Since resource modules are imported into the module to test, there is
a risk that attributes of the modules to test can be overwritten. To avoid this
it is important to pick unique names for resource folders or structure your
project in a way to avoid naming conflicts.
## Assertions
### Methods
We can define methods to compare expected and actual output from a function being tested. Methods should have the parameters `expected` and `output`, and return a **SimpleNamespace** object containing `expected`, `output`, `passed` (a boolean indicating whether the assertion passed or failed). Methods can also be reused between tests.
#### Example
Here we define a method for verifying that a function's output is of the correct type.
```python
from types import SimpleNamespace as sns
from typing import Any
def check_type(
output: Any,
expected: str,
) -> sns:
passed = expected == type(output).__name__
return sns(**locals())
```
### Schema
Assertions are defined in YAML test files under the key `assertions`, and a
single assertion has the following fields:
| Field | Type | Description | Default |
| - | - | - | - |
| method | str | Function or method used to verify the result of test | pass_through |
| expected | Any | The expected output of the function | null |
| field | str | Sets the output to a dot-delimited route to an attribute or key within the output. | null |
| cast_output | dict or list | Converts output or an attribute or key in the output before processing an assertion method | null |
And single test can have multiple assertions
```yaml
tests:
- assertions:
- method: method_1
expected: expected_1
field: null
cast_output: []
- method: method_2
expected: expected_2
field: null
cast_output: []
```
## Cast arguments and output
We can convert arguments passed to functions and output from functions to other data types. To do this we define cast objects and list them under the keys `cast_arguments` and `cast_output` for tests or `cast_output` for assertions.
### Schema
The following fields make up a cast object:
| Field | Description | Default |
| ----- | ----------- | ------- |
| method | Dot-delimited route to a function or object to cast a value to| null |
| field | Dot-delimited route to a field, attribute, or key of an object. When set the specified field of the object is cast | null |
| unpack | Boolean indicating whether to unpack an object when casting| False |
```yaml
tests:
- cast_arguments:
- method: method_0
field: field_0
unpack: false
- method: method_1
field: field_1
unpack: false
...
assertions:
- cast_output:
- method: method_2
field: field_2
unpack: false
...
```
## Patches
We can patch objects in the module to test before running tests, and since tests are run in individual threads we can different patches for the same object without interference between tests.
### Methods
There are four patch methods:
- `value` - A value to return when the patched object is used.
- `callable` - A value to return when the patched object is called as function.
- `side_effect_list` - A list of values to call based off of the number of
times the object is called. Returns the item at index `n - 1` of the list for
the `nth` call of the object. Reverts to index 0 when number of calls exceeds
the length of the list.
- `side_effect_dict` - A dictionary of key, values for to patch an object
with. When the patched object is called with a key, the key's associated value
is returned
### Schema
Patches are defined at a list of objects in YAML test files under the key
`patches`, and a single patch object has the following fields:
| Field | Type | Description | Default |
| - | - | - | - |
| method | str | One of the four patch methods defined above | null |
| value | Any | The value the patched object should return when called or used | null
| name | str | The dot-delimited route to the object we wish to patch, in the module to test | null |
```yaml
tests:
- patches:
- method: value
value: value
name: name
```
## Environment
For modules containing a global variable `CONFIG`, we can perform tests using different environment variables by the variables as adding key/value pairs under the key `set_environment` in YAML files. The environment variables are accessible from `CONFIG.environment.[name]`, where `[name]` is the name of the variable.
### Example
```yaml
configurations:
set_environment:
NAME_A: a
NAME_C: c
tests:
- set_environment:
NAME_A: A
NAME_B: b
```
## Advanced example
You can find examples with more advanced usage of the app here: https://github.com/fjemi/yaml-testing-framework/tree/main/examples.
<br>
<a
href="https://www.buymeacoffee.com/olufemijemo"
target="_blank"
>
<img
src="https://cdn.buymeacoffee.com/buttons/default-orange.png"
alt="Buy Me A Coffee"
height="41"
width="174"
>
</a>
Raw data
{
"_id": null,
"home_page": "https://github.com/fjemi/yaml-testing-framework",
"name": "yaml-testing-framework",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": null,
"keywords": "pytest, yaml, testing",
"author": "Olufemi Jemilohun",
"author_email": "olufemi.jemilohun@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/0b/ed/cbbc58af53e163a4fcce119ce73711773c69573e1bb6ec0e87f80d8c1b27/yaml_testing_framework-0.0.5.tar.gz",
"platform": null,
"description": "# YAML Testing Framework\n\nA simple, low-code framework for unit testing in Python with tests are defined in YAML files.\n\n## Features\n\n- Standardized data/tests defined in YAML files\n- App functions as a pytest plugin\n- Support functional programming\n- Can be used to test synchronous and asynchronous logic\n- Easy to use API\n\n\n## Requirements\n\nPython 3.7+\n\n\n## Installation\n\n```bash\npip install yaml-testing-framework\n```\n\n\n## Example\n\n\nCreate the files\n- `test_entrypoint.py` - uses pytest to collect and run tests\n```python\nfrom types import SimpleNamespace as sns\n\nimport pytest\n\n\n@pytest.mark.parametrize(argnames='test', argvalues=pytest.yaml_tests)\ndef test_(test: sns) -> None:\n assert test.expected == test.output\n```\n\n- `assertions.py` - contains logic for verifying the output from a function\n```python\nfrom types import SimpleNamespace as sns\nfrom typing import Any\n\n\n equals(expected: Any, output: Any) -> sns:\n passed = expected == output\n return sns(**locals())\n```\n\n- `add.py` - contains the function to test\n```python\n def main(a: int, b: int) -> int:\n return a + b\n```\n\n- `add_test.yaml` - contains tests for the function\n```yaml\nconfigurations:\n resources:\n - assertions.py\n\ntests:\n- function: main\n tests:\n - arguments:\n a: 1\n b: 1\n assertions:\n - method: assertions.equals\n expected: 2\n - arguments:\n a: 1\n b: 2\n assertions:\n - method: assertions.equals\n expected: 3\n - arguments:\n a: 1\n b: '1'\n assertions:\n - method: assertions.equals\n field: 2\n```\n\nExecute the following command in your command line to run the tests.\n```bash\npytest --project-directory=add.py\n```\n\n\n## Configuration\n\nThe app can be configured within the pytest settings of a configuration file,\n such as a `pytest.ini`, or in the console when invoking pytest. The\n configurations are\n\n| Field | Type | Description | Default |\n| - | - | - | - |\n| project-directory | str | Location of a directory containing files or an an individual module or YAML file to test. | . |\n| exclude-files | str or list| A list of patterns. Exclude files from testing that match a specified pattern . | [] |\n| resources | str or list | The locations of modules to use as resources during tests | [] |\n| resources_folder_name | str | Name of folders containing resources to use for tests| _resources |\n| yaml-suffix | str | Suffix in the names of YAML files containing tests | _test |\n\n\n#### Configure pytest.ini\n\n\n```ini\n[pytest]\nproject-directory = .\nexclude_files =\n matching\n patterns\n to\n exclude\nresources =\n resource_location_a\n resource_location_b\nresources_folder_name = _resources\nyaml_suffix = _test\n```\n\n#### Configure command line command\n\n```console\npytest \\\n--project-directory=.app.py \\\n--exclude_files matching patterns to exclude \\\n--resources resource_location_a resource_location_b \\\n--resource-folder-name _resources \\\n--yaml-suffix _test\n```\n\n\n## YAML Test Files\n\nTests are defined in YAML files with the top level keys picked up by the app\nbeing:\n- `configurations` - Configurations to be used locally for each test in the YAML files\n- `tests` - Configurations used for multiple of individual tests.\n\n\n### Expanding and Collating Tests\n\nUsing the app we can define configurations for tests at various levels\n(configurations, tests, nested tests), expand those configurations to lower\nconfigurations, and collate individual tests. This allows us to reuse\nconfigurations and reduce the duplication of content across a YAML file. This is\nsimilar to [anchors](https://yaml.org/spec/1.2.2/#anchors-and-aliases) in YAML,\nwhich we can take advantage, along with the other features available in YAML.\n\n#### Example\n\nThis is an abstract example of the expanding/collating configurations done by\nthe app, where the configurations for tests are comprised of:\n- `config_a` - a list\n- `config_b` - an object\n- `config_c` - a string\n- `config_d` - null\n\nIn this example, we set these configurations at various levels, globally, tests,\nand nested tests; and the expanded/collated results are three individual tests\ncontaining various values for each configuration.\n\n```yaml\n# Defined\n\nconfigurations:\n config_a:\n - A\n config_b:\n b: B\n config_c: C\n\n\ntests:\n- config_a:\n - B\n- config_b:\n c: C\n tests:\n - config_a:\n - C\n config_c: C0\n - config_d: D\n tests:\n - config_a:\n - B\n config_b:\n b: B0\n```\n\n```yaml\n# Expanded\n\ntests:\n- config_a: # test 1\n - A\n - B # Appended item\n config_b:\n b: B\n config_c: C\n config_d: null # Standard test config not defined\n- config_a: # test 2\n - A\n - C # Appended item\n config_b:\n b: B\n c: C # Added key/value\n config_c: C0 # Replace string\n config_d: null\n- config_a: # test 3\n - A\n config_b:\n b: B0 # Updated key/value pair\n c: C\n config_c: C\n config_d: D # Standard test config defined\n```\n\n\n### Schema\n\nDetails for configurations or fields of an actual test are defined below. These\nfields can be defined globally or at different test levels.\n\n| Field | Type | Description | Expand Action |\n| - | - | - | - |\n| function | str | Name of function to test | replace |\n| environment | dict | Environment variables used by functions in a module | update |\n| description | str or list | Additional details about the module, function, or test | append |\n| resources | str or list | Resources or modules to use during test | append |\n| patches | dict or list | Objects in a module to patch for tests | append |\n| cast_arguments | dict or list | Convert function arguments to other data types | append |\n| cast_output | dict or list | Convert function output to other data types | append |\n| assertions | dict or list | Verifies the output of functions | append |\n| tests | dict or list | Nested configurations that get expanded into individual tests | append |\n\n\n## Resources\n\nResources represent the location of modules to import and use during tests. Resources can be defined globally when configuring the app, or at the module or test levels under the key `resources` in a YAML file.\n\n```yaml\nconfigurations:\n resources: # module level definition\n - resource_a.py\n\ntests:\n- resources: # test level definition\n - resource_b.py\n```\n\nResources are defined at various levels are aggregated into a single list for each test. Each resource listed is imported into the module to test, and is accessible from the module using dot notation based on the locations of the resource and module: `[module_name].[resource_name]`.\n\n**Note**: Since resource modules are imported into the module to test, there is\na risk that attributes of the modules to test can be overwritten. To avoid this\nit is important to pick unique names for resource folders or structure your\nproject in a way to avoid naming conflicts.\n\n\n\n## Assertions\n\n### Methods\n\nWe can define methods to compare expected and actual output from a function being tested. Methods should have the parameters `expected` and `output`, and return a **SimpleNamespace** object containing `expected`, `output`, `passed` (a boolean indicating whether the assertion passed or failed). Methods can also be reused between tests.\n\n#### Example\n\nHere we define a method for verifying that a function's output is of the correct type.\n\n```python\nfrom types import SimpleNamespace as sns\nfrom typing import Any\n\n\ndef check_type(\n output: Any,\n expected: str,\n) -> sns:\n passed = expected == type(output).__name__\n return sns(**locals())\n```\n\n### Schema\n\nAssertions are defined in YAML test files under the key `assertions`, and a\nsingle assertion has the following fields:\n\n| Field | Type | Description | Default |\n| - | - | - | - |\n| method | str | Function or method used to verify the result of test | pass_through |\n| expected | Any | The expected output of the function | null |\n| field | str | Sets the output to a dot-delimited route to an attribute or key within the output. | null |\n| cast_output | dict or list | Converts output or an attribute or key in the output before processing an assertion method | null |\n\n\nAnd single test can have multiple assertions\n\n```yaml\ntests:\n- assertions:\n - method: method_1\n expected: expected_1\n field: null\n cast_output: []\n - method: method_2\n expected: expected_2\n field: null\n cast_output: []\n```\n\n## Cast arguments and output\n\nWe can convert arguments passed to functions and output from functions to other data types. To do this we define cast objects and list them under the keys `cast_arguments` and `cast_output` for tests or `cast_output` for assertions.\n\n### Schema\n\nThe following fields make up a cast object:\n\n| Field | Description | Default |\n| ----- | ----------- | ------- |\n| method | Dot-delimited route to a function or object to cast a value to| null |\n| field | Dot-delimited route to a field, attribute, or key of an object. When set the specified field of the object is cast | null |\n| unpack | Boolean indicating whether to unpack an object when casting| False |\n\n```yaml\ntests:\n- cast_arguments:\n - method: method_0\n field: field_0\n unpack: false\n - method: method_1\n field: field_1\n unpack: false\n ...\n assertions:\n - cast_output:\n - method: method_2\n field: field_2\n unpack: false\n ...\n```\n\n## Patches\n\nWe can patch objects in the module to test before running tests, and since tests are run in individual threads we can different patches for the same object without interference between tests.\n\n### Methods\n\nThere are four patch methods:\n\n- `value` - A value to return when the patched object is used.\n- `callable` - A value to return when the patched object is called as function.\n- `side_effect_list` - A list of values to call based off of the number of\ntimes the object is called. Returns the item at index `n - 1` of the list for\nthe `nth` call of the object. Reverts to index 0 when number of calls exceeds\nthe length of the list.\n- `side_effect_dict` - A dictionary of key, values for to patch an object\nwith. When the patched object is called with a key, the key's associated value\nis returned\n\n### Schema\n\nPatches are defined at a list of objects in YAML test files under the key\n`patches`, and a single patch object has the following fields:\n\n| Field | Type | Description | Default |\n| - | - | - | - |\n| method | str | One of the four patch methods defined above | null |\n| value | Any | The value the patched object should return when called or used | null\n| name | str | The dot-delimited route to the object we wish to patch, in the module to test | null |\n\n\n```yaml\ntests:\n- patches:\n - method: value\n value: value\n name: name\n```\n\n\n## Environment\n\nFor modules containing a global variable `CONFIG`, we can perform tests using different environment variables by the variables as adding key/value pairs under the key `set_environment` in YAML files. The environment variables are accessible from `CONFIG.environment.[name]`, where `[name]` is the name of the variable.\n\n### Example\n\n```yaml\nconfigurations:\n set_environment:\n NAME_A: a\n NAME_C: c\n \n\ntests:\n- set_environment:\n NAME_A: A\n NAME_B: b\n```\n\n\n## Advanced example\n\nYou can find examples with more advanced usage of the app here: https://github.com/fjemi/yaml-testing-framework/tree/main/examples.\n\n\n<br>\n<a\n href=\"https://www.buymeacoffee.com/olufemijemo\"\n target=\"_blank\"\n>\n <img\n src=\"https://cdn.buymeacoffee.com/buttons/default-orange.png\"\n alt=\"Buy Me A Coffee\"\n height=\"41\"\n width=\"174\"\n >\n</a>\n",
"bugtrack_url": null,
"license": "MIT license",
"summary": "A testing framework where tests are defined in YAML files",
"version": "0.0.5",
"project_urls": {
"Homepage": "https://github.com/fjemi/yaml-testing-framework"
},
"split_keywords": [
"pytest",
" yaml",
" testing"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "85ba8f954f2c2854b65733cc3d27924022f59c9c3a55d6139dcf995377f63904",
"md5": "3a27864d49de8389ab412af5ad792079",
"sha256": "b3989e7160c673d17f876b046e4976c4d60fbf7b60ffaa226a4f6884e9dab3b1"
},
"downloads": -1,
"filename": "yaml_testing_framework-0.0.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3a27864d49de8389ab412af5ad792079",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 32944,
"upload_time": "2024-04-26T21:45:18",
"upload_time_iso_8601": "2024-04-26T21:45:18.236060Z",
"url": "https://files.pythonhosted.org/packages/85/ba/8f954f2c2854b65733cc3d27924022f59c9c3a55d6139dcf995377f63904/yaml_testing_framework-0.0.5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "0bedcbbc58af53e163a4fcce119ce73711773c69573e1bb6ec0e87f80d8c1b27",
"md5": "7960bb0bd35744d9d7bfff48f410f72c",
"sha256": "ff97c534bacaa7018ae8c71e0e5076bf6eab3e1f83f653a59ddf85d1421f65d6"
},
"downloads": -1,
"filename": "yaml_testing_framework-0.0.5.tar.gz",
"has_sig": false,
"md5_digest": "7960bb0bd35744d9d7bfff48f410f72c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 14542,
"upload_time": "2024-04-26T21:45:19",
"upload_time_iso_8601": "2024-04-26T21:45:19.505008Z",
"url": "https://files.pythonhosted.org/packages/0b/ed/cbbc58af53e163a4fcce119ce73711773c69573e1bb6ec0e87f80d8c1b27/yaml_testing_framework-0.0.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-26 21:45:19",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "fjemi",
"github_project": "yaml-testing-framework",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "yaml-testing-framework"
}
JSON
