# pytest_cache_assert
Cache assertion data to simplify regression testing of complex serializable data
## Installation
`poetry add pytest_assert_check --dev`
## Usage
The primary use case of this package is regression testing of large, serializable dictionaries, such as from an API under development.
You may have parameterized test cases where you need to assert that the created dictionary stays the same, but you don’t want to manually generate the expected fields and values to compare. Instead you can capture a snapshot of the serialized data and cache the result then use the cached data to check for consistency in repeated test runs. The cached files should be checked into version control, which can be very useful as documentation
This package can minimize test case logic, while improving regression testing thoroughness
This project was heavily inspired by the excellent [pytest-recording](https://github.com/kiwicom/pytest-recording)
### Alternatives
- [pytest-recording](https://github.com/kiwicom/pytest-recording): this is the package I use and highly recommend for recording and replaying **external** API communication so that API requests only need to be made once for unit testing (i.e. recording API responses from Github's API called from a test suite)
- [pytest-snapshot](https://pypi.org/project/pytest-snapshot/): I only found this package after already releasing a 1.0.0 version of `pytest_assert_cache`. This package can be more configurable with a user-specified serializer and might be a good alternative. See their documentation for more info
- [snapshottest](https://github.com/syrusakbary/snapshottest): This was another find after releasing a 1.0.0 version and would probably be **a good alterantive for most users**
- `pytest-snapshot` is much more configurable, has many more users, and is a better name
- I really like the ability to quickly regenerate the cached files with [--snapshot-update](https://github.com/syrusakbary/snapshottest/blob/master/snapshottest/pytest.py)
- [There is some interesting discussion on how best to handle fields that change between tests](https://github.com/syrusakbary/snapshottest/issues/21)
- [syrupy](https://github.com/tophat/syrupy): a well designed alternative that is extensible and provides classes like `JSONSnapshotExtension` to store the cached data in VCS
- [dirty-equals](https://github.com/samuelcolvin/dirty-equals): broadly check values (i.e. `assert result == {'counter': IsPositiveInt, ...}`, etc.) rather than accessing and checking each field individual, which makes test easier to write and output errors easier to review
- [pytest-pinned](https://github.com/freol35241/pytest-pinned): write assertions like `assert result == pinned` where the results are stored in JSON
- [touca](https://github.com/trytouca/trytouca): more general implementation of snapshot-style testing using a cloud service
### Basic Example
You've created a new project called `package_a` with one file `package_a/source_file.py` and test `tests/test_file.py`
```py
"""package_a/source_file.py"""
import sys
from datetime import datetime
from typing import Any, Dict, List, Optional
from beartype import beartype
from pydantic import BaseModel
class User(BaseModel): # noqa: H601
"""Example from pydantic documentation."""
id: int # noqa: A003,VNE003
name: str = 'John Doe'
signup_ts: Optional[datetime] = None
friends: List[int] = []
@beartype
def create_data(name: str) -> Dict:
"""Arbitrary function that returns a dictionary.
This demonstration uses pydantic, but any dictionary can be tested!
"""
return User(id=sys.maxsize, name=name).dict()
```
```py
"""tests/test_file.py"""
import pytest
from package_a.source_file import create_data
@pytest.mark.parametrize('name', ['Test Name 1', 'Test Name 2'])
def test_create_data(name, assert_against_cache):
"""Basic test of create_data()."""
result = create_data(name=name)
# One could manually create the expected dictionary
cache = {'id': 9223372036854775807, 'signup_ts': None, 'friends': [], 'name': name}
assert result == cache
# ----------------------------------------------------------------------------------
# Or utilize the pytest_cache_assert fixture to compare against the last cached version
assert_against_cache(result)
```
`pytest_cache_assert` will automatically create: `tests/cache-assert/source_file/test_file/test_create_data-[Test Name 1].json` (and `test_create_data[Test Name 2].json`) for each of the parameters when first run by caching the `result`. Below is the example for `test_create_data-[Test Name 1].json`
```json
{
"_info": [
{
"func_args": {
"name": "Test Name 1"
},
"test_file": "test_readme.py",
"test_name": "test_create_data"
}
],
"_json": {
"friends": [],
"id": 9223372036854775807,
"name": "Test Name 1",
"signup_ts": null
}
}
```
The cached JSON files must be checked into version control and if needed, can be manually edited or deleted so that they will be regenerated when the test suite is next run
### More Examples
In your cached dictionary, you may have variable values with more complex logic to verify, such as dates, UUIDs, etc. These can be selectively ignored, matched-if-null, or some other user-specified check:
```py
"""tests/test_readme_more.py."""
from contextlib import suppress
from datetime import datetime, timedelta
from uuid import uuid4
import pytest
from pytest_cache_assert import AssertRule, Wild, check_suppress, check_type
def test_assert_against_cache_key_rules(assert_against_cache):
"""Demonstrate use of `assert_rules`."""
now = datetime.now() # noqa: DTZ005
cached_data = {
'date': str(now),
'nested': {'uuid': str(uuid4())},
'ignored': {'a': 1, 'b': 2},
}
test_data = {
'date': str(now + timedelta(hours=3)),
'nested': {'uuid': str(uuid4())},
'ignored': {'recursively': {'a': {'b': {'c': 1}}}},
}
with suppress(AssertionError):
# Ensures that the cache file has been created
assert_against_cache(cached_data)
assert_rules = [
# To ignore values for 'ignored.a' and 'ignored.b', create a rule
# Here, we use the wildcard for dictionary keys
AssertRule.build_re(pattern=['ignored', Wild.recur()], func=check_suppress),
# Instead of suppressing, the type of data could be resolved and compared
# This is useful for datetime or UUID values where we expect variability
AssertRule(pattern='date', func=check_type),
AssertRule.build_re(pattern=['nested', 'uuid'], func=check_type),
# Any "func" with arguments 'old' and 'new' can be used as a rule
]
# Without assert rules, an AssertionError is raised
with pytest.raises(AssertionError):
assert_against_cache(test_data)
# But, with the custom logic, the cache assertion check will succeed
assert_against_cache(test_data, assert_rules=assert_rules)
```
### Even More Examples
For more example code, see the [scripts] directory or the [tests].
## Customization (`beta`)
> Note: this feature is to be considered `beta` and may change, however, I will do my best to keep the same interfaces
For 2.0.0, `pytest_cache_assert` was refactored to be more easily customizable with configuration options for not just the cache directory, but also for a way to override how files are named and to override how the cached test data is serialized and validated.
With these configuration options, users or 3rd party packages can replace the default package behavior, such as changing the file format for data serialization (`yaml`, `jsonlines`, etc.) and/or specifying a different serialization logic. All configuration options are available by creating a `cache_assert_config` fixture with the provided implementations.
- See `AssertConfig` in `plugin.py` for configuration options and more information
- `always_write`: Always write to the cached file so that diffs can be examined in the user's VCS.
- `cache_dir_rel_path`: String relative directory from `tests/`. Default resolves to `tests/assert-cache/`.
- `cache_store`: Configurable class for managing the cache representation. Default is local JSON.
- `converters`: register functions that handle conversion of unhandled types, such as pandas DataFrames
- `validator`: Custom validator for identifying and summarizing the deviations from the cache.
```py
import pytest
from pytest_cache_assert.plugin import AssertConfig
@pytest.fixture(scope='module')
def cache_assert_config():
return AssertConfig(cache_dir_rel_path='custom/cache/dir')
```
## Project Status
See the `Open Issues` and/or the [CODE_TAG_SUMMARY]. For release history, see the [CHANGELOG].
### Planned Global Configuration Options
These are ideas for future options that are not currently implemented, but could be if there is enough interest:
- PLANNED: [Provide CLI arguments like `pytest-recording`](https://github.com/kiwicom/pytest-recording/blob/484bb887dd43fcaf44149160d57b58a7215e2c8a/src/pytest_recording/plugin.py#L37-L70) (`request.config.getoption("--record-mode") or "none"`) for one-time changes to configuration
- PLANNED: Consider filters to prevent secrets from being cached: `filter_headers=[['authorization', 'id'], ['authorization', 'cookies']]` (Although, you should be using a pre-commit hook and formatting the dict before passing to to the cache)
- TODO: [Add tips from Jest on best practices](https://jestjs.io/docs/snapshot-testing#best-practices) -- treat snapshots as code, etc.
## Contributing
We welcome pull requests! For your pull request to be accepted smoothly, we suggest that you first open a GitHub issue to discuss your idea. For resources on getting started with the code base, see the below documentation:
- [DEVELOPER_GUIDE]
- [STYLE_GUIDE]
## Code of Conduct
We follow the [Contributor Covenant Code of Conduct][contributor-covenant].
### Open Source Status
We try to reasonably meet most aspects of the "OpenSSF scorecard" from [Open Source Insights](https://deps.dev/pypi/pytest_cache_assert)
## Responsible Disclosure
If you have any security issue to report, please contact the project maintainers privately. You can reach us at [dev.act.kyle@gmail.com](mailto:dev.act.kyle@gmail.com).
## License
[LICENSE]
[changelog]: https://pytest_cache_assert.kyleking.me/docs/CHANGELOG
[code_tag_summary]: https://pytest_cache_assert.kyleking.me/docs/CODE_TAG_SUMMARY
[contributor-covenant]: https://www.contributor-covenant.org
[developer_guide]: https://pytest_cache_assert.kyleking.me/docs/DEVELOPER_GUIDE
[license]: https://github.com/kyleking/pytest_cache_assert/blob/main/LICENSE
[scripts]: https://github.com/kyleking/pytest_cache_assert/blob/main/scripts
[style_guide]: https://pytest_cache_assert.kyleking.me/docs/STYLE_GUIDE
[tests]: https://github.com/kyleking/pytest_cache_assert/blob/main/tests
Raw data
{
"_id": null,
"home_page": "https://github.com/kyleking/pytest_cache_assert",
"name": "pytest-cache-assert",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8.12,<4.0.0",
"maintainer_email": "",
"keywords": "",
"author": "Kyle King",
"author_email": "dev.act.kyle@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/9a/28/5c068e253637f58de064d270e32487a9dab258d4e17d55539bbe16237c73/pytest_cache_assert-4.0.0.tar.gz",
"platform": null,
"description": "# pytest_cache_assert\n\nCache assertion data to simplify regression testing of complex serializable data\n\n## Installation\n\n`poetry add pytest_assert_check --dev`\n\n## Usage\n\nThe primary use case of this package is regression testing of large, serializable dictionaries, such as from an API under development.\n\nYou may have parameterized test cases where you need to assert that the created dictionary stays the same, but you don\u2019t want to manually generate the expected fields and values to compare. Instead you can capture a snapshot of the serialized data and cache the result then use the cached data to check for consistency in repeated test runs. The cached files should be checked into version control, which can be very useful as documentation\n\nThis package can minimize test case logic, while improving regression testing thoroughness\n\nThis project was heavily inspired by the excellent [pytest-recording](https://github.com/kiwicom/pytest-recording)\n\n### Alternatives\n\n- [pytest-recording](https://github.com/kiwicom/pytest-recording): this is the package I use and highly recommend for recording and replaying **external** API communication so that API requests only need to be made once for unit testing (i.e. recording API responses from Github's API called from a test suite)\n- [pytest-snapshot](https://pypi.org/project/pytest-snapshot/): I only found this package after already releasing a 1.0.0 version of `pytest_assert_cache`. This package can be more configurable with a user-specified serializer and might be a good alternative. See their documentation for more info\n- [snapshottest](https://github.com/syrusakbary/snapshottest): This was another find after releasing a 1.0.0 version and would probably be **a good alterantive for most users**\n - `pytest-snapshot` is much more configurable, has many more users, and is a better name\n - I really like the ability to quickly regenerate the cached files with [--snapshot-update](https://github.com/syrusakbary/snapshottest/blob/master/snapshottest/pytest.py)\n - [There is some interesting discussion on how best to handle fields that change between tests](https://github.com/syrusakbary/snapshottest/issues/21)\n - [syrupy](https://github.com/tophat/syrupy): a well designed alternative that is extensible and provides classes like `JSONSnapshotExtension` to store the cached data in VCS\n- [dirty-equals](https://github.com/samuelcolvin/dirty-equals): broadly check values (i.e. `assert result == {'counter': IsPositiveInt, ...}`, etc.) rather than accessing and checking each field individual, which makes test easier to write and output errors easier to review\n- [pytest-pinned](https://github.com/freol35241/pytest-pinned): write assertions like `assert result == pinned` where the results are stored in JSON\n- [touca](https://github.com/trytouca/trytouca): more general implementation of snapshot-style testing using a cloud service\n\n### Basic Example\n\nYou've created a new project called `package_a` with one file `package_a/source_file.py` and test `tests/test_file.py`\n\n```py\n\"\"\"package_a/source_file.py\"\"\"\n\nimport sys\nfrom datetime import datetime\nfrom typing import Any, Dict, List, Optional\n\nfrom beartype import beartype\nfrom pydantic import BaseModel\n\n\nclass User(BaseModel): # noqa: H601\n \"\"\"Example from pydantic documentation.\"\"\"\n\n id: int # noqa: A003,VNE003\n name: str = 'John Doe'\n signup_ts: Optional[datetime] = None\n friends: List[int] = []\n\n\n@beartype\ndef create_data(name: str) -> Dict:\n \"\"\"Arbitrary function that returns a dictionary.\n\n This demonstration uses pydantic, but any dictionary can be tested!\n\n \"\"\"\n return User(id=sys.maxsize, name=name).dict()\n```\n\n```py\n\"\"\"tests/test_file.py\"\"\"\n\nimport pytest\n\nfrom package_a.source_file import create_data\n\n\n@pytest.mark.parametrize('name', ['Test Name 1', 'Test Name 2'])\ndef test_create_data(name, assert_against_cache):\n \"\"\"Basic test of create_data().\"\"\"\n result = create_data(name=name)\n\n # One could manually create the expected dictionary\n cache = {'id': 9223372036854775807, 'signup_ts': None, 'friends': [], 'name': name}\n assert result == cache\n # ----------------------------------------------------------------------------------\n # Or utilize the pytest_cache_assert fixture to compare against the last cached version\n assert_against_cache(result)\n```\n\n`pytest_cache_assert` will automatically create: `tests/cache-assert/source_file/test_file/test_create_data-[Test Name 1].json` (and `test_create_data[Test Name 2].json`) for each of the parameters when first run by caching the `result`. Below is the example for `test_create_data-[Test Name 1].json`\n\n```json\n{\n \"_info\": [\n {\n \"func_args\": {\n \"name\": \"Test Name 1\"\n },\n \"test_file\": \"test_readme.py\",\n \"test_name\": \"test_create_data\"\n }\n ],\n \"_json\": {\n \"friends\": [],\n \"id\": 9223372036854775807,\n \"name\": \"Test Name 1\",\n \"signup_ts\": null\n }\n}\n```\n\nThe cached JSON files must be checked into version control and if needed, can be manually edited or deleted so that they will be regenerated when the test suite is next run\n\n### More Examples\n\nIn your cached dictionary, you may have variable values with more complex logic to verify, such as dates, UUIDs, etc. These can be selectively ignored, matched-if-null, or some other user-specified check:\n\n```py\n\"\"\"tests/test_readme_more.py.\"\"\"\n\nfrom contextlib import suppress\nfrom datetime import datetime, timedelta\nfrom uuid import uuid4\n\nimport pytest\n\nfrom pytest_cache_assert import AssertRule, Wild, check_suppress, check_type\n\n\ndef test_assert_against_cache_key_rules(assert_against_cache):\n \"\"\"Demonstrate use of `assert_rules`.\"\"\"\n now = datetime.now() # noqa: DTZ005\n cached_data = {\n 'date': str(now),\n 'nested': {'uuid': str(uuid4())},\n 'ignored': {'a': 1, 'b': 2},\n }\n test_data = {\n 'date': str(now + timedelta(hours=3)),\n 'nested': {'uuid': str(uuid4())},\n 'ignored': {'recursively': {'a': {'b': {'c': 1}}}},\n }\n with suppress(AssertionError):\n # Ensures that the cache file has been created\n assert_against_cache(cached_data)\n\n assert_rules = [\n # To ignore values for 'ignored.a' and 'ignored.b', create a rule\n # Here, we use the wildcard for dictionary keys\n AssertRule.build_re(pattern=['ignored', Wild.recur()], func=check_suppress),\n\n # Instead of suppressing, the type of data could be resolved and compared\n # This is useful for datetime or UUID values where we expect variability\n AssertRule(pattern='date', func=check_type),\n AssertRule.build_re(pattern=['nested', 'uuid'], func=check_type),\n\n # Any \"func\" with arguments 'old' and 'new' can be used as a rule\n ]\n\n # Without assert rules, an AssertionError is raised\n with pytest.raises(AssertionError):\n assert_against_cache(test_data)\n # But, with the custom logic, the cache assertion check will succeed\n assert_against_cache(test_data, assert_rules=assert_rules)\n```\n\n### Even More Examples\n\nFor more example code, see the [scripts] directory or the [tests].\n\n## Customization (`beta`)\n\n> Note: this feature is to be considered `beta` and may change, however, I will do my best to keep the same interfaces\n\nFor 2.0.0, `pytest_cache_assert` was refactored to be more easily customizable with configuration options for not just the cache directory, but also for a way to override how files are named and to override how the cached test data is serialized and validated.\n\nWith these configuration options, users or 3rd party packages can replace the default package behavior, such as changing the file format for data serialization (`yaml`, `jsonlines`, etc.) and/or specifying a different serialization logic. All configuration options are available by creating a `cache_assert_config` fixture with the provided implementations.\n\n- See `AssertConfig` in `plugin.py` for configuration options and more information\n - `always_write`: Always write to the cached file so that diffs can be examined in the user's VCS.\n - `cache_dir_rel_path`: String relative directory from `tests/`. Default resolves to `tests/assert-cache/`.\n - `cache_store`: Configurable class for managing the cache representation. Default is local JSON.\n - `converters`: register functions that handle conversion of unhandled types, such as pandas DataFrames\n - `validator`: Custom validator for identifying and summarizing the deviations from the cache.\n\n```py\nimport pytest\n\nfrom pytest_cache_assert.plugin import AssertConfig\n\n\n@pytest.fixture(scope='module')\ndef cache_assert_config():\n return AssertConfig(cache_dir_rel_path='custom/cache/dir')\n```\n\n## Project Status\n\nSee the `Open Issues` and/or the [CODE_TAG_SUMMARY]. For release history, see the [CHANGELOG].\n\n### Planned Global Configuration Options\n\nThese are ideas for future options that are not currently implemented, but could be if there is enough interest:\n\n- PLANNED: [Provide CLI arguments like `pytest-recording`](https://github.com/kiwicom/pytest-recording/blob/484bb887dd43fcaf44149160d57b58a7215e2c8a/src/pytest_recording/plugin.py#L37-L70) (`request.config.getoption(\"--record-mode\") or \"none\"`) for one-time changes to configuration\n- PLANNED: Consider filters to prevent secrets from being cached: `filter_headers=[['authorization', 'id'], ['authorization', 'cookies']]` (Although, you should be using a pre-commit hook and formatting the dict before passing to to the cache)\n- TODO: [Add tips from Jest on best practices](https://jestjs.io/docs/snapshot-testing#best-practices) -- treat snapshots as code, etc.\n\n## Contributing\n\nWe welcome pull requests! For your pull request to be accepted smoothly, we suggest that you first open a GitHub issue to discuss your idea. For resources on getting started with the code base, see the below documentation:\n\n- [DEVELOPER_GUIDE]\n- [STYLE_GUIDE]\n\n## Code of Conduct\n\nWe follow the [Contributor Covenant Code of Conduct][contributor-covenant].\n\n### Open Source Status\n\nWe try to reasonably meet most aspects of the \"OpenSSF scorecard\" from [Open Source Insights](https://deps.dev/pypi/pytest_cache_assert)\n\n## Responsible Disclosure\n\nIf you have any security issue to report, please contact the project maintainers privately. You can reach us at [dev.act.kyle@gmail.com](mailto:dev.act.kyle@gmail.com).\n\n## License\n\n[LICENSE]\n\n[changelog]: https://pytest_cache_assert.kyleking.me/docs/CHANGELOG\n[code_tag_summary]: https://pytest_cache_assert.kyleking.me/docs/CODE_TAG_SUMMARY\n[contributor-covenant]: https://www.contributor-covenant.org\n[developer_guide]: https://pytest_cache_assert.kyleking.me/docs/DEVELOPER_GUIDE\n[license]: https://github.com/kyleking/pytest_cache_assert/blob/main/LICENSE\n[scripts]: https://github.com/kyleking/pytest_cache_assert/blob/main/scripts\n[style_guide]: https://pytest_cache_assert.kyleking.me/docs/STYLE_GUIDE\n[tests]: https://github.com/kyleking/pytest_cache_assert/blob/main/tests\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Cache assertion data to simplify regression testing of complex serializable data",
"version": "4.0.0",
"project_urls": {
"Bug Tracker": "https://github.com/kyleking/pytest_cache_assert/issues",
"Changelog": "https://github.com/kyleking/pytest_cache_assert/blob/main/docs/docs/CHANGELOG.md",
"Documentation": "https://pytest_cache_assert.kyleking.me",
"Homepage": "https://github.com/kyleking/pytest_cache_assert",
"Repository": "https://github.com/kyleking/pytest_cache_assert"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "85415a47d60a6ebb29dcdeeed9eb109fbc81f1be5fa39768ea04aac957601929",
"md5": "0d8ed4723348e9025645563e9b498cb2",
"sha256": "e7b0f1081989379c1ed257c0603f1055056d4ab0fa574627061efa4774c67bde"
},
"downloads": -1,
"filename": "pytest_cache_assert-4.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0d8ed4723348e9025645563e9b498cb2",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8.12,<4.0.0",
"size": 24250,
"upload_time": "2023-08-14T23:41:49",
"upload_time_iso_8601": "2023-08-14T23:41:49.099453Z",
"url": "https://files.pythonhosted.org/packages/85/41/5a47d60a6ebb29dcdeeed9eb109fbc81f1be5fa39768ea04aac957601929/pytest_cache_assert-4.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9a285c068e253637f58de064d270e32487a9dab258d4e17d55539bbe16237c73",
"md5": "93d19d7cc4c7f4146bccbe9edb511574",
"sha256": "68dde4904154704cbe166a23b108f21787e66807e36d45b600ba609d9ee7fbff"
},
"downloads": -1,
"filename": "pytest_cache_assert-4.0.0.tar.gz",
"has_sig": false,
"md5_digest": "93d19d7cc4c7f4146bccbe9edb511574",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8.12,<4.0.0",
"size": 21440,
"upload_time": "2023-08-14T23:41:50",
"upload_time_iso_8601": "2023-08-14T23:41:50.377752Z",
"url": "https://files.pythonhosted.org/packages/9a/28/5c068e253637f58de064d270e32487a9dab258d4e17d55539bbe16237c73/pytest_cache_assert-4.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-14 23:41:50",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "kyleking",
"github_project": "pytest_cache_assert",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "pytest-cache-assert"
}
JSON
