pytest-regtest
==============
pytest-regtest is a *pytest*-plugin for implementing regression tests.
Compared to functional testing a regression test does not test if
software produces correct results, instead a regression test checks if
software behaves the same way as it did before introduced changes.
More about regression testing at
<https://en.wikipedia.org/wiki/Regression_testing>. Regression testing
is a common technique to get started when refactoring legacy code
lacking a test suite.
*pytest-regtest* allows capturing selected output which then can be
compared to the captured output from former runs.
To install and activate this plugin execute:
$ pip install pytest-regtest
*pytest-regtest* plugin provides a fixture named *regtest* which can be
used as a file handle for recording data:
from __future__ import print_function
def test_squares_up_to_ten(regtest):
result = [i*i for i in range(10)]
# one way to record output:
print(result, file=regtest)
# alternative method to record output:
regtest.write("done")
# or using a context manager:
with regtest:
print("this will be recorded")
If you run this test script with *pytest* the first time there is no
recorded output for this test function so far and thus the test will
fail with a message including a diff:
$ py.test
...
regression test output differences for test_demo.py::test_squares_up_to_ten:
> --- current
> +++ tobe
> @@ -1,2 +1 @@
> -[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
> -done
> +
The output tells us what the current output is, and that the "tobe" output
is still empty.
For accepting this output, we run *pytest* with the *--reset-regtest*
flag:
$ py.test --regtest-reset
Now the next execution of *py.test* will succeed:
$ py.test
Now we break the test by modifying the code under test to compute the first
eleven square numbers:
from __future__ import print_function
def test_squares_up_to_ten(regtest):
result = [i*i for i in range(11)] # changed !
# one way to record output:
print(result, file=regtest)
# alternative method to record output:
regtest.write("done")
The next run of pytest delivers a nice diff of the current and expected output
from this test function:
$ py.test
...
> --- current
> +++ tobe
> @@ -1,2 +1,2 @@
> -[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
> +[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
> done
The recorded output was written to files in the subfolder
`_regtest_outputs` next to your test script(s). You might keep this
folder under version control.
Other features
--------------
Another way to record output is to capture all output to `sys.stdout`:
def test_squares_up_to_ten(regtest):
result = [i*i for i in range(10)]
with regtest():
print result
You can reset recorded output of files and functions individually as:
$ py.test --regtest-reset tests/test_00.py
$ py.test --regtest-reset tests/test_00.py::test_squares_up_to_ten
To supress the diff and only see the stats use:
$ py.test --regtest-nodiff
To see recorded output during test execution run:
$ py.test --regtest-tee -s
If you develop on mixed platforms it might be usefull to ignore white
spaces at the end of the lines when comparing output. This can be
achieved by specifying:
$ py.test --regtest-ignore-line-endings
Fixing unavoidable changes in recorded output
----------------------------------------------
The recorded output can contain data which is changing from test run to test
run, e.g. pathes created with the `tmpdir` fixture or hexadecimal object ids,
when objects are printed.
The plugin already replaces such changing data in the recorded output,
and one can register own converters in `conftest.py` in the tests
folder. For example:
import pytest_regtest
@pytest_regtest.register_converter_pre
def fix_before(txt):
"""modify recorded output before the default fixes
like temp folders or hex object ids are applied"""
# remove lines with passwords:
lines = txt.split('\n')
lines = [l for l in lines if "password is" not in l]
return '\n'.join(lines)
@pytest_regtest.register_converter_post
def after(txt):
"""modify recorded output after the default fixes
like temp folders or hex object ids are applied"""
# for demo only
return txt.upper()
This can be used to fix substrings like "computation need 1.23 seconds"
to "computation needed <TIME> seconds" etc.
One can register multiple such converters which will be applied in
order of registration.
Raw data
{
"_id": null,
"home_page": "https://gitlab.com/uweschmitt/pytest-regtest",
"name": "pytest-regtest",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "",
"author": "Uwe Schmitt",
"author_email": "uwe.schmitt@id.ethz.ch",
"download_url": "https://files.pythonhosted.org/packages/8d/83/750161e38ce7e67cd75bd62c6b89bf35dce04532caa2116500fd3534feac/pytest-regtest-1.4.5.tar.gz",
"platform": "",
"description": "pytest-regtest\n==============\n\npytest-regtest is a *pytest*-plugin for implementing regression tests.\nCompared to functional testing a regression test does not test if\nsoftware produces correct results, instead a regression test checks if\nsoftware behaves the same way as it did before introduced changes.\n\nMore about regression testing at\n<https://en.wikipedia.org/wiki/Regression_testing>. Regression testing\nis a common technique to get started when refactoring legacy code\nlacking a test suite.\n\n*pytest-regtest* allows capturing selected output which then can be\ncompared to the captured output from former runs.\n\nTo install and activate this plugin execute:\n\n $ pip install pytest-regtest\n\n*pytest-regtest* plugin provides a fixture named *regtest* which can be\nused as a file handle for recording data:\n\n from __future__ import print_function\n\n def test_squares_up_to_ten(regtest):\n\n result = [i*i for i in range(10)]\n\n # one way to record output:\n print(result, file=regtest)\n\n # alternative method to record output:\n regtest.write(\"done\")\n\n # or using a context manager:\n with regtest:\n print(\"this will be recorded\")\n\nIf you run this test script with *pytest* the first time there is no\nrecorded output for this test function so far and thus the test will\nfail with a message including a diff:\n\n $ py.test\n ...\n\n regression test output differences for test_demo.py::test_squares_up_to_ten:\n\n > --- current\n > +++ tobe\n > @@ -1,2 +1 @@\n > -[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]\n > -done\n > +\n\nThe output tells us what the current output is, and that the \"tobe\" output\nis still empty.\n\nFor accepting this output, we run *pytest* with the *--reset-regtest*\nflag:\n\n $ py.test --regtest-reset\n\nNow the next execution of *py.test* will succeed:\n\n $ py.test\n\nNow we break the test by modifying the code under test to compute the first\neleven square numbers:\n\n from __future__ import print_function\n\n def test_squares_up_to_ten(regtest):\n\n result = [i*i for i in range(11)] # changed !\n\n # one way to record output:\n print(result, file=regtest)\n\n # alternative method to record output:\n regtest.write(\"done\")\n\nThe next run of pytest delivers a nice diff of the current and expected output\nfrom this test function:\n\n $ py.test\n\n ...\n > --- current\n > +++ tobe\n > @@ -1,2 +1,2 @@\n > -[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100]\n > +[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]\n > done\n\n\nThe recorded output was written to files in the subfolder\n`_regtest_outputs` next to your test script(s). You might keep this\nfolder under version control.\n\n\nOther features\n--------------\n\nAnother way to record output is to capture all output to `sys.stdout`:\n\n def test_squares_up_to_ten(regtest):\n\n result = [i*i for i in range(10)]\n\n with regtest():\n print result\n\nYou can reset recorded output of files and functions individually as:\n\n $ py.test --regtest-reset tests/test_00.py\n $ py.test --regtest-reset tests/test_00.py::test_squares_up_to_ten\n\nTo supress the diff and only see the stats use:\n\n $ py.test --regtest-nodiff\n\nTo see recorded output during test execution run:\n\n $ py.test --regtest-tee -s\n\nIf you develop on mixed platforms it might be usefull to ignore white\nspaces at the end of the lines when comparing output. This can be\nachieved by specifying:\n\n $ py.test --regtest-ignore-line-endings\n\n\nFixing unavoidable changes in recorded output\n----------------------------------------------\n\nThe recorded output can contain data which is changing from test run to test\nrun, e.g. pathes created with the `tmpdir` fixture or hexadecimal object ids,\nwhen objects are printed.\n\nThe plugin already replaces such changing data in the recorded output,\nand one can register own converters in `conftest.py` in the tests\nfolder. For example:\n\n import pytest_regtest\n\n @pytest_regtest.register_converter_pre\n def fix_before(txt):\n \"\"\"modify recorded output before the default fixes\n like temp folders or hex object ids are applied\"\"\"\n\n # remove lines with passwords:\n lines = txt.split('\\n')\n lines = [l for l in lines if \"password is\" not in l]\n return '\\n'.join(lines)\n\n @pytest_regtest.register_converter_post\n def after(txt):\n \"\"\"modify recorded output after the default fixes\n like temp folders or hex object ids are applied\"\"\"\n\n # for demo only\n return txt.upper()\n\nThis can be used to fix substrings like \"computation need 1.23 seconds\"\nto \"computation needed <TIME> seconds\" etc.\n\nOne can register multiple such converters which will be applied in\norder of registration.",
"bugtrack_url": null,
"license": "https://opensource.org/licenses/MIT",
"summary": "pytest plugin for regression tests",
"version": "1.4.5",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"md5": "524cedd133b465ebf727dcb12f5b85bf",
"sha256": "a1e4e87d483e493d9bf6d15a71a46dedaccb6caab79fa8066f47a8eb3e6f86e3"
},
"downloads": -1,
"filename": "pytest-regtest-1.4.5.tar.gz",
"has_sig": false,
"md5_digest": "524cedd133b465ebf727dcb12f5b85bf",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 9480,
"upload_time": "2020-09-16T17:57:07",
"upload_time_iso_8601": "2020-09-16T17:57:07.722467Z",
"url": "https://files.pythonhosted.org/packages/8d/83/750161e38ce7e67cd75bd62c6b89bf35dce04532caa2116500fd3534feac/pytest-regtest-1.4.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2020-09-16 17:57:07",
"github": false,
"gitlab": true,
"bitbucket": false,
"gitlab_user": null,
"gitlab_project": "uweschmitt",
"lcname": "pytest-regtest"
}
JSON