pytest-regtest


Namepytest-regtest JSON
Version 1.4.5 PyPI version JSON
download
home_pagehttps://gitlab.com/uweschmitt/pytest-regtest
Summarypytest plugin for regression tests
upload_time2020-09-16 17:57:07
maintainer
docs_urlNone
authorUwe Schmitt
requires_python
licensehttps://opensource.org/licenses/MIT
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            pytest-regtest
==============

pytest-regtest is a *pytest*-plugin for implementing regression tests.
Compared to functional testing a regression test does not test if
software produces correct results, instead a regression test checks if
software behaves the same way as it did before introduced changes.

More about regression testing at
<https://en.wikipedia.org/wiki/Regression_testing>. Regression testing
is a common technique to get started when refactoring legacy code
lacking a test suite.

*pytest-regtest* allows capturing selected output which then can be
compared to the captured output from former runs.

To install and activate this plugin execute:

    $ pip install pytest-regtest

*pytest-regtest* plugin provides a fixture named *regtest* which can be
used as a file handle for recording data:

    from __future__ import print_function

    def test_squares_up_to_ten(regtest):

        result = [i*i for i in range(10)]

        # one way to record output:
        print(result, file=regtest)

        # alternative method to record output:
        regtest.write("done")

        # or using a context manager:
        with regtest:
            print("this will be recorded")

If you run this test script with *pytest* the first time there is no
recorded output for this test function so far and thus the test will
fail with a message including a diff:

    $ py.test
    ...

    regression test output differences for test_demo.py::test_squares_up_to_ten:

    >   --- current
    >   +++ tobe
    >   @@ -1,2 +1 @@
    >   -[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
    >   -done
    >   +

The output tells us what the current output is, and that the "tobe" output
is still empty.

For accepting this output, we run *pytest* with the *--reset-regtest*
flag:

    $ py.test --regtest-reset

Now the next execution of *py.test* will succeed:

    $ py.test

Now we break the test by modifying the code under test to compute the first
eleven square numbers:

    from __future__ import print_function

    def test_squares_up_to_ten(regtest):

        result = [i*i for i in range(11)]  # changed !

        # one way to record output:
        print(result, file=regtest)

        # alternative method to record output:
        regtest.write("done")

The next run of pytest delivers a nice diff of the current and expected output
from this test function:

    $ py.test

    ...
    >   --- current
    >   +++ tobe
    >   @@ -1,2 +1,2 @@
    >   -[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
    >   +[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
    >    done


The recorded output was written to files in the subfolder
`_regtest_outputs` next to your test script(s). You might keep this
folder under version control.


Other features
--------------

Another way to record output is to capture all output to `sys.stdout`:

    def test_squares_up_to_ten(regtest):

        result = [i*i for i in range(10)]

        with regtest():
            print result

You can reset recorded output of files and functions individually as:

    $ py.test --regtest-reset tests/test_00.py
    $ py.test --regtest-reset tests/test_00.py::test_squares_up_to_ten

To supress the diff and only see the stats use:

    $ py.test --regtest-nodiff

To see recorded output during test execution run:

    $ py.test --regtest-tee -s

If you develop on mixed platforms it might be usefull to ignore white
spaces at the end of the lines when comparing output. This can be
achieved by specifying:

    $ py.test --regtest-ignore-line-endings


Fixing unavoidable changes in recorded  output
----------------------------------------------

The recorded output can contain data which is changing from test run to test
run, e.g. pathes created with the `tmpdir` fixture or hexadecimal object ids,
when objects are printed.

The plugin already replaces such changing data in the recorded output,
and one can register own converters in `conftest.py` in the tests
folder. For example:

    import pytest_regtest

    @pytest_regtest.register_converter_pre
    def fix_before(txt):
        """modify recorded output before the default fixes
        like temp folders or hex object ids are applied"""

        # remove lines with passwords:
        lines = txt.split('\n')
        lines = [l for l in lines if "password is" not in l]
        return '\n'.join(lines)

    @pytest_regtest.register_converter_post
    def after(txt):
        """modify recorded output after the default fixes
        like temp folders or hex object ids are applied"""

        # for demo only
        return txt.upper()

This can be used to fix substrings like "computation need 1.23 seconds"
to "computation needed <TIME> seconds" etc.

One can register multiple such converters which will be applied in
order of registration.
            

Raw data

            {
    "_id": null,
    "home_page": "https://gitlab.com/uweschmitt/pytest-regtest",
    "name": "pytest-regtest",
    "maintainer": "",
    "docs_url": null,
    "requires_python": "",
    "maintainer_email": "",
    "keywords": "",
    "author": "Uwe Schmitt",
    "author_email": "uwe.schmitt@id.ethz.ch",
    "download_url": "https://files.pythonhosted.org/packages/8d/83/750161e38ce7e67cd75bd62c6b89bf35dce04532caa2116500fd3534feac/pytest-regtest-1.4.5.tar.gz",
    "platform": "",
    "description": "pytest-regtest\n==============\n\npytest-regtest is a *pytest*-plugin for implementing regression tests.\nCompared to functional testing a regression test does not test if\nsoftware produces correct results, instead a regression test checks if\nsoftware behaves the same way as it did before introduced changes.\n\nMore about regression testing at\n<https://en.wikipedia.org/wiki/Regression_testing>. Regression testing\nis a common technique to get started when refactoring legacy code\nlacking a test suite.\n\n*pytest-regtest* allows capturing selected output which then can be\ncompared to the captured output from former runs.\n\nTo install and activate this plugin execute:\n\n    $ pip install pytest-regtest\n\n*pytest-regtest* plugin provides a fixture named *regtest* which can be\nused as a file handle for recording data:\n\n    from __future__ import print_function\n\n    def test_squares_up_to_ten(regtest):\n\n        result = [i*i for i in range(10)]\n\n        # one way to record output:\n        print(result, file=regtest)\n\n        # alternative method to record output:\n        regtest.write(\"done\")\n\n        # or using a context manager:\n        with regtest:\n            print(\"this will be recorded\")\n\nIf you run this test script with *pytest* the first time there is no\nrecorded output for this test function so far and thus the test will\nfail with a message including a diff:\n\n    $ py.test\n    ...\n\n    regression test output differences for test_demo.py::test_squares_up_to_ten:\n\n    >   --- current\n    >   +++ tobe\n    >   @@ -1,2 +1 @@\n    >   -[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]\n    >   -done\n    >   +\n\nThe output tells us what the current output is, and that the \"tobe\" output\nis still empty.\n\nFor accepting this output, we run *pytest* with the *--reset-regtest*\nflag:\n\n    $ py.test --regtest-reset\n\nNow the next execution of *py.test* will succeed:\n\n    $ py.test\n\nNow we break the test by modifying the code under test to compute the first\neleven square numbers:\n\n    from __future__ import print_function\n\n    def test_squares_up_to_ten(regtest):\n\n        result = [i*i for i in range(11)]  # changed !\n\n        # one way to record output:\n        print(result, file=regtest)\n\n        # alternative method to record output:\n        regtest.write(\"done\")\n\nThe next run of pytest delivers a nice diff of the current and expected output\nfrom this test function:\n\n    $ py.test\n\n    ...\n    >   --- current\n    >   +++ tobe\n    >   @@ -1,2 +1,2 @@\n    >   -[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100]\n    >   +[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]\n    >    done\n\n\nThe recorded output was written to files in the subfolder\n`_regtest_outputs` next to your test script(s). You might keep this\nfolder under version control.\n\n\nOther features\n--------------\n\nAnother way to record output is to capture all output to `sys.stdout`:\n\n    def test_squares_up_to_ten(regtest):\n\n        result = [i*i for i in range(10)]\n\n        with regtest():\n            print result\n\nYou can reset recorded output of files and functions individually as:\n\n    $ py.test --regtest-reset tests/test_00.py\n    $ py.test --regtest-reset tests/test_00.py::test_squares_up_to_ten\n\nTo supress the diff and only see the stats use:\n\n    $ py.test --regtest-nodiff\n\nTo see recorded output during test execution run:\n\n    $ py.test --regtest-tee -s\n\nIf you develop on mixed platforms it might be usefull to ignore white\nspaces at the end of the lines when comparing output. This can be\nachieved by specifying:\n\n    $ py.test --regtest-ignore-line-endings\n\n\nFixing unavoidable changes in recorded  output\n----------------------------------------------\n\nThe recorded output can contain data which is changing from test run to test\nrun, e.g. pathes created with the `tmpdir` fixture or hexadecimal object ids,\nwhen objects are printed.\n\nThe plugin already replaces such changing data in the recorded output,\nand one can register own converters in `conftest.py` in the tests\nfolder. For example:\n\n    import pytest_regtest\n\n    @pytest_regtest.register_converter_pre\n    def fix_before(txt):\n        \"\"\"modify recorded output before the default fixes\n        like temp folders or hex object ids are applied\"\"\"\n\n        # remove lines with passwords:\n        lines = txt.split('\\n')\n        lines = [l for l in lines if \"password is\" not in l]\n        return '\\n'.join(lines)\n\n    @pytest_regtest.register_converter_post\n    def after(txt):\n        \"\"\"modify recorded output after the default fixes\n        like temp folders or hex object ids are applied\"\"\"\n\n        # for demo only\n        return txt.upper()\n\nThis can be used to fix substrings like \"computation need 1.23 seconds\"\nto \"computation needed <TIME> seconds\" etc.\n\nOne can register multiple such converters which will be applied in\norder of registration.",
    "bugtrack_url": null,
    "license": "https://opensource.org/licenses/MIT",
    "summary": "pytest plugin for regression tests",
    "version": "1.4.5",
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "524cedd133b465ebf727dcb12f5b85bf",
                "sha256": "a1e4e87d483e493d9bf6d15a71a46dedaccb6caab79fa8066f47a8eb3e6f86e3"
            },
            "downloads": -1,
            "filename": "pytest-regtest-1.4.5.tar.gz",
            "has_sig": false,
            "md5_digest": "524cedd133b465ebf727dcb12f5b85bf",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 9480,
            "upload_time": "2020-09-16T17:57:07",
            "upload_time_iso_8601": "2020-09-16T17:57:07.722467Z",
            "url": "https://files.pythonhosted.org/packages/8d/83/750161e38ce7e67cd75bd62c6b89bf35dce04532caa2116500fd3534feac/pytest-regtest-1.4.5.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2020-09-16 17:57:07",
    "github": false,
    "gitlab": true,
    "bitbucket": false,
    "gitlab_user": null,
    "gitlab_project": "uweschmitt",
    "lcname": "pytest-regtest"
}
        
Elapsed time: 0.15170s