# Mutagen
Mutagen is a plugin to pytest that makes it easy to do mutation testing. Mutation testing is a
method of testing your tests. Mutagen helps you to define "mutant" versions of your code---code
which is intentionally buggy---then you run your test suite on these mutants and verify that your
tests actually catch the bugs. Mutation testing helps you to gauge test coverage and verify that
your tests are good enough to exercise interesting behaviors in your code.
## For Property-Based Testing
If you are a user of a *property-based testing* framework such as Hypothesis, mutation testing can
also be used to test your input generators. It is relatively easy to write a generator that cannot
generate a certain kind of input. Mutation testing can be used to find those gaps.
# Installation
```
python3 -m pip install pytest-mutagen
```
# Usage
## Python import
`import pytest_mutagen as mg`
## Declare a mutant
* **Mutant function** \
To mutate a whole function you have to write the new version of the function, decorated with `@mg.mutant_of(function_qual_name, mutant_name, file (optional), description (optional))`. If the mutations affect an object (function or class) you have to be sure that this object exists in the `__globals__` symbols table of the mutant functions. For this purpose you can simply write `from [your_module] import [target_object]` in the mutation file.
Example:
```python
def inc(x):
return x + 1
@mg.mutant_of("inc", "INC_OBO", description="Increment is off by one.")
def inc_mut(x):
return x + 2
```
* **Mutant expression** \
If you don't want to change the whole function but only one line, you must decorate the function with `@mg.has_mutant(mutant_name, file (optional), description (optional))`. Then you have two ways to do it:
* By replacing the expression by the `mg.mut(mutant_name, normal_expression, mutant_expression)` function, using lambda expressions. \
Example:
`mg.mut("FLIP_LT", lambda: a < b, lambda: b < a)`
* Using the `mg.not_mutant(mutant_name)` function combined with an `if` statement. \
Example:
`k = inc(k) if mg.not_mutant("INC_OBO2") else inc(k) + 1`
### Mutating a class method
In fact the `@mutant_of` decorator doesn't require the function name but its fully qualified name. It does not change anything for top-level functions but in the case of a class method you need to write the dotted path leading to the object from the module top-level.
Example:
```python
class Foo:
def bar(self):
pass
@staticmethod
def static_bar():
pass
@mg.mutant_of("Foo.bar", "")
def bar_mut(self):
pass
@mg.mutant_of("Foo.static_bar", "")
def static_bar_mut():
pass
```
## Global functioning
Mutagen collects all declared mutants, stored per file names. Then it looks through all tests collected by pytest and apply the mutants to the matching files. This is handled by the optional file parameter in `@has_mutant` and `@mutant_of` which can be a file name or a list of file names where you want your mutant to be applied. You can set it to APPLY_TO_ALL (constant string declared in mutagen) if you want it to be applied to all collected files. By default, file is:
* APPLY_TO_ALL for `@has_mutant`
* the current file name for `@mutant_of` (the one where it is written)
Therefore you can either:
* write your mutations and specify for each one where you want it to be applied (use the function `mg.link_to_file(filename)` at the beginning of your file to link the current file to the specified filename)
* or create a mutations.py file where you import all test files you want (`from testfile.py import *`), write your `mutant_of` with no file specified and run pytest on mutation.py.
## Run the tests
`python3 -m pytest --mutate`
### Quick run
The `--quick-mut` option will stop each mutant after its first failed test. If not specified each mutant will run the whole test suite
### Cache use
Mutagen stores in the pytest cache the functions that failed during the last run, for each mutant. For the next runs it will try these functions first, in order to find failures more quickly. If you don't need this feature you can simply use the `--cache-clear` option that will clear the cache before running the tests.
### Run only the mutations
If you don't want to run the original test suite but only the mutations you can use the pytest option `--collect-only`
### Selective run of mutants
The `--select` option expects a comma-separated list of mutants (no spaces) and will run these ones exclusively.
Example:
```sh
python3 -m pytest --mutate --select INC_OBO,FLIP_LT
```
### Mutagen stats
The `--mutagen-stats` option adds a section to the terminal summary, which displays the number of tests that caught each mutant.
## Add trivial mutations
To find holes in a test suite with mutagen, we often try trivial mutations on some functions (like
replacing them with pass) to see whether a lot of tests catch them or not.
For this purpose the `trivial_mutations(functions, obj=None, file=APPLY_TO_ALL)` function with a
list of functions as input adds all mutants corresponding to replacing them by an empty function.
There are two ways to use it:
```python
from module import sort, invert, ExampleClass
# With a list of top-level functions
mg.trivial_mutations([sort, invert])
# With a list of method names and the corresponding object
mg.trivial_mutations(["sort", "clear"], ExampleClass)
```
This is equivalent to doing this:
```python
from module import sort, invert, ExampleClass
mg.link_to_file(mg.APPLY_TO_ALL)
@mg.mutant_of("sort", "SORT_NOTHING")
def sort_mut(*args, **kwargs):
pass
@mg.mutant_of("invert", "INVERT_NOTHING")
def invert_mut(*args, **kwargs):
pass
@mg.mutant_of("ExampleClass.sort", "EXAMPLECLASS.SORT_NOTHING")
def sort_mut(*args, **kwargs):
pass
@mg.mutant_of("ExampleClass.clear", "EXAMPLECLASS.CLEAR_NOTHING")
def clear_mut(*args, **kwargs):
pass
```
`trivial_mutations` has an optional _file_ parameter to specify the test file where the mutations
should be applied, which is by default set to APPLY_TO_ALL.
The function `trivial_mutations_all(object, file=APPLY_TO_ALL)` applies this process to each
method of the class (or list of classes) given as a parameter.
Example:
```python
from module import ExampleClass
mg.trivial_mutations_all(ExampleClass)
```
## Examples
You can find some examples in the examples folder
* The file short_example.py is a very simple example of the use of mutagen to test a merge sort function
* The file BST_mutations.py implements the Binary Search Tree data structure, and the test suite and mutations from _How to specify it!_ (John Hughes, 2019)
* The subfolder separate_files is an example of the separation between the source file, the test file and the mutation file
The run-tests.py scripts show how to run these tests
# Automatic tool
Writing mutations by hand can be very long and we are aware that it can discourage a lot of
programmers from using pytest-mutagen, that is initially a manual mutation-testing tool. To fix
this problem while keeping the ability to manually edit the mutants we've added the possibility
to interactively generate a bunch of mutants following a set of rules.
## Usage
```sh
python3 -m pytest_mutagen [-h] [-o OUTPUT_PATH] [-m MODULE_PATH] input_path
```
This command will browse the provided `input_path` (that can be a file or a directory) and
interactively propose several mutants. You can accept them by pressing ENTER and refuse them by
typing 'n' then ENTER. The purpose of this is to avoid false positives and equivalent mutants, that
are among the main problems of mutation testing. Finally all accepted mutants are written in the
mutagen syntax (ready to be use with `pytest --mutate`) in _mutation.py_ or the file/directory
specified with the `-o` command-line option.
For more details on its use you can use `python3 -m pytest_mutagen --help`.
## Rules
* Integers are incremented
* Operators are switched to a different (but close) operator
* In assignments, the right value is replaced with _None_
* The return statement is removed
* The condition of _if_ statements are replaced with _not (condition)_
Raw data
{
"_id": null,
"home_page": "https://github.com/hgoldstein95/pytest-mutagen",
"name": "pytest-mutagen",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": "",
"keywords": "python testing mutation mutant mutagen test",
"author": "Timothee Paquatte <timothee.paquatte@polytechnique.edu>, Harrison Goldstein <hgo@seas.upenn.edu>",
"author_email": "timothee.paquatte@polytechnique.edu",
"download_url": "https://files.pythonhosted.org/packages/f4/4a/a54c2695ecddcb1caa06decc6371cb3977cf7385ad52667087f233c82f41/pytest-mutagen-1.3.tar.gz",
"platform": "",
"description": "# Mutagen\n\nMutagen is a plugin to pytest that makes it easy to do mutation testing. Mutation testing is a\nmethod of testing your tests. Mutagen helps you to define \"mutant\" versions of your code---code\nwhich is intentionally buggy---then you run your test suite on these mutants and verify that your\ntests actually catch the bugs. Mutation testing helps you to gauge test coverage and verify that\nyour tests are good enough to exercise interesting behaviors in your code.\n\n## For Property-Based Testing\n\nIf you are a user of a *property-based testing* framework such as Hypothesis, mutation testing can\nalso be used to test your input generators. It is relatively easy to write a generator that cannot\ngenerate a certain kind of input. Mutation testing can be used to find those gaps.\n\n\n# Installation\n\n```\npython3 -m pip install pytest-mutagen\n```\n\n# Usage\n## Python import\n`import pytest_mutagen as mg`\n\n## Declare a mutant\n* **Mutant function** \\\n\tTo mutate a whole function you have to write the new version of the function, decorated with `@mg.mutant_of(function_qual_name, mutant_name, file (optional), description (optional))`. If the mutations affect an object (function or class) you have to be sure that this object exists in the `__globals__` symbols table of the mutant functions. For this purpose you can simply write `from [your_module] import [target_object]` in the mutation file.\n\tExample:\n\n\t```python\n\tdef inc(x):\n\t\treturn x + 1\n\n\t@mg.mutant_of(\"inc\", \"INC_OBO\", description=\"Increment is off by one.\")\n\tdef inc_mut(x):\n\t\treturn x + 2\n\t```\n\n* **Mutant expression** \\\n\tIf you don't want to change the whole function but only one line, you must decorate the function with `@mg.has_mutant(mutant_name, file (optional), description (optional))`. Then you have two ways to do it:\n\n * By replacing the expression by the `mg.mut(mutant_name, normal_expression, mutant_expression)` function, using lambda expressions. \\\n\t\t\tExample:\n\t\t\t`mg.mut(\"FLIP_LT\", lambda: a < b, lambda: b < a)`\n\n * Using the `mg.not_mutant(mutant_name)` function combined with an `if` statement. \\\n\t\t\tExample:\n\t\t\t`k = inc(k) if mg.not_mutant(\"INC_OBO2\") else inc(k) + 1`\n\n### Mutating a class method\n\nIn fact the `@mutant_of` decorator doesn't require the function name but its fully qualified name. It does not change anything for top-level functions but in the case of a class method you need to write the dotted path leading to the object from the module top-level.\nExample:\n```python\nclass Foo:\n\tdef bar(self):\n\t\tpass\n\n\t@staticmethod\n\tdef static_bar():\n\t\tpass\n\n@mg.mutant_of(\"Foo.bar\", \"\")\ndef bar_mut(self):\n\tpass\n\n@mg.mutant_of(\"Foo.static_bar\", \"\")\ndef static_bar_mut():\n\tpass\n```\n\n## Global functioning\n\nMutagen collects all declared mutants, stored per file names. Then it looks through all tests collected by pytest and apply the mutants to the matching files. This is handled by the optional file parameter in `@has_mutant` and `@mutant_of` which can be a file name or a list of file names where you want your mutant to be applied. You can set it to APPLY_TO_ALL (constant string declared in mutagen) if you want it to be applied to all collected files. By default, file is:\n* APPLY_TO_ALL for `@has_mutant`\n* the current file name for `@mutant_of` (the one where it is written)\n\nTherefore you can either:\n* write your mutations and specify for each one where you want it to be applied (use the function `mg.link_to_file(filename)` at the beginning of your file to link the current file to the specified filename)\n* or create a mutations.py file where you import all test files you want (`from testfile.py import *`), write your `mutant_of` with no file specified and run pytest on mutation.py.\n\n## Run the tests\n\n`python3 -m pytest --mutate`\n\n### Quick run\n\nThe `--quick-mut` option will stop each mutant after its first failed test. If not specified each mutant will run the whole test suite\n\n### Cache use\n\nMutagen stores in the pytest cache the functions that failed during the last run, for each mutant. For the next runs it will try these functions first, in order to find failures more quickly. If you don't need this feature you can simply use the `--cache-clear` option that will clear the cache before running the tests.\n\n### Run only the mutations\n\nIf you don't want to run the original test suite but only the mutations you can use the pytest option `--collect-only`\n\n### Selective run of mutants\n\nThe `--select` option expects a comma-separated list of mutants (no spaces) and will run these ones exclusively.\nExample:\n```sh\npython3 -m pytest --mutate --select INC_OBO,FLIP_LT\n```\n\n### Mutagen stats\n\nThe `--mutagen-stats` option adds a section to the terminal summary, which displays the number of tests that caught each mutant.\n\n## Add trivial mutations\n\nTo find holes in a test suite with mutagen, we often try trivial mutations on some functions (like\nreplacing them with pass) to see whether a lot of tests catch them or not.\nFor this purpose the `trivial_mutations(functions, obj=None, file=APPLY_TO_ALL)` function with a\nlist of functions as input adds all mutants corresponding to replacing them by an empty function.\nThere are two ways to use it:\n\n```python\nfrom module import sort, invert, ExampleClass\n\n# With a list of top-level functions\nmg.trivial_mutations([sort, invert])\n\n# With a list of method names and the corresponding object\nmg.trivial_mutations([\"sort\", \"clear\"], ExampleClass)\n\n```\n\nThis is equivalent to doing this:\n\n```python\nfrom module import sort, invert, ExampleClass\n\nmg.link_to_file(mg.APPLY_TO_ALL)\n\n@mg.mutant_of(\"sort\", \"SORT_NOTHING\")\ndef sort_mut(*args, **kwargs):\n\tpass\n\n@mg.mutant_of(\"invert\", \"INVERT_NOTHING\")\ndef invert_mut(*args, **kwargs):\n\tpass\n\n@mg.mutant_of(\"ExampleClass.sort\", \"EXAMPLECLASS.SORT_NOTHING\")\ndef sort_mut(*args, **kwargs):\n\tpass\n\n@mg.mutant_of(\"ExampleClass.clear\", \"EXAMPLECLASS.CLEAR_NOTHING\")\ndef clear_mut(*args, **kwargs):\n\tpass\n```\n\n`trivial_mutations` has an optional _file_ parameter to specify the test file where the mutations\nshould be applied, which is by default set to APPLY_TO_ALL.\n\nThe function `trivial_mutations_all(object, file=APPLY_TO_ALL)` applies this process to each\nmethod of the class (or list of classes) given as a parameter.\nExample:\n\n```python\nfrom module import ExampleClass\n\nmg.trivial_mutations_all(ExampleClass)\n```\n\n## Examples\nYou can find some examples in the examples folder\n* The file short_example.py is a very simple example of the use of mutagen to test a merge sort function\n* The file BST_mutations.py implements the Binary Search Tree data structure, and the test suite and mutations from _How to specify it!_ (John Hughes, 2019)\n* The subfolder separate_files is an example of the separation between the source file, the test file and the mutation file\n\n\nThe run-tests.py scripts show how to run these tests\n\n# Automatic tool\n\nWriting mutations by hand can be very long and we are aware that it can discourage a lot of\nprogrammers from using pytest-mutagen, that is initially a manual mutation-testing tool. To fix\nthis problem while keeping the ability to manually edit the mutants we've added the possibility\nto interactively generate a bunch of mutants following a set of rules.\n\n## Usage\n\n```sh\npython3 -m pytest_mutagen [-h] [-o OUTPUT_PATH] [-m MODULE_PATH] input_path\n```\n\nThis command will browse the provided `input_path` (that can be a file or a directory) and\ninteractively propose several mutants. You can accept them by pressing ENTER and refuse them by\ntyping 'n' then ENTER. The purpose of this is to avoid false positives and equivalent mutants, that\nare among the main problems of mutation testing. Finally all accepted mutants are written in the\nmutagen syntax (ready to be use with `pytest --mutate`) in _mutation.py_ or the file/directory\nspecified with the `-o` command-line option.\nFor more details on its use you can use `python3 -m pytest_mutagen --help`.\n\n## Rules\n\n* Integers are incremented\n* Operators are switched to a different (but close) operator\n* In assignments, the right value is replaced with _None_\n* The return statement is removed\n* The condition of _if_ statements are replaced with _not (condition)_\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Add the mutation testing feature to pytest",
"version": "1.3",
"project_urls": {
"Homepage": "https://github.com/hgoldstein95/pytest-mutagen"
},
"split_keywords": [
"python",
"testing",
"mutation",
"mutant",
"mutagen",
"test"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "12207cf83208d035169d57905510b8bd8911bde94ebe1a97cbf956312eae5e78",
"md5": "344193555890d4b68f2f46fddd0bef41",
"sha256": "c36369d73bb1ee46a03db10eda3ce9cb8304adb96cf01b8618968032723de7b7"
},
"downloads": -1,
"filename": "pytest_mutagen-1.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "344193555890d4b68f2f46fddd0bef41",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 16954,
"upload_time": "2020-07-24T14:34:42",
"upload_time_iso_8601": "2020-07-24T14:34:42.417989Z",
"url": "https://files.pythonhosted.org/packages/12/20/7cf83208d035169d57905510b8bd8911bde94ebe1a97cbf956312eae5e78/pytest_mutagen-1.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f44aa54c2695ecddcb1caa06decc6371cb3977cf7385ad52667087f233c82f41",
"md5": "50266f1d271c9b5d4c54ce01f6fe3028",
"sha256": "a0f6780f718f83dcb0e96b74e66332a42d1d3abe64a4c23d1cb7fca3de2ecc97"
},
"downloads": -1,
"filename": "pytest-mutagen-1.3.tar.gz",
"has_sig": false,
"md5_digest": "50266f1d271c9b5d4c54ce01f6fe3028",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 18106,
"upload_time": "2020-07-24T14:34:43",
"upload_time_iso_8601": "2020-07-24T14:34:43.951649Z",
"url": "https://files.pythonhosted.org/packages/f4/4a/a54c2695ecddcb1caa06decc6371cb3977cf7385ad52667087f233c82f41/pytest-mutagen-1.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2020-07-24 14:34:43",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "hgoldstein95",
"github_project": "pytest-mutagen",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "pytest-mutagen"
}
JSON
