# pytest-sherlock
[](https://github.com/DKorytkin/pytest-sherlock/actions/workflows/main.yml?query=branch%3Amaster)
[](https://codecov.io/gh/DKorytkin/pytest-sherlock/branch/master)
[](https://pypi.org/project/pytest-sherlock/)
[](https://pypi.org/project/pytest-sherlock/)
[](https://pypi.org/project/pytest-sherlock/)
Pytest plugin which help to find coupled tests.
Sometimes we have coupled tests which depend from ordering
For example:
- **PASSES** `tests/exmaple/test_all_read.py tests/exmaple/test_b_modify.py tests/exmaple/test_c_delete.py`
- **FAILED** `tests/exmaple/test_c_delete.py tests/exmaple/test_b_modify.py tests/exmaple/test_all_read.py`
In this case pretty simple to detect coupled tests, but if we have >=1k tests which called before it will hard
## Content:
- [install](#install)
- [how to use](#how-to-use)
- [waiting in the future](#todo)
### Install
```bash
pip install pytest-sherlock
```
### how to use:
```bash
pytest tests/exmaple/test_c_delete.py tests/exmaple/test_b_modify.py tests/exmaple/test_all_read.py --flaky-test="test_read_params" -vv
```
Plugin didn't run all tests, it try to find some possible guilty test and will run first
```bash
Try to find coupled tests in [3-4] steps
__________________________ Step [1 of 4]: __________________________
tests/exmaple/test_c_delete.py::test_delete_random_param PASSED [ 20%]
tests/exmaple/test_b_modify.py::test_modify_random_param PASSED [ 40%]
tests/exmaple/test_c_delete.py::test_deleted_passed PASSED [ 60%]
tests/exmaple/test_c_delete.py::test_do_not_delete PASSED [ 80%]
tests/exmaple/test_all_read.py::test_read_params FAILED [100%]
__________________________ Step [2 of 4]: __________________________
tests/exmaple/test_c_delete.py::test_delete_random_param PASSED [ 33%]
tests/exmaple/test_b_modify.py::test_modify_random_param PASSED [ 66%]
tests/exmaple/test_all_read.py::test_read_params FAILED [100%]
__________________________ Step [3 of 4]: __________________________
tests/exmaple/test_c_delete.py::test_delete_random_param PASSED [ 50%]
tests/exmaple/test_all_read.py::test_read_params PASSED [100%]
__________________________ Step [4 of 4]: __________________________
tests/exmaple/test_b_modify.py::test_modify_random_param PASSED [ 50%]
tests/exmaple/test_all_read.py::test_read_params FAILED [100%]
============================== FAILURES ==============================
__________________________ test_read_params __________________________
Found coupled tests:
tests/exmaple/test_b_modify.py::test_modify_random_param
tests/exmaple/test_all_read.py::test_read_params
Common fixtures:
config
How to reproduce:
pytest -l -vv tests/exmaple/test_b_modify.py::test_modify_random_param tests/exmaple/test_all_read.py::test_read_params
AssertionError: assert 13 == 2
+ where 13 = <built-in method get of dict object at 0x102664280>('b')
+ where <built-in method get of dict object at 0x102664280> = {'a': 1, 'b': 13, 'c': 3}.get
tests/exmaple/test_all_read.py:8: AssertionError
=================== 1 failed, 9 passed in 0.08 seconds ===================
```
The plugin also supports `--lf` (last failed) option to rerun found coupled tests.
It allows us to make sure that coupled tests are true.
```shell
pytest tests/exmaple/test_c_delete.py tests/exmaple/test_b_modify.py tests/exmaple/test_all_read.py --lf
```
```shell
=========================================================================== test session starts ============================================================================
platform darwin -- Python 3.7.10, pytest-3.5.1, py-1.7.0, pluggy-0.6.0
collected 10 items / 8 deselected
run-last-failure: rerun previous 2 failures
tests/exmaple/test_b_modify.py . [ 50%]
tests/exmaple/test_all_read.py F [100%]
================================================================================= FAILURES =================================================================================
_____________________________________________________________________________ test_read_params _____________________________________________________________________________
config = {'a': 1, 'b': 13, 'c': 3}, param = 'b'
def test_read_params(config, param):
> assert config.get(param) == 2
E AssertionError: assert 13 == 2
E + where 13 = <built-in method get of dict object at 0x10c964cd0>('b')
E + where <built-in method get of dict object at 0x10c964cd0> = {'a': 1, 'b': 13, 'c': 3}.get
tests/exmaple/test_all_read.py:6: AssertionError
============================================================= 1 failed, 1 passed, 8 deselected in 0.05 seconds =============================================================
```
### TODO
I have a couple ideas, how to improve finder coupled tests:
- use **AST** for detect common peace of code *(variables, functions, etc...)*
Raw data
{
"_id": null,
"home_page": "https://github.com/DKorytkin/pytest-sherlock",
"name": "pytest-sherlock",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "py.test,pytest,flaky tests,coupled tests,debug tests",
"author": "Denys Korytkin",
"author_email": "dkorytkin@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/ff/67/acf66aa57e8fcc8ce1f6398e92a4d34ce8752561f87f1f33042b3f020442/pytest-sherlock-0.3.4.tar.gz",
"platform": null,
"description": "# pytest-sherlock\n\n[](https://github.com/DKorytkin/pytest-sherlock/actions/workflows/main.yml?query=branch%3Amaster)\n[](https://codecov.io/gh/DKorytkin/pytest-sherlock/branch/master)\n[](https://pypi.org/project/pytest-sherlock/)\n[](https://pypi.org/project/pytest-sherlock/)\n[](https://pypi.org/project/pytest-sherlock/)\n\nPytest plugin which help to find coupled tests.\n\nSometimes we have coupled tests which depend from ordering\n\nFor example:\n- **PASSES** `tests/exmaple/test_all_read.py tests/exmaple/test_b_modify.py tests/exmaple/test_c_delete.py`\n- **FAILED** `tests/exmaple/test_c_delete.py tests/exmaple/test_b_modify.py tests/exmaple/test_all_read.py`\n\nIn this case pretty simple to detect coupled tests, but if we have >=1k tests which called before it will hard\n\n\n## Content:\n- [install](#install)\n- [how to use](#how-to-use)\n- [waiting in the future](#todo)\n\n### Install\n```bash\npip install pytest-sherlock\n```\n\n### how to use:\n```bash\npytest tests/exmaple/test_c_delete.py tests/exmaple/test_b_modify.py tests/exmaple/test_all_read.py --flaky-test=\"test_read_params\" -vv\n```\n\nPlugin didn't run all tests, it try to find some possible guilty test and will run first\n\n```bash\nTry to find coupled tests in [3-4] steps\n__________________________ Step [1 of 4]: __________________________\n\ntests/exmaple/test_c_delete.py::test_delete_random_param PASSED [ 20%]\ntests/exmaple/test_b_modify.py::test_modify_random_param PASSED [ 40%]\ntests/exmaple/test_c_delete.py::test_deleted_passed PASSED [ 60%]\ntests/exmaple/test_c_delete.py::test_do_not_delete PASSED [ 80%]\ntests/exmaple/test_all_read.py::test_read_params FAILED [100%]\n__________________________ Step [2 of 4]: __________________________\n\ntests/exmaple/test_c_delete.py::test_delete_random_param PASSED [ 33%]\ntests/exmaple/test_b_modify.py::test_modify_random_param PASSED [ 66%]\ntests/exmaple/test_all_read.py::test_read_params FAILED [100%]\n__________________________ Step [3 of 4]: __________________________\n\ntests/exmaple/test_c_delete.py::test_delete_random_param PASSED [ 50%]\ntests/exmaple/test_all_read.py::test_read_params PASSED [100%]\n__________________________ Step [4 of 4]: __________________________\n\ntests/exmaple/test_b_modify.py::test_modify_random_param PASSED [ 50%]\ntests/exmaple/test_all_read.py::test_read_params FAILED [100%]\n\n============================== FAILURES ==============================\n__________________________ test_read_params __________________________\n\nFound coupled tests:\ntests/exmaple/test_b_modify.py::test_modify_random_param\ntests/exmaple/test_all_read.py::test_read_params\n\nCommon fixtures:\nconfig\n\nHow to reproduce:\npytest -l -vv tests/exmaple/test_b_modify.py::test_modify_random_param tests/exmaple/test_all_read.py::test_read_params\n\n\nAssertionError: assert 13 == 2\n + where 13 = <built-in method get of dict object at 0x102664280>('b')\n + where <built-in method get of dict object at 0x102664280> = {'a': 1, 'b': 13, 'c': 3}.get\n\ntests/exmaple/test_all_read.py:8: AssertionError\n=================== 1 failed, 9 passed in 0.08 seconds ===================\n```\n\nThe plugin also supports `--lf` (last failed) option to rerun found coupled tests.\nIt allows us to make sure that coupled tests are true.\n\n```shell\npytest tests/exmaple/test_c_delete.py tests/exmaple/test_b_modify.py tests/exmaple/test_all_read.py --lf\n```\n```shell\n=========================================================================== test session starts ============================================================================\nplatform darwin -- Python 3.7.10, pytest-3.5.1, py-1.7.0, pluggy-0.6.0\ncollected 10 items / 8 deselected\nrun-last-failure: rerun previous 2 failures\n\ntests/exmaple/test_b_modify.py . [ 50%]\ntests/exmaple/test_all_read.py F [100%]\n\n================================================================================= FAILURES =================================================================================\n_____________________________________________________________________________ test_read_params _____________________________________________________________________________\n\nconfig = {'a': 1, 'b': 13, 'c': 3}, param = 'b'\n\n def test_read_params(config, param):\n> assert config.get(param) == 2\nE AssertionError: assert 13 == 2\nE + where 13 = <built-in method get of dict object at 0x10c964cd0>('b')\nE + where <built-in method get of dict object at 0x10c964cd0> = {'a': 1, 'b': 13, 'c': 3}.get\n\ntests/exmaple/test_all_read.py:6: AssertionError\n============================================================= 1 failed, 1 passed, 8 deselected in 0.05 seconds =============================================================\n```\n\n### TODO\nI have a couple ideas, how to improve finder coupled tests:\n- use **AST** for detect common peace of code *(variables, functions, etc...)*\n",
"bugtrack_url": null,
"license": "MIT license",
"summary": "pytest plugin help to find coupled tests",
"version": "0.3.4",
"project_urls": {
"Homepage": "https://github.com/DKorytkin/pytest-sherlock"
},
"split_keywords": [
"py.test",
"pytest",
"flaky tests",
"coupled tests",
"debug tests"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f7a55ba64c646ca1cace7a0797e3a847a76d9154b4056305f5b52d7bab06bdec",
"md5": "454014f590186c826b5746f2b2cc8015",
"sha256": "be70850cd7504e343f17537a892cbe0ce7dff95db440331a25188370da98ab30"
},
"downloads": -1,
"filename": "pytest_sherlock-0.3.4-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "454014f590186c826b5746f2b2cc8015",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": ">=3.7",
"size": 11318,
"upload_time": "2023-08-14T16:10:20",
"upload_time_iso_8601": "2023-08-14T16:10:20.721561Z",
"url": "https://files.pythonhosted.org/packages/f7/a5/5ba64c646ca1cace7a0797e3a847a76d9154b4056305f5b52d7bab06bdec/pytest_sherlock-0.3.4-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ff67acf66aa57e8fcc8ce1f6398e92a4d34ce8752561f87f1f33042b3f020442",
"md5": "3e7e898126581438fae03cd8b8bbe818",
"sha256": "82e5958ac459f69858e8b268e7e1a9078498b8ab3d5f6a0a29e6a9cba42d5f8a"
},
"downloads": -1,
"filename": "pytest-sherlock-0.3.4.tar.gz",
"has_sig": false,
"md5_digest": "3e7e898126581438fae03cd8b8bbe818",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 18361,
"upload_time": "2023-08-14T16:10:21",
"upload_time_iso_8601": "2023-08-14T16:10:21.699748Z",
"url": "https://files.pythonhosted.org/packages/ff/67/acf66aa57e8fcc8ce1f6398e92a4d34ce8752561f87f1f33042b3f020442/pytest-sherlock-0.3.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-14 16:10:21",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "DKorytkin",
"github_project": "pytest-sherlock",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "pytest-sherlock"
}
JSON
