# pytest-ranking
[](https://github.com/softwareTestingResearch/pytest-ranking/actions?workflow=CI)
[](https://pypi.org/project/pytest-ranking)
[](https://results.pre-commit.ci/latest/github/softwareTestingResearch/pytest-ranking/main)
A Pytest plugin for reducing the failure detection time with automated test prioritization/ranking.
This [pytest](https://github.com/pytest-dev/pytest) plugin allows you to find failures faster and receive sooner debugging feedback from CI. It speeds up the failure detection of your test suite by prioritizing the execution of tests that are faster, recently failed, and/or related to code change.
## Installation
To install `pytest-ranking` via [pip](https://pypi.org/project/pip/) from [PyPI](https://pypi.org/project):
```bash
pip install pytest-ranking
```
## Usage
Pytest will automatically find the plugin and use it when you run ``pytest``. You can use the default prioritization heuristic, which runs tests that have shorter execution times first by passing the ``--rank`` option:
```bash
pytest --rank
```
After the test run finishes, the terminal summary will show the config and overhead of running the plugin of the finished run, for example:
```text
============================================= pytest-ranking summary info =============================================
weights: 1-0-0
level: param
look-back history length: 50
number of *.py src files with new hashes: 0
test-change similarity compute time (s): 0.00225
test order compute time (s): 0.00033
feature collection time (s): 0.00246
```
### Weighting ranking heuristics
You can configure the weights of different prioritization heuristics by passing the optional `--rank-weight` flag with formatted values:
```bash
pytest --rank --rank-weight=0-1-0
```
Weights are separated by ``-``. The 1st weight is for running faster tests, the 2nd weight is for running recently failed tests, and the 3rd weight is for running tests more similar to the changed `*.py` files since the last run.
All weights must be integers or floats, and their sum will be normalized to 1.
A higher weight means that a corresponding heuristic is favored. The default value is ``1-0-0``, which only prioritizes faster tests.
### Running in different granularities
You can configure at which granularity your test suite will be re-ordered by passing the optional `--rank-level` flag in one of these values: `param`, `method`, `file`, `folder`. For example:
```bash
pytest --rank --rank-level=method
```
The smallest test item that can be re-ordered in the test suite in pytest is [parametrized unit test](https://docs.pytest.org/en/7.1.x/example/parametrize.html) (PUT). `param` ranks each PUT and re-arranges their execution order based on their assigned ranks; `method` ranks each test method, parametrized values of that test method will follow pytest's default execution order (alphabetical); `file` ranks each test file, all tests in the test file will follow pytest's default execution order; `folder` ranks each test directory that hosts the test files, all tests hosted in the folder will follow the default order.
### Tracking heuristics from historical runs
You can also configure the maximum window size for looking into previous test runs, which is used to compute the number of runs since a test had failed, by passing the optional `--rank-hist-len` flag (the default value is 50):
```bash
pytest --rank --rank-hist-len=30
```
Note that the plugin does not store any historical run logs, it merely resets cached ranking heuristics after every `rank-hist-len` number of runs.
### Running tests in random order
You can prompt `pytest-ranking` to run tests in random order, by setting the sum of `--rank-weight` option to 0, e.g., `--rank-weight=0-0-0`.
You can also configure the seed used when running tests in random order, via setting an integer to the option `--rank-seed`.
For example, the command below runs tests randomly with seed `1234`:
```bash
pytest --rank --rank-weight=0-0-0 --rank-seed=1234
```
### Passing plugin options via config file
You can always apply available options by adding them to the ``addopts`` setting in your [pytest.ini](https://docs.pytest.org/en/latest/reference/customize.html#configuration).
For example, create `pytest.ini` in your codebase root folder as such:
```ini
[pytest]
addopts = --rank --rank-weight=0-1-0 --rank-hist-len=30
```
and run `pytest` on the command line.
Alternatively, you can also create `pytest.ini` in your codebase root folder as such:
```ini
[pytest]
rank_weight=0-1-0
rank_hist_len=30
```
and run `pytest --rank` on the command line.
### Compatibility
Because `pytest-ranking` re-orders tests, it is not compatible with other pytest plugins that enforce other test orders, e.g., [pytest-randomly](https://github.com/pytest-dev/pytest-randomly), [pytest-random-order](https://github.com/pytest-dev/pytest-random-order), [pytest-reverse](https://github.com/adamchainz/pytest-reverse).
## Contributing
Contributions are very welcome. Tests can be run with [tox](https://tox.readthedocs.io/en/latest/).
## License
Distributed under the terms of the [MIT](http://opensource.org/licenses/MIT) license, `pytest-ranking` is free and open-source software.
## Issues
If you encounter any problems, please [file an issue](https://github.com/softwareTestingResearch/pytest-ranking/issues) or [pull request](https://github.com/softwareTestingResearch/pytest-ranking/pulls) along with a detailed description.
Raw data
{
"_id": null,
"home_page": "https://github.com/softwareTestingResearch/pytest-ranking",
"name": "pytest-ranking",
"maintainer": "softwareTestingResearch",
"docs_url": null,
"requires_python": ">=3.5",
"maintainer_email": "testingresearch4all@gmail.com",
"keywords": "software testing, regression testing, test prioritization, pytest",
"author": "softwareTestingResearch",
"author_email": "testingresearch4all@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/47/50/58f0db49b9bfcfa452b95c7744c83796ffbbd445481a27826382f74957db/pytest_ranking-0.3.1.tar.gz",
"platform": null,
"description": "\n# pytest-ranking\n\n\n\n[](https://github.com/softwareTestingResearch/pytest-ranking/actions?workflow=CI)\n[](https://pypi.org/project/pytest-ranking)\n[](https://results.pre-commit.ci/latest/github/softwareTestingResearch/pytest-ranking/main)\n\n\nA Pytest plugin for reducing the failure detection time with automated test prioritization/ranking.\n\nThis [pytest](https://github.com/pytest-dev/pytest) plugin allows you to find failures faster and receive sooner debugging feedback from CI. It speeds up the failure detection of your test suite by prioritizing the execution of tests that are faster, recently failed, and/or related to code change.\n\n## Installation\n\nTo install `pytest-ranking` via [pip](https://pypi.org/project/pip/) from [PyPI](https://pypi.org/project):\n\n```bash\npip install pytest-ranking\n```\n\n\n## Usage\n\nPytest will automatically find the plugin and use it when you run ``pytest``. You can use the default prioritization heuristic, which runs tests that have shorter execution times first by passing the ``--rank`` option:\n\n```bash\npytest --rank\n```\n\nAfter the test run finishes, the terminal summary will show the config and overhead of running the plugin of the finished run, for example:\n\n ```text\n============================================= pytest-ranking summary info =============================================\nweights: 1-0-0\nlevel: param\nlook-back history length: 50\nnumber of *.py src files with new hashes: 0\ntest-change similarity compute time (s): 0.00225\ntest order compute time (s): 0.00033\nfeature collection time (s): 0.00246\n```\n\n\n### Weighting ranking heuristics\n\nYou can configure the weights of different prioritization heuristics by passing the optional `--rank-weight` flag with formatted values:\n\n```bash\npytest --rank --rank-weight=0-1-0\n```\n\nWeights are separated by ``-``. The 1st weight is for running faster tests, the 2nd weight is for running recently failed tests, and the 3rd weight is for running tests more similar to the changed `*.py` files since the last run.\nAll weights must be integers or floats, and their sum will be normalized to 1.\nA higher weight means that a corresponding heuristic is favored. The default value is ``1-0-0``, which only prioritizes faster tests.\n\n\n### Running in different granularities\n\nYou can configure at which granularity your test suite will be re-ordered by passing the optional `--rank-level` flag in one of these values: `param`, `method`, `file`, `folder`. For example:\n\n```bash\npytest --rank --rank-level=method\n```\n\nThe smallest test item that can be re-ordered in the test suite in pytest is [parametrized unit test](https://docs.pytest.org/en/7.1.x/example/parametrize.html) (PUT). `param` ranks each PUT and re-arranges their execution order based on their assigned ranks; `method` ranks each test method, parametrized values of that test method will follow pytest's default execution order (alphabetical); `file` ranks each test file, all tests in the test file will follow pytest's default execution order; `folder` ranks each test directory that hosts the test files, all tests hosted in the folder will follow the default order.\n\n### Tracking heuristics from historical runs\n\nYou can also configure the maximum window size for looking into previous test runs, which is used to compute the number of runs since a test had failed, by passing the optional `--rank-hist-len` flag (the default value is 50):\n\n```bash\npytest --rank --rank-hist-len=30\n```\n\nNote that the plugin does not store any historical run logs, it merely resets cached ranking heuristics after every `rank-hist-len` number of runs.\n\n### Running tests in random order\n\nYou can prompt `pytest-ranking` to run tests in random order, by setting the sum of `--rank-weight` option to 0, e.g., `--rank-weight=0-0-0`.\nYou can also configure the seed used when running tests in random order, via setting an integer to the option `--rank-seed`.\nFor example, the command below runs tests randomly with seed `1234`:\n\n```bash\npytest --rank --rank-weight=0-0-0 --rank-seed=1234\n```\n\n### Passing plugin options via config file\n\nYou can always apply available options by adding them to the ``addopts`` setting in your [pytest.ini](https://docs.pytest.org/en/latest/reference/customize.html#configuration).\n\nFor example, create `pytest.ini` in your codebase root folder as such:\n```ini\n[pytest]\naddopts = --rank --rank-weight=0-1-0 --rank-hist-len=30\n```\nand run `pytest` on the command line.\n\nAlternatively, you can also create `pytest.ini` in your codebase root folder as such:\n```ini\n[pytest]\nrank_weight=0-1-0\nrank_hist_len=30\n```\n\nand run `pytest --rank` on the command line.\n\n\n\n### Compatibility\n\nBecause `pytest-ranking` re-orders tests, it is not compatible with other pytest plugins that enforce other test orders, e.g., [pytest-randomly](https://github.com/pytest-dev/pytest-randomly), [pytest-random-order](https://github.com/pytest-dev/pytest-random-order), [pytest-reverse](https://github.com/adamchainz/pytest-reverse).\n\n## Contributing\n\nContributions are very welcome. Tests can be run with [tox](https://tox.readthedocs.io/en/latest/).\n\n\n\n## License\n\nDistributed under the terms of the [MIT](http://opensource.org/licenses/MIT) license, `pytest-ranking` is free and open-source software.\n\n## Issues\n\nIf you encounter any problems, please [file an issue](https://github.com/softwareTestingResearch/pytest-ranking/issues) or [pull request](https://github.com/softwareTestingResearch/pytest-ranking/pulls) along with a detailed description.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A Pytest plugin for automatically prioritizing/ranking tests to speed up failure detection",
"version": "0.3.1",
"project_urls": {
"Homepage": "https://github.com/softwareTestingResearch/pytest-ranking"
},
"split_keywords": [
"software testing",
" regression testing",
" test prioritization",
" pytest"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8901fd7a465d1402b54a8af35965e0c97b76ab3143fd8d6d50f723d0936c0d6f",
"md5": "0c5bccb68bb52bae516138e1fec1ce2d",
"sha256": "563babe1feda395953025fe5a767d4c2617e787b7352ab706c9106f732a26371"
},
"downloads": -1,
"filename": "pytest_ranking-0.3.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0c5bccb68bb52bae516138e1fec1ce2d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.5",
"size": 11179,
"upload_time": "2024-06-07T20:31:38",
"upload_time_iso_8601": "2024-06-07T20:31:38.042907Z",
"url": "https://files.pythonhosted.org/packages/89/01/fd7a465d1402b54a8af35965e0c97b76ab3143fd8d6d50f723d0936c0d6f/pytest_ranking-0.3.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "475058f0db49b9bfcfa452b95c7744c83796ffbbd445481a27826382f74957db",
"md5": "2f9f62a0cd24f204ce64361262991261",
"sha256": "e1d8d5942f34f7ac5e6af411a7dd217e277e9606d609f38b27ce540be1bde0a0"
},
"downloads": -1,
"filename": "pytest_ranking-0.3.1.tar.gz",
"has_sig": false,
"md5_digest": "2f9f62a0cd24f204ce64361262991261",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.5",
"size": 14424,
"upload_time": "2024-06-07T20:31:39",
"upload_time_iso_8601": "2024-06-07T20:31:39.655561Z",
"url": "https://files.pythonhosted.org/packages/47/50/58f0db49b9bfcfa452b95c7744c83796ffbbd445481a27826382f74957db/pytest_ranking-0.3.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-06-07 20:31:39",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "softwareTestingResearch",
"github_project": "pytest-ranking",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "pytest-ranking"
}
JSON
