# pytest-split
[](https://pypi.python.org/pypi/pytest-split/)
[](https://pypi.python.org/pypi/pytest-split/)
[](https://pypi.python.org/pypi/pytest-split/)
[](https://github.com/woltapp/wolt-python-package-cookiecutter)
---
**Documentation**: [https://jerry-git.github.io/pytest-split](https://jerry-git.github.io/pytest-split)
**Source Code**: [https://github.com/jerry-git/pytest-split](https://github.com/jerry-git/pytest-split)
**PyPI**: [https://pypi.org/project/pytest-split/](https://pypi.org/project/pytest-split/)
---
Pytest plugin which splits the test suite to equally sized "sub suites" based on test execution time.
## Motivation
* Splitting the test suite is a prerequisite for parallelization (who does not want faster CI builds?). It's valuable to have sub suites which execution time is around the same.
* [`pytest-test-groups`](https://pypi.org/project/pytest-test-groups/) is great but it does not take into account the execution time of sub suites which can lead to notably unbalanced execution times between the sub suites.
* [`pytest-xdist`](https://pypi.org/project/pytest-xdist/) is great but it's not suitable for all use cases.
For example, some test suites may be fragile considering the order in which the tests are executed.
This is of course a fundamental problem in the suite itself but sometimes it's not worth the effort to refactor, especially if the suite is huge (and smells a bit like legacy).
Additionally, `pytest-split` may be a better fit in some use cases considering distributed execution.
## Installation
```sh
pip install pytest-split
```
## Usage
First we have to store test durations from a complete test suite run.
This produces .test_durations file which should be stored in the repo in order to have it available during future test runs.
The file path is configurable via `--durations-path` CLI option.
```sh
pytest --store-durations
```
Then we can have as many splits as we want:
```sh
pytest --splits 3 --group 1
pytest --splits 3 --group 2
pytest --splits 3 --group 3
```
Time goes by, new tests are added and old ones are removed/renamed during development. No worries!
`pytest-split` assumes average test execution time (calculated based on the stored information) for every test which does not have duration information stored.
Thus, there's no need to store durations after changing the test suite.
However, when there are major changes in the suite compared to what's stored in .test_durations, it's recommended to update the duration information with `--store-durations` to ensure that the splitting is in balance.
The splitting algorithm can be controlled with the `--splitting-algorithm` CLI option and defaults to `duration_based_chunks`. For more information about the different algorithms and their tradeoffs, please see the section below.
### CLI commands
#### slowest-tests
Lists the slowest tests based on the information stored in the test durations file. See `slowest-tests --help` for more
information.
## Interactions with other pytest plugins
* [`pytest-random-order`](https://github.com/jbasko/pytest-random-order) and [`pytest-randomly`](https://github.com/pytest-dev/pytest-randomly):
⚠️ `pytest-split` running with the `duration_based_chunks` algorithm is **incompatible** with test-order-randomization plugins.
Test selection in the groups happens after randomization, potentially causing some tests to be selected in several groups and others not at all.
Instead, a global random seed needs to be computed before running the tests (for example using `$RANDOM` from the shell) and that single seed then needs to be used for all groups by setting the `--random-order-seed` option.
* [`nbval`](https://github.com/computationalmodelling/nbval): `pytest-split` could, in principle, break up a single IPython Notebook into different test groups. This most likely causes broken up pieces to fail (for the very least, package `import`s are usually done at Cell 1, and so, any broken up piece that doesn't contain Cell 1 will certainly fail). To avoid this, after splitting step is done, test groups are reorganized based on a simple algorithm illustrated in the following cartoon:
where the letters (A to E) refer to individual IPython Notebooks, and the numbers refer to the corresponding cell number.
## Splitting algorithms
The plugin supports multiple algorithms to split tests into groups.
Each algorithm makes different tradeoffs, but generally `least_duration` should give more balanced groups.
| Algorithm | Maintains Absolute Order | Maintains Relative Order | Split Quality | Works with random ordering |
|----------------|--------------------------|--------------------------|---------------|----------------------------|
| duration_based_chunks | ✅ | ✅ | Good | ❌ |
| least_duration | ❌ | ✅ | Better | ✅ |
Explanation of the terms in the table:
* Absolute Order: whether each group contains all tests between first and last element in the same order as the original list of tests
* Relative Order: whether each test in each group has the same relative order to its neighbours in the group as in the original list of tests
* Works with random ordering: whether the algorithm works with test-shuffling tools such as [`pytest-randomly`](https://github.com/pytest-dev/pytest-randomly)
The `duration_based_chunks` algorithm aims to find optimal boundaries for the list of tests and every test group contains all tests between the start and end boundary.
The `least_duration` algorithm walks the list of tests and assigns each test to the group with the smallest current duration.
[**Demo with GitHub Actions**](https://github.com/jerry-git/pytest-split-gh-actions-demo)
## Development
* Clone this repository
* Requirements:
* [Poetry](https://python-poetry.org/)
* Python 3.8+
* Create a virtual environment and install the dependencies
```sh
poetry install
```
* Activate the virtual environment
```sh
poetry shell
```
### Testing
```sh
pytest
```
### Documentation
The documentation is automatically generated from the content of the [docs directory](https://github.com/jerry-git/pytest-split/tree/master/docs) and from the docstrings
of the public signatures of the source code. The documentation is updated and published as a [Github Pages page](https://pages.github.com/) automatically as part each release.
### Releasing
Trigger the [Draft release workflow](https://github.com/jerry-git/pytest-split/actions/workflows/draft_release.yml)
(press _Run workflow_). This will update the changelog & version and create a GitHub release which is in _Draft_ state.
Find the draft release from the
[GitHub releases](https://github.com/jerry-git/pytest-split/releases) and publish it. When
a release is published, it'll trigger [release](https://github.com/jerry-git/pytest-split/blob/master/.github/workflows/release.yml) workflow which creates PyPI
release and deploys updated documentation.
### Pre-commit
Pre-commit hooks run all the auto-formatting (`ruff format`), linters (e.g. `ruff` and `mypy`), and other quality
checks to make sure the changeset is in good shape before a commit/push happens.
You can install the hooks with (runs for each commit):
```sh
pre-commit install
```
Or if you want them to run only for each push:
```sh
pre-commit install -t pre-push
```
Or if you want e.g. want to run all checks manually for all files:
```sh
pre-commit run --all-files
```
---
This project was generated using the [wolt-python-package-cookiecutter](https://github.com/woltapp/wolt-python-package-cookiecutter) template.
Raw data
{
"_id": null,
"home_page": "https://jerry-git.github.io/pytest-split",
"name": "pytest-split",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.8.1",
"maintainer_email": null,
"keywords": "pytest, plugin, split, tests",
"author": "Jerry Pussinen",
"author_email": "jerry.pussinen@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/5e/a9/7b8327a605a48d5094b331de10e01f1f1aec0cb8c5790b5e4d98ac327f05/pytest_split-0.9.0.tar.gz",
"platform": null,
"description": "# pytest-split\n\n[](https://pypi.python.org/pypi/pytest-split/)\n[](https://pypi.python.org/pypi/pytest-split/)\n[](https://pypi.python.org/pypi/pytest-split/)\n[](https://github.com/woltapp/wolt-python-package-cookiecutter)\n\n\n---\n\n**Documentation**: [https://jerry-git.github.io/pytest-split](https://jerry-git.github.io/pytest-split)\n\n**Source Code**: [https://github.com/jerry-git/pytest-split](https://github.com/jerry-git/pytest-split)\n\n**PyPI**: [https://pypi.org/project/pytest-split/](https://pypi.org/project/pytest-split/)\n\n---\n\nPytest plugin which splits the test suite to equally sized \"sub suites\" based on test execution time.\n\n## Motivation\n* Splitting the test suite is a prerequisite for parallelization (who does not want faster CI builds?). It's valuable to have sub suites which execution time is around the same.\n* [`pytest-test-groups`](https://pypi.org/project/pytest-test-groups/) is great but it does not take into account the execution time of sub suites which can lead to notably unbalanced execution times between the sub suites.\n* [`pytest-xdist`](https://pypi.org/project/pytest-xdist/) is great but it's not suitable for all use cases.\nFor example, some test suites may be fragile considering the order in which the tests are executed.\nThis is of course a fundamental problem in the suite itself but sometimes it's not worth the effort to refactor, especially if the suite is huge (and smells a bit like legacy).\nAdditionally, `pytest-split` may be a better fit in some use cases considering distributed execution.\n\n## Installation\n```sh\npip install pytest-split\n```\n\n## Usage\nFirst we have to store test durations from a complete test suite run.\nThis produces .test_durations file which should be stored in the repo in order to have it available during future test runs.\nThe file path is configurable via `--durations-path` CLI option.\n```sh\npytest --store-durations\n```\n\nThen we can have as many splits as we want:\n```sh\npytest --splits 3 --group 1\npytest --splits 3 --group 2\npytest --splits 3 --group 3\n```\n\nTime goes by, new tests are added and old ones are removed/renamed during development. No worries!\n`pytest-split` assumes average test execution time (calculated based on the stored information) for every test which does not have duration information stored.\nThus, there's no need to store durations after changing the test suite.\nHowever, when there are major changes in the suite compared to what's stored in .test_durations, it's recommended to update the duration information with `--store-durations` to ensure that the splitting is in balance.\n\nThe splitting algorithm can be controlled with the `--splitting-algorithm` CLI option and defaults to `duration_based_chunks`. For more information about the different algorithms and their tradeoffs, please see the section below.\n\n### CLI commands\n#### slowest-tests\nLists the slowest tests based on the information stored in the test durations file. See `slowest-tests --help` for more\n information.\n\n## Interactions with other pytest plugins\n* [`pytest-random-order`](https://github.com/jbasko/pytest-random-order) and [`pytest-randomly`](https://github.com/pytest-dev/pytest-randomly):\n \u26a0\ufe0f `pytest-split` running with the `duration_based_chunks` algorithm is **incompatible** with test-order-randomization plugins.\n Test selection in the groups happens after randomization, potentially causing some tests to be selected in several groups and others not at all.\n Instead, a global random seed needs to be computed before running the tests (for example using `$RANDOM` from the shell) and that single seed then needs to be used for all groups by setting the `--random-order-seed` option.\n\n* [`nbval`](https://github.com/computationalmodelling/nbval): `pytest-split` could, in principle, break up a single IPython Notebook into different test groups. This most likely causes broken up pieces to fail (for the very least, package `import`s are usually done at Cell 1, and so, any broken up piece that doesn't contain Cell 1 will certainly fail). To avoid this, after splitting step is done, test groups are reorganized based on a simple algorithm illustrated in the following cartoon:\n\n\n\nwhere the letters (A to E) refer to individual IPython Notebooks, and the numbers refer to the corresponding cell number.\n\n## Splitting algorithms\nThe plugin supports multiple algorithms to split tests into groups.\nEach algorithm makes different tradeoffs, but generally `least_duration` should give more balanced groups.\n\n| Algorithm | Maintains Absolute Order | Maintains Relative Order | Split Quality | Works with random ordering |\n|----------------|--------------------------|--------------------------|---------------|----------------------------|\n| duration_based_chunks | \u2705 | \u2705 | Good | \u274c |\n| least_duration | \u274c | \u2705 | Better | \u2705 |\n\nExplanation of the terms in the table:\n* Absolute Order: whether each group contains all tests between first and last element in the same order as the original list of tests\n* Relative Order: whether each test in each group has the same relative order to its neighbours in the group as in the original list of tests\n* Works with random ordering: whether the algorithm works with test-shuffling tools such as [`pytest-randomly`](https://github.com/pytest-dev/pytest-randomly)\n\nThe `duration_based_chunks` algorithm aims to find optimal boundaries for the list of tests and every test group contains all tests between the start and end boundary.\nThe `least_duration` algorithm walks the list of tests and assigns each test to the group with the smallest current duration.\n\n\n[**Demo with GitHub Actions**](https://github.com/jerry-git/pytest-split-gh-actions-demo)\n\n\n## Development\n\n* Clone this repository\n* Requirements:\n * [Poetry](https://python-poetry.org/)\n * Python 3.8+\n* Create a virtual environment and install the dependencies\n\n```sh\npoetry install\n```\n\n* Activate the virtual environment\n\n```sh\npoetry shell\n```\n\n### Testing\n\n```sh\npytest\n```\n\n### Documentation\n\nThe documentation is automatically generated from the content of the [docs directory](https://github.com/jerry-git/pytest-split/tree/master/docs) and from the docstrings\n of the public signatures of the source code. The documentation is updated and published as a [Github Pages page](https://pages.github.com/) automatically as part each release.\n\n### Releasing\n\nTrigger the [Draft release workflow](https://github.com/jerry-git/pytest-split/actions/workflows/draft_release.yml)\n(press _Run workflow_). This will update the changelog & version and create a GitHub release which is in _Draft_ state.\n\nFind the draft release from the\n[GitHub releases](https://github.com/jerry-git/pytest-split/releases) and publish it. When\n a release is published, it'll trigger [release](https://github.com/jerry-git/pytest-split/blob/master/.github/workflows/release.yml) workflow which creates PyPI\n release and deploys updated documentation.\n\n### Pre-commit\n\nPre-commit hooks run all the auto-formatting (`ruff format`), linters (e.g. `ruff` and `mypy`), and other quality\n checks to make sure the changeset is in good shape before a commit/push happens.\n\nYou can install the hooks with (runs for each commit):\n\n```sh\npre-commit install\n```\n\nOr if you want them to run only for each push:\n\n```sh\npre-commit install -t pre-push\n```\n\nOr if you want e.g. want to run all checks manually for all files:\n\n```sh\npre-commit run --all-files\n```\n\n---\n\nThis project was generated using the [wolt-python-package-cookiecutter](https://github.com/woltapp/wolt-python-package-cookiecutter) template.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Pytest plugin which splits the test suite to equally sized sub suites based on test execution time.",
"version": "0.9.0",
"project_urls": {
"Documentation": "https://jerry-git.github.io/pytest-split",
"Homepage": "https://jerry-git.github.io/pytest-split",
"Repository": "https://github.com/jerry-git/pytest-split"
},
"split_keywords": [
"pytest",
" plugin",
" split",
" tests"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "6b678cff7bf04d78ac7fbb88b0985061347943dc3cbeafada27b4accb4527579",
"md5": "6ae208b8aaf16a63c18323e2c38a1365",
"sha256": "9e197df601828d76a1ab615158d9c6253ec9f96e46c1d3ea27187aa5ac0ef9de"
},
"downloads": -1,
"filename": "pytest_split-0.9.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "6ae208b8aaf16a63c18323e2c38a1365",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.8.1",
"size": 11790,
"upload_time": "2024-06-19T12:17:48",
"upload_time_iso_8601": "2024-06-19T12:17:48.888849Z",
"url": "https://files.pythonhosted.org/packages/6b/67/8cff7bf04d78ac7fbb88b0985061347943dc3cbeafada27b4accb4527579/pytest_split-0.9.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "5ea97b8327a605a48d5094b331de10e01f1f1aec0cb8c5790b5e4d98ac327f05",
"md5": "cbcbe2fd0b48b82861d2347a06553dc9",
"sha256": "ca52527e4d9024f6ec3aba723527bd276d12096024999b1f5b8445a38da1e81c"
},
"downloads": -1,
"filename": "pytest_split-0.9.0.tar.gz",
"has_sig": false,
"md5_digest": "cbcbe2fd0b48b82861d2347a06553dc9",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.8.1",
"size": 13599,
"upload_time": "2024-06-19T12:17:53",
"upload_time_iso_8601": "2024-06-19T12:17:53.969303Z",
"url": "https://files.pythonhosted.org/packages/5e/a9/7b8327a605a48d5094b331de10e01f1f1aec0cb8c5790b5e4d98ac327f05/pytest_split-0.9.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-06-19 12:17:53",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "jerry-git",
"github_project": "pytest-split",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "pytest-split"
}
JSON
