===================================
pytest-random-order
===================================
.. image:: https://img.shields.io/badge/python-3.7%2C%203.8%2C%203.9%2C%203.10-blue.svg
:target: https://github.com/jbasko/pytest-random-order
**pytest-random-order** is a `pytest <http://pytest.org>`_ plugin that randomises the order of tests.
This can be useful to detect a test that passes just because it happens to run after an unrelated test that
leaves the system in a *favourable* state.
The plugin allows user to control the level of randomness they want to introduce and to disable
reordering on subsets of tests. Tests can be rerun in a specific order by passing a seed value reported
in a previous test run.
.. image:: https://raw.githubusercontent.com/jbasko/pytest-random-order/master/docs/pytest-random-order-design.png
-----------
Quick Start
-----------
Installation:
::
$ pip install pytest-random-order
From v1.0.0 onwards, **this plugin no longer randomises tests by default**. To enable randomisation, you have to run
pytest in one of the following ways:
::
pytest --random-order
pytest --random-order-bucket=<bucket_type>
pytest --random-order-seed=<seed>
If you want to always randomise the order of tests, configure pytest. There are many ways to do it,
my favourite one is to add ``addopts = --random-order`` in your project-specific configuration file
under the pytest options (usually ``[pytest]`` or ``[tool:pytest]`` section).
Alternatively, you can set environment variable ``PYTEST_ADDOPTS``:
::
export PYTEST_ADDOPTS="--random-order"
To randomise the order of tests within modules and shuffle the order of
test modules (which is the default behaviour of the plugin), run pytest as follows:
::
$ pytest --random-order
To change the scope of re-ordering, run pytest with ``--random-order-bucket=<bucket-type>`` option
where ``<bucket-type>`` can be ``class``, ``module``, ``package``, ``global``:
::
$ pytest -v --random-order-bucket=package
To disable reordering of tests in a module or class, use pytest marker notation:
::
pytestmark = pytest.mark.random_order(disabled=True)
To rerun tests in a particular order:
::
$ pytest -v --random-order-seed=<seed>
All runs in which the randomisation is enabled report seed so if you encounter a specific ordering of tests
that causes problems you can look up the value in the test report and repeat the run with the above command.
::
platform darwin -- Python 3.5.6, pytest-3.9.1, py-1.7.0, pluggy-0.8.0
Using --random-order-bucket=module
Using --random-order-seed=383013
------
Design
------
.. image:: https://raw.githubusercontent.com/jbasko/pytest-random-order/master/docs/pytest-random-order-design.png
The plugin groups tests in buckets, shuffles them within buckets and then shuffles the buckets.
Given the test suite above, here are two of a few possible generated orders of tests:
.. image:: https://raw.githubusercontent.com/jbasko/pytest-random-order/master/docs/pytest-random-order-example1.png
.. image:: https://raw.githubusercontent.com/jbasko/pytest-random-order/master/docs/pytest-random-order-example2.png
You can choose from a few types of buckets:
class
Tests will be shuffled within a class and classes will be shuffled,
but tests from one class will never have tests from other classes or modules run in-between them.
module
Same as above at module level. This is the setting applied if you run pytest with just ``--random-order`` flag
or ``--random-order-seed=<seed>``.
package
Same as above at package level. Note that modules (and hence tests inside those modules) that
belong to package ``x.y.z`` do not belong to package ``x.y``, so they will fall in different buckets
when randomising with ``package`` bucket type.
parent
If you are using custom test items which don't belong to any module, you can use this to
limit reordering of test items to within the ``parent`` to which they belong. For normal test
functions the parent is the module in which they are declared.
grandparent
Similar to *parent* above, but use the parent of the parent of the test item as the bucket key instead.
global
All tests fall in the same bucket, full randomness, tests probably take longer to run.
none (deprecated)
Disable shuffling. *Deprecated since 1.0.4 because this plugin no longer shuffles tests by default
so there is nothing to disable.*
If you have three buckets of tests ``A``, ``B``, and ``C`` with three tests ``1`` and ``2``, and ``3`` in each of them,
then one of many potential orderings that non-global randomisation can produce could be:
::
c2, c1, c3, a3, a1, a2, b3, b2, b1
As you can see, all C tests are executed "next" to each other and so are tests in buckets A and B.
Tests from any bucket X are guaranteed to not be interspersed with tests from another bucket Y.
For example, if you choose bucket type ``module`` then bucket X contains all tests that are in this module.
By default, when randomisation is enabled, your tests will be randomised at ``module`` level which means that
tests within a single module X will be executed in no particular order, but tests from
other modules will not be mixed in between tests of module X.
The randomised reordering can be disabled per module or per class irrespective of the chosen bucket type.
--------------
Usage and Tips
--------------
Bucket Type Choice
++++++++++++++++++
It is best to start with smallest bucket type (``class`` or ``module`` depending on whether you have class-based tests),
and switch to a larger bucket type when you are sure your tests handle that.
If your tests rely on fixtures that are module or session-scoped, more randomised order of tests will mean slower tests.
You probably don't want to randomise at ``global`` or ``package`` level while you are coding and need a quick confirmation
that nothing big is broken.
Disable Shuffling in Module or Class
++++++++++++++++++++++++++++++++++++
You can disable shuffling of tests within a single module or class by marking the module or class
with ``random_order`` marker and passing ``disabled=True`` to it:
::
pytestmark = pytest.mark.random_order(disabled=True)
def test_number_one():
assert True
def test_number_two():
assert True
::
class MyTest(TestCase):
pytestmark = pytest.mark.random_order(disabled=True)
def test_number_one(self):
self.assertTrue(True)
No matter what will be the bucket type for the test run, ``test_number_one`` will always run
before ``test_number_two``.
Rerun Tests in the Same Order (Same Seed)
+++++++++++++++++++++++++++++++++++++++++
If you discover a failing test because you reordered tests, you will probably want to be able to rerun the tests
in the same failing order. To allow reproducing test order, the plugin reports the seed value it used with pseudo random number
generator:
::
============================= test session starts ==============================
..
Using --random-order-bucket=module
Using --random-order-seed=24775
...
You can now use the ``--random-order-seed=...`` bit as an argument to the next run to produce the same order:
::
$ pytest -v --random-order-seed=24775
Run Last Failed Tests First
+++++++++++++++++++++++++++
Since v0.8.0 pytest cache plugin's ``--failed-first`` flag is supported -- tests that failed in the last run
will be run before tests that passed irrespective of shuffling bucket type.
Disable the Plugin
+++++++++++++++++++++++++++++++++++
If the plugin misbehaves or you just want to assure yourself that it is not the plugin making your tests fail or
pass undeservedly, you can disable it:
::
$ pytest -p no:random_order
Note that randomisation is disabled by default. By passing ``-p no:random_order`` you are stopping the plugin
from being registered so its hooks won't be registered and its command line options won't appear in ``--help``.
--------------
Changelog
--------------
v1.1.1 (2024-01-20)
+++++++++++++++++++
* Fixes #54 - ``AttributeError`` when cacheprovider plugin disabled. Thanks @jhanm12
v1.1.0 (2022-12-03)
+++++++++++++++++++
* Fixes xdist support (thanks @matejsp)
v1.0.4 (2018-11-30)
+++++++++++++++++++
* Fixes issues with doctests reported in #36 - ``class``, ``package`` and ``module`` didn't work
because ``DoctestItem`` doesn't have ``cls`` or ``module`` attributes. Thanks @tobywf.
* Deprecate ``none`` bucket type.
* With tox, run tests of pytest-random-order with both pytest 3 and 4.
v1.0.3 (2018-11-16)
+++++++++++++++++++
* Fixes compatibility issues with pytest 4.0.0, works with pytest 3.0+ as before.
* Tests included in the source distribution.
v1.0.0 (2018-10-20)
+++++++++++++++++++
* Plugin no longer alters the test order by default. You will have to either 1) pass ``--random-order``,
or ``--random-order-bucket=<bucket>``, or ``--random-order-seed=<seed>``, or
2) edit your pytest configuration file and add one of these options
there under ``addopts``, or 3) specify these flags in environment variable ``PYTEST_ADDOPTS``.
* Python 3.5+ is required. If you want to use this plugin with Python 2.7, use v0.8.0 which is stable and fine
if you are happy with it randomising the test order by default.
* The name under which the plugin registers itself is changed from ``random-order`` (hyphen) to ``random_order``
(underscore). This addresses the issue of consistency when disabling or enabling this plugin via the standard
``-p`` flag. Previously, the plugin could be disabled by passing ``-p no:random-order`` yet re-enabled
only by passing ``-p pytest_random_order.plugin``. Now they are ``-p no:random_order``
to disable and ``-p random_order.plugin`` to enable (The ``.plugin`` bit, I think, is required because
pytest probably thinks it's an unrelated thing to ``random_order`` and import it, yet without it it's the
same thing so doesn't import it).
v0.8.0
++++++
* pytest cache plugin's ``--failed-first`` works now.
-------
Credits
-------
* The shuffle icon in the diagram is by artist `Daniele De Santis`_ and it was found on
`iconarchive`_.
* The diagram is drawn with `sketchboard.io`_
.. _Daniele De Santis: https://www.danieledesantis.net/
.. _iconarchive: http://www.iconarchive.com/artist/danieledesantis.html
.. _sketchboard.io: https://sketchboard.io/
Raw data
{
"_id": null,
"home_page": "https://github.com/jbasko/pytest-random-order",
"name": "pytest-random-order",
"maintainer": "Jazeps Basko",
"docs_url": null,
"requires_python": ">=3.5.0",
"maintainer_email": "jazeps.basko@gmail.com",
"keywords": "pytest random test order shuffle",
"author": "Jazeps Basko",
"author_email": "jazeps.basko@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/93/e5/89654b4354b10e89969a74130f391b017dbdc113ce27f0e8ff9fa23e44e1/pytest-random-order-1.1.1.tar.gz",
"platform": null,
"description": "===================================\npytest-random-order\n===================================\n\n.. image:: https://img.shields.io/badge/python-3.7%2C%203.8%2C%203.9%2C%203.10-blue.svg\n :target: https://github.com/jbasko/pytest-random-order\n\n**pytest-random-order** is a `pytest <http://pytest.org>`_ plugin that randomises the order of tests.\nThis can be useful to detect a test that passes just because it happens to run after an unrelated test that\nleaves the system in a *favourable* state.\n\nThe plugin allows user to control the level of randomness they want to introduce and to disable\nreordering on subsets of tests. Tests can be rerun in a specific order by passing a seed value reported\nin a previous test run.\n\n.. image:: https://raw.githubusercontent.com/jbasko/pytest-random-order/master/docs/pytest-random-order-design.png\n\n-----------\nQuick Start\n-----------\n\nInstallation:\n\n::\n\n $ pip install pytest-random-order\n\nFrom v1.0.0 onwards, **this plugin no longer randomises tests by default**. To enable randomisation, you have to run\npytest in one of the following ways:\n\n::\n\n pytest --random-order\n pytest --random-order-bucket=<bucket_type>\n pytest --random-order-seed=<seed>\n\nIf you want to always randomise the order of tests, configure pytest. There are many ways to do it,\nmy favourite one is to add ``addopts = --random-order`` in your project-specific configuration file\nunder the pytest options (usually ``[pytest]`` or ``[tool:pytest]`` section).\n\nAlternatively, you can set environment variable ``PYTEST_ADDOPTS``:\n\n::\n\n export PYTEST_ADDOPTS=\"--random-order\"\n\n\nTo randomise the order of tests within modules and shuffle the order of\ntest modules (which is the default behaviour of the plugin), run pytest as follows:\n\n::\n\n $ pytest --random-order\n\nTo change the scope of re-ordering, run pytest with ``--random-order-bucket=<bucket-type>`` option\nwhere ``<bucket-type>`` can be ``class``, ``module``, ``package``, ``global``:\n\n::\n\n $ pytest -v --random-order-bucket=package\n\nTo disable reordering of tests in a module or class, use pytest marker notation:\n\n::\n\n pytestmark = pytest.mark.random_order(disabled=True)\n\nTo rerun tests in a particular order:\n\n::\n\n $ pytest -v --random-order-seed=<seed>\n\nAll runs in which the randomisation is enabled report seed so if you encounter a specific ordering of tests\nthat causes problems you can look up the value in the test report and repeat the run with the above command.\n\n::\n\n platform darwin -- Python 3.5.6, pytest-3.9.1, py-1.7.0, pluggy-0.8.0\n Using --random-order-bucket=module\n Using --random-order-seed=383013\n\n------\nDesign\n------\n\n.. image:: https://raw.githubusercontent.com/jbasko/pytest-random-order/master/docs/pytest-random-order-design.png\n\nThe plugin groups tests in buckets, shuffles them within buckets and then shuffles the buckets.\n\nGiven the test suite above, here are two of a few possible generated orders of tests:\n\n.. image:: https://raw.githubusercontent.com/jbasko/pytest-random-order/master/docs/pytest-random-order-example1.png\n\n.. image:: https://raw.githubusercontent.com/jbasko/pytest-random-order/master/docs/pytest-random-order-example2.png\n\nYou can choose from a few types of buckets:\n\nclass\n Tests will be shuffled within a class and classes will be shuffled,\n but tests from one class will never have tests from other classes or modules run in-between them.\n\nmodule\n Same as above at module level. This is the setting applied if you run pytest with just ``--random-order`` flag\n or ``--random-order-seed=<seed>``.\n\npackage\n Same as above at package level. Note that modules (and hence tests inside those modules) that\n belong to package ``x.y.z`` do not belong to package ``x.y``, so they will fall in different buckets\n when randomising with ``package`` bucket type.\n\nparent\n If you are using custom test items which don't belong to any module, you can use this to\n limit reordering of test items to within the ``parent`` to which they belong. For normal test\n functions the parent is the module in which they are declared.\n\ngrandparent\n Similar to *parent* above, but use the parent of the parent of the test item as the bucket key instead.\n\nglobal\n All tests fall in the same bucket, full randomness, tests probably take longer to run.\n\nnone (deprecated)\n Disable shuffling. *Deprecated since 1.0.4 because this plugin no longer shuffles tests by default\n so there is nothing to disable.*\n\n\nIf you have three buckets of tests ``A``, ``B``, and ``C`` with three tests ``1`` and ``2``, and ``3`` in each of them,\nthen one of many potential orderings that non-global randomisation can produce could be:\n\n::\n\n c2, c1, c3, a3, a1, a2, b3, b2, b1\n\nAs you can see, all C tests are executed \"next\" to each other and so are tests in buckets A and B.\nTests from any bucket X are guaranteed to not be interspersed with tests from another bucket Y.\nFor example, if you choose bucket type ``module`` then bucket X contains all tests that are in this module.\n\nBy default, when randomisation is enabled, your tests will be randomised at ``module`` level which means that\ntests within a single module X will be executed in no particular order, but tests from\nother modules will not be mixed in between tests of module X.\n\nThe randomised reordering can be disabled per module or per class irrespective of the chosen bucket type.\n\n--------------\nUsage and Tips\n--------------\n\nBucket Type Choice\n++++++++++++++++++\n\nIt is best to start with smallest bucket type (``class`` or ``module`` depending on whether you have class-based tests),\nand switch to a larger bucket type when you are sure your tests handle that.\n\nIf your tests rely on fixtures that are module or session-scoped, more randomised order of tests will mean slower tests.\nYou probably don't want to randomise at ``global`` or ``package`` level while you are coding and need a quick confirmation\nthat nothing big is broken.\n\nDisable Shuffling in Module or Class\n++++++++++++++++++++++++++++++++++++\n\nYou can disable shuffling of tests within a single module or class by marking the module or class\nwith ``random_order`` marker and passing ``disabled=True`` to it:\n\n::\n\n pytestmark = pytest.mark.random_order(disabled=True)\n\n def test_number_one():\n assert True\n\n def test_number_two():\n assert True\n\n::\n\n class MyTest(TestCase):\n pytestmark = pytest.mark.random_order(disabled=True)\n\n def test_number_one(self):\n self.assertTrue(True)\n\n\nNo matter what will be the bucket type for the test run, ``test_number_one`` will always run\nbefore ``test_number_two``.\n\n\nRerun Tests in the Same Order (Same Seed)\n+++++++++++++++++++++++++++++++++++++++++\n\nIf you discover a failing test because you reordered tests, you will probably want to be able to rerun the tests\nin the same failing order. To allow reproducing test order, the plugin reports the seed value it used with pseudo random number\ngenerator:\n\n::\n\n ============================= test session starts ==============================\n ..\n Using --random-order-bucket=module\n Using --random-order-seed=24775\n ...\n\nYou can now use the ``--random-order-seed=...`` bit as an argument to the next run to produce the same order:\n\n::\n\n $ pytest -v --random-order-seed=24775\n\n\nRun Last Failed Tests First\n+++++++++++++++++++++++++++\n\nSince v0.8.0 pytest cache plugin's ``--failed-first`` flag is supported -- tests that failed in the last run\nwill be run before tests that passed irrespective of shuffling bucket type.\n\n\nDisable the Plugin\n+++++++++++++++++++++++++++++++++++\n\nIf the plugin misbehaves or you just want to assure yourself that it is not the plugin making your tests fail or\npass undeservedly, you can disable it:\n\n::\n\n $ pytest -p no:random_order\n\nNote that randomisation is disabled by default. By passing ``-p no:random_order`` you are stopping the plugin\nfrom being registered so its hooks won't be registered and its command line options won't appear in ``--help``.\n\n--------------\nChangelog\n--------------\n\nv1.1.1 (2024-01-20)\n+++++++++++++++++++\n\n * Fixes #54 - ``AttributeError`` when cacheprovider plugin disabled. Thanks @jhanm12\n\n\nv1.1.0 (2022-12-03)\n+++++++++++++++++++\n\n * Fixes xdist support (thanks @matejsp)\n\n\nv1.0.4 (2018-11-30)\n+++++++++++++++++++\n\n* Fixes issues with doctests reported in #36 - ``class``, ``package`` and ``module`` didn't work\n because ``DoctestItem`` doesn't have ``cls`` or ``module`` attributes. Thanks @tobywf.\n* Deprecate ``none`` bucket type.\n* With tox, run tests of pytest-random-order with both pytest 3 and 4.\n\nv1.0.3 (2018-11-16)\n+++++++++++++++++++\n\n* Fixes compatibility issues with pytest 4.0.0, works with pytest 3.0+ as before.\n* Tests included in the source distribution.\n\nv1.0.0 (2018-10-20)\n+++++++++++++++++++\n\n* Plugin no longer alters the test order by default. You will have to either 1) pass ``--random-order``,\n or ``--random-order-bucket=<bucket>``, or ``--random-order-seed=<seed>``, or\n 2) edit your pytest configuration file and add one of these options\n there under ``addopts``, or 3) specify these flags in environment variable ``PYTEST_ADDOPTS``.\n* Python 3.5+ is required. If you want to use this plugin with Python 2.7, use v0.8.0 which is stable and fine\n if you are happy with it randomising the test order by default.\n* The name under which the plugin registers itself is changed from ``random-order`` (hyphen) to ``random_order``\n (underscore). This addresses the issue of consistency when disabling or enabling this plugin via the standard\n ``-p`` flag. Previously, the plugin could be disabled by passing ``-p no:random-order`` yet re-enabled\n only by passing ``-p pytest_random_order.plugin``. Now they are ``-p no:random_order``\n to disable and ``-p random_order.plugin`` to enable (The ``.plugin`` bit, I think, is required because\n pytest probably thinks it's an unrelated thing to ``random_order`` and import it, yet without it it's the\n same thing so doesn't import it).\n\n\nv0.8.0\n++++++\n\n* pytest cache plugin's ``--failed-first`` works now.\n\n-------\nCredits\n-------\n\n* The shuffle icon in the diagram is by artist `Daniele De Santis`_ and it was found on\n `iconarchive`_.\n\n* The diagram is drawn with `sketchboard.io`_\n\n.. _Daniele De Santis: https://www.danieledesantis.net/\n.. _iconarchive: http://www.iconarchive.com/artist/danieledesantis.html\n.. _sketchboard.io: https://sketchboard.io/\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Randomise the order in which pytest tests are run with some control over the randomness",
"version": "1.1.1",
"project_urls": {
"Homepage": "https://github.com/jbasko/pytest-random-order"
},
"split_keywords": [
"pytest",
"random",
"test",
"order",
"shuffle"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "9102944cf846bcd6027a1805c69fec90581f916e99ccafcbe409ae6c76833255",
"md5": "9fdd933e17cc1735d58bb3e2e9486b2c",
"sha256": "882727a8b597ecd06ede28654ffeb8a6d511a1e4abe1054cca7982f2e42008cd"
},
"downloads": -1,
"filename": "pytest_random_order-1.1.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "9fdd933e17cc1735d58bb3e2e9486b2c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.5.0",
"size": 11521,
"upload_time": "2024-01-20T09:25:05",
"upload_time_iso_8601": "2024-01-20T09:25:05.098129Z",
"url": "https://files.pythonhosted.org/packages/91/02/944cf846bcd6027a1805c69fec90581f916e99ccafcbe409ae6c76833255/pytest_random_order-1.1.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "93e589654b4354b10e89969a74130f391b017dbdc113ce27f0e8ff9fa23e44e1",
"md5": "6bd9833def40268d17b57890b93ac594",
"sha256": "4472d7d34f1f1c5f3a359c4ffc5c13ed065232f31eca19c8844c1ab406e79080"
},
"downloads": -1,
"filename": "pytest-random-order-1.1.1.tar.gz",
"has_sig": false,
"md5_digest": "6bd9833def40268d17b57890b93ac594",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.5.0",
"size": 14626,
"upload_time": "2024-01-20T09:25:07",
"upload_time_iso_8601": "2024-01-20T09:25:07.218372Z",
"url": "https://files.pythonhosted.org/packages/93/e5/89654b4354b10e89969a74130f391b017dbdc113ce27f0e8ff9fa23e44e1/pytest-random-order-1.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-20 09:25:07",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "jbasko",
"github_project": "pytest-random-order",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "coverage",
"specs": []
},
{
"name": "py",
"specs": []
},
{
"name": "pytest",
"specs": []
},
{
"name": "pytest-xdist",
"specs": []
},
{
"name": "sphinx",
"specs": []
}
],
"lcname": "pytest-random-order"
}
JSON
