======================
pytest-coveragemarkers
======================
.. image:: https://img.shields.io/badge/security-bandit-yellow.svg
:target: https://github.com/PyCQA/bandit
:alt: Security Status
.. image:: https://img.shields.io/badge/code%20style-black-000000.svg
:target: https://github.com/psf/black
Using pytest markers to track functional coverage and filtering of tests
----
This `pytest`_ plugin was generated with `Cookiecutter`_ along with `@hackebrot`_'s `cookiecutter-pytest-plugin`_ template.
Features
--------
* Definition of CoverageMarkers© in YAML format
* Support for applying CoverageMarkers© to tests
* Filtering of tests based on CoverageMarkers©
* Inclusion of CoverageMarkers© in JSON report
Installation
------------
You can install "pytest-coveragemarkers" from `PyPI`_::
$ pip install pytest-coveragemarkers
# or
$ poetry add pytest-coveragemarkers
Usage
-----
Step 1: Define your coverage markers yaml.
Using the format:
.. code-block:: yaml
markers:
- name: <marker_name>
allowed:
- <marker_value_1>
- <marker_value_2>
- name: <marker2_name>
allowed:
- <marker2_value_1>
- <marker2_value_2>
Then decorate your tests with them
.. code-block:: python
import pytest
@pytest.mark.marker_name(['value1', 'value2'])
@pytest.mark.marker2_name(['value1', 'value2'])
def test_x():
...
@pytest.mark.marker2_name(['value1', 'value2'])
def test_y():
...
Then when the tests are executed with
.. code-block:: shell
pytest --json-report --markers-location=/full/path/to/coverage_markers.yml
Then the JSON Test Report output from the test execution contains:
.. code-block:: json
"tests": [
{
"nodeid": "...",
"metadata": {
"cov_markers": {
"marker_name": {
"value1": true,
"value2": true
},
"marker2_name": {
"value1": true,
"value2": true
}
}
}
},
...
]
This can then be used to generate test coverage details based on the coverage markers.
A nice demo will be produced to give examples of usage.
But wait there is another benefit:
We can filter tests for execution based on their coverage markers
.. code-block:: shell
pytest \
--filter='{"and": [{"eq": ["marker_name.value1", true]}]}' \
--json-report \
--markers-location=/full/path/to/coverage_markers.yml
The above command run against the tests defined above would select 'test_x' and deselect 'test_y' for execution
Other examples of filters are:
.. code-block: shell
'{"or": [{"eq": ["marker_name.value1", true]}, {"eq": ["marker_name.value2", true]}]}'
You can also supply the path to a json file containing your filter.
Use argument --filter-location or key FilterLocation in the pytest.ini file.
Coverage Marker Argument Format
-------------------------------
The arguments supplied to Coverage Markers can follow multiple formats which allows the user to define the format that best suites them.
E.g.
.. code-block:: python
import pytest
@pytest.mark.marker_1('value1') # single string argument
@pytest.mark.marker_2('value1', 'value2') # multiple string arguments
@pytest.mark.marker_3(['value1', 'value2']) # list of arguments
@pytest.mark.marker_4(('value1', 'value2')) # tuple of arguments
def test_x():
...
Testing
-------
Nox is used by this project to execute all tests.
To run a specific set of tests execute the below line::
$ poetry run nox -s <session_name>
Where session_name can be one of the following
.. list-table:: Nox Sessions
:widths: 25 75
:header-rows: 1
* - Session Name
- Session Details
* - unit_tests
- Execute all tests marked as unit
* - functional_tests
- Execute all tests marked as functional
Thought Process
---------------
* The `pytest_docs`_ talks about using markers to set metadata on tests and use the markers to select required tests for execution.
* For the markers I want to add, I also want to specify a list of values that go along with that marker.
E.g. If the marker was 'colour' then supported values may be 'Red', 'Green', 'Gold'.
* I also want the list of values validated against supported values so no unsupported values can be added.
E.g. If the marker was 'colour' then a value of 'Panda' would not be allowed.
* Then all this meta data I want to come out in the junit json report.
* Next I want to use these markers and their supported values to filter tests. For this I need a more powerful filter engine.
Documentation
-------------
To build the docs run::
poetry run mkdocs serve
License
-------
Distributed under the terms of the `MIT`_ license, "pytest-coveragemarkers" is free and open source software
Issues
------
If you encounter any problems, please `file an issue`_ along with a detailed description.
Future Changes
--------------
* Type-Hints
* Full Test Coverage
* Full Documentation
* Refactor/rewrite
* Pick one, yml or json?
.. _`Cookiecutter`: https://github.com/audreyr/cookiecutter
.. _`@hackebrot`: https://github.com/hackebrot
.. _`MIT`: http://opensource.org/licenses/MIT
.. _`BSD-3`: http://opensource.org/licenses/BSD-3-Clause
.. _`GNU GPL v3.0`: http://www.gnu.org/licenses/gpl-3.0.txt
.. _`Apache Software License 2.0`: http://www.apache.org/licenses/LICENSE-2.0
.. _`cookiecutter-pytest-plugin`: https://github.com/pytest-dev/cookiecutter-pytest-plugin
.. _`file an issue`: https://github.com/Gleams99/pytest-coveragemarkers/issues
.. _`pytest`: https://github.com/pytest-dev/pytest
.. _`nox`: https://nox.thea.codes/en/stable/
.. _`pip`: https://pypi.org/project/pip/
.. _`PyPI`: https://pypi.org/project
.. _`pytest_docs`: https://docs.pytest.org/en/7.1.x/how-to/mark.html?highlight=slow
Raw data
{
"_id": null,
"home_page": null,
"name": "pytest-coveragemarkers",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.11.0,>=3.8",
"maintainer_email": null,
"keywords": "test, pytest, markers, coverage",
"author": "Gleams API user",
"author_email": "Stephen.Swannell+ghapi@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/2e/e2/9980a189ebc43c43b43750ea23047c75e7f75564b9fbe56cbd807fd1d908/pytest_coveragemarkers-2.1.0.tar.gz",
"platform": null,
"description": "======================\npytest-coveragemarkers\n======================\n\n.. image:: https://img.shields.io/badge/security-bandit-yellow.svg\n :target: https://github.com/PyCQA/bandit\n :alt: Security Status\n\n.. image:: https://img.shields.io/badge/code%20style-black-000000.svg\n :target: https://github.com/psf/black\n\nUsing pytest markers to track functional coverage and filtering of tests\n\n----\n\nThis `pytest`_ plugin was generated with `Cookiecutter`_ along with `@hackebrot`_'s `cookiecutter-pytest-plugin`_ template.\n\n\nFeatures\n--------\n\n* Definition of CoverageMarkers\u00a9 in YAML format\n* Support for applying CoverageMarkers\u00a9 to tests\n* Filtering of tests based on CoverageMarkers\u00a9\n* Inclusion of CoverageMarkers\u00a9 in JSON report\n\n\nInstallation\n------------\n\nYou can install \"pytest-coveragemarkers\" from `PyPI`_::\n\n $ pip install pytest-coveragemarkers\n # or\n $ poetry add pytest-coveragemarkers\n\nUsage\n-----\n\nStep 1: Define your coverage markers yaml.\n\n Using the format:\n\n.. code-block:: yaml\n\n markers:\n - name: <marker_name>\n allowed:\n - <marker_value_1>\n - <marker_value_2>\n - name: <marker2_name>\n allowed:\n - <marker2_value_1>\n - <marker2_value_2>\n\nThen decorate your tests with them\n\n\n.. code-block:: python\n\n import pytest\n\n @pytest.mark.marker_name(['value1', 'value2'])\n @pytest.mark.marker2_name(['value1', 'value2'])\n def test_x():\n ...\n\n @pytest.mark.marker2_name(['value1', 'value2'])\n def test_y():\n ...\n\n\nThen when the tests are executed with\n\n.. code-block:: shell\n\n pytest --json-report --markers-location=/full/path/to/coverage_markers.yml\n\nThen the JSON Test Report output from the test execution contains:\n\n.. code-block:: json\n\n \"tests\": [\n {\n \"nodeid\": \"...\",\n \"metadata\": {\n \"cov_markers\": {\n \"marker_name\": {\n \"value1\": true,\n \"value2\": true\n },\n \"marker2_name\": {\n \"value1\": true,\n \"value2\": true\n }\n }\n }\n },\n ...\n ]\n\nThis can then be used to generate test coverage details based on the coverage markers.\nA nice demo will be produced to give examples of usage.\n\nBut wait there is another benefit:\n\nWe can filter tests for execution based on their coverage markers\n\n.. code-block:: shell\n\n pytest \\\n --filter='{\"and\": [{\"eq\": [\"marker_name.value1\", true]}]}' \\\n --json-report \\\n --markers-location=/full/path/to/coverage_markers.yml\n\nThe above command run against the tests defined above would select 'test_x' and deselect 'test_y' for execution\n\nOther examples of filters are:\n\n.. code-block: shell\n\n '{\"or\": [{\"eq\": [\"marker_name.value1\", true]}, {\"eq\": [\"marker_name.value2\", true]}]}'\n\nYou can also supply the path to a json file containing your filter.\nUse argument --filter-location or key FilterLocation in the pytest.ini file.\n\nCoverage Marker Argument Format\n-------------------------------\n\nThe arguments supplied to Coverage Markers can follow multiple formats which allows the user to define the format that best suites them.\n\nE.g.\n\n.. code-block:: python\n\n import pytest\n\n @pytest.mark.marker_1('value1') # single string argument\n @pytest.mark.marker_2('value1', 'value2') # multiple string arguments\n @pytest.mark.marker_3(['value1', 'value2']) # list of arguments\n @pytest.mark.marker_4(('value1', 'value2')) # tuple of arguments\n def test_x():\n ...\n\n\n\nTesting\n-------\n\nNox is used by this project to execute all tests.\nTo run a specific set of tests execute the below line::\n\n $ poetry run nox -s <session_name>\n\nWhere session_name can be one of the following\n\n.. list-table:: Nox Sessions\n :widths: 25 75\n :header-rows: 1\n\n * - Session Name\n - Session Details\n * - unit_tests\n - Execute all tests marked as unit\n * - functional_tests\n - Execute all tests marked as functional\n\nThought Process\n---------------\n\n* The `pytest_docs`_ talks about using markers to set metadata on tests and use the markers to select required tests for execution.\n* For the markers I want to add, I also want to specify a list of values that go along with that marker.\n E.g. If the marker was 'colour' then supported values may be 'Red', 'Green', 'Gold'.\n* I also want the list of values validated against supported values so no unsupported values can be added.\n E.g. If the marker was 'colour' then a value of 'Panda' would not be allowed.\n* Then all this meta data I want to come out in the junit json report.\n* Next I want to use these markers and their supported values to filter tests. For this I need a more powerful filter engine.\n\nDocumentation\n-------------\n\nTo build the docs run::\n\n poetry run mkdocs serve\n\n\nLicense\n-------\n\nDistributed under the terms of the `MIT`_ license, \"pytest-coveragemarkers\" is free and open source software\n\n\nIssues\n------\n\nIf you encounter any problems, please `file an issue`_ along with a detailed description.\n\n\nFuture Changes\n--------------\n\n* Type-Hints\n* Full Test Coverage\n* Full Documentation\n* Refactor/rewrite\n* Pick one, yml or json?\n\n.. _`Cookiecutter`: https://github.com/audreyr/cookiecutter\n.. _`@hackebrot`: https://github.com/hackebrot\n.. _`MIT`: http://opensource.org/licenses/MIT\n.. _`BSD-3`: http://opensource.org/licenses/BSD-3-Clause\n.. _`GNU GPL v3.0`: http://www.gnu.org/licenses/gpl-3.0.txt\n.. _`Apache Software License 2.0`: http://www.apache.org/licenses/LICENSE-2.0\n.. _`cookiecutter-pytest-plugin`: https://github.com/pytest-dev/cookiecutter-pytest-plugin\n.. _`file an issue`: https://github.com/Gleams99/pytest-coveragemarkers/issues\n.. _`pytest`: https://github.com/pytest-dev/pytest\n.. _`nox`: https://nox.thea.codes/en/stable/\n.. _`pip`: https://pypi.org/project/pip/\n.. _`PyPI`: https://pypi.org/project\n.. _`pytest_docs`: https://docs.pytest.org/en/7.1.x/how-to/mark.html?highlight=slow\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Using pytest markers to track functional coverage and filtering of tests",
"version": "2.1.0",
"project_urls": null,
"split_keywords": [
"test",
" pytest",
" markers",
" coverage"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4460fc272aac35ffb8219b5d2da158b39cfbd1d81c44e17414d8ce0a4e973eff",
"md5": "ecbbaba3463188665e8f1a112f4de7e3",
"sha256": "023b7f803dbc436d0702217feeba9115d79f266b47260ededc55c7e8e5586560"
},
"downloads": -1,
"filename": "pytest_coveragemarkers-2.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "ecbbaba3463188665e8f1a112f4de7e3",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.11.0,>=3.8",
"size": 15418,
"upload_time": "2024-04-15T10:54:23",
"upload_time_iso_8601": "2024-04-15T10:54:23.264823Z",
"url": "https://files.pythonhosted.org/packages/44/60/fc272aac35ffb8219b5d2da158b39cfbd1d81c44e17414d8ce0a4e973eff/pytest_coveragemarkers-2.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "2ee29980a189ebc43c43b43750ea23047c75e7f75564b9fbe56cbd807fd1d908",
"md5": "560229678cc3c1c99fef29da0275d637",
"sha256": "6d0f1c74f962c8bf2030ab000016743b7f70e6cbe8ec42f54785419884995a21"
},
"downloads": -1,
"filename": "pytest_coveragemarkers-2.1.0.tar.gz",
"has_sig": false,
"md5_digest": "560229678cc3c1c99fef29da0275d637",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.11.0,>=3.8",
"size": 13779,
"upload_time": "2024-04-15T10:54:24",
"upload_time_iso_8601": "2024-04-15T10:54:24.529667Z",
"url": "https://files.pythonhosted.org/packages/2e/e2/9980a189ebc43c43b43750ea23047c75e7f75564b9fbe56cbd807fd1d908/pytest_coveragemarkers-2.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-15 10:54:24",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "pytest-coveragemarkers"
}
JSON