nuts


Namenuts JSON
Version 3.2.0 PyPI version JSON
download
home_pagehttps://github.com/network-unit-testing-system/nuts
SummaryNetwork Unit Testing System
upload_time2023-08-11 15:40:35
maintainerMarco Martinez
docs_urlNone
authorLukas Murer, Méline Sieber, Urs Baumann, Matthias Gabriel, Florian Bruhin
requires_python>=3.8.1,<4.0.0
licenseMIT
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # Network Unit Testing System

## Introduction

The Network Unit Testing System or "nuts" in short draws on the concept of unit tests, known from the domain of programming, and applies it to the domain of networking.

One major difference between unit tests in programming and network tests is the definition of what a test actually is. 
In programming, unit tests normally focus on testing edge cases, since the amount of non-edge cases is not definable.
In the network testing domain, tests are less about edge cases, but more about testing existing network states with 
pre-defined test cases. Such a single test case might be "can host A reach neighbors X, Y, Z?" on many different devices. 
This is what nuts tries to achieve:
Apply test cases based on your pre-defined network topology to your actual network and have the tests confirm the correct state.

## Installation Instructions

### Using pip

Run `pip install nuts` 

### Using poetry

Nuts uses [poetry](https://python-poetry.org/) as a dependency manager.

1. [Install poetry](https://python-poetry.org/docs/#installation).
2. Clone this repository.
3. Run `$ poetry install`

## How It Works: Test Bundles and Test Definitions

The project relies on the [pytest framework](https://docs.pytest.org/) to setup and execute the tests. 
Nuts itself is written as a custom pytest plugin. In the background, [nornir](https://nornir.readthedocs.io/) 
executes specific network tasks for the actual tests. This can be extended to use other context as well in the future.

Nuts treats the test definition and the so-called test bundle as separate entities. The *test definition* is modeled as a custom `pytest.Class`, and a predefined set of test definitions can be found in the nuts module `base_tests`. New test definitions can be added easily by the user of the plugin.

The *test bundle* is a file that is parsed by pytest. The file provides data on the desired network state and describes which test definitions should be collected and executed by pytest. 
The structure of the test bundle should enable people without in-depth python knowledge to add new test bundles or update existing ones to reflect changes in the network. 

While the readme here is only a short overview, find the [documentation of nuts on readthedocs](https://nuts.readthedocs.io/en/latest/).

### Test Bundle Structure

Currently only yaml files are supported as test bundles, 
but other sources such as other file formats or database entries can be considered in later nuts versions.

Each test bundle contains the following structure:
```yaml
---
- test_module: <module that contains the test class> # optional
  test_class: <name of the test class>
  label: <label to uniquely identify the test> # optional 
  test_execution: <additional data used to execute the test> # optional
  test_data: <data used to generate the test cases>
...
```
`test_module`: The full path of the Python module that contains the test class to be used.
This value is optional if the test class is registered in `index.py` of the pytest-nuts plugin.
Note that it can be relevant in which directory `pytest` is started if local test modules are used. Using `test_modules` allows you to write your own test classes. **Note: We currently do not support self-written test modules, since upcoming refactorings might introduce breaking changes.**

`test_class`: The name of the Python class which contains the tests that should be executed.
Note that currently every test in this class will be executed.

`label`: Additional identifier that can be used to distinguish between multiple occurrences of the same 
 test class in a test bundle.

`test_execution`: Data that is exposed as part of the `nuts_parameters` property. 
By convention, this contains additional information that is passed directly to the nornir task in the background. 
Therefore the key-value pairs must be consistent with the key-value pairs of the specific nornir task. 
As an example, the test definition `napalm_ping.py` calls a nornir task to execute napalm's ping-command. 
This allows the additional `max_drop` parameter in `test_execution`, since it is in turn pre-defined by napalm.

`test_data`: Data that is used to parametrize the tests in the test class which have the `pytest.mark.nuts` annotation. It is additionally part of the `nuts_parameters` property.

### Example: CDP Neighbors
Example of a test bundle for `TestNetmikoCdpNeighbors` which tests that `R1` is a CDP Neighbor of both `R2` and `R3`.
This example creates three different tests, one for each entry in the `test_data` list.

```yaml
---
- test_module: nuts.base_tests.netmiko_cdp_neighbors
  test_class: TestNetmikoCdpNeighbors
  test_data:
    - host: R1
      local_port: GigabitEthernet3
      destination_host: R2
      management_ip: 172.16.12.2
      remote_port: GigabitEthernet2
    - host: R1
      local_port: GigabitEthernet4
      destination_host: R3
      management_ip: 172.16.13.3
      remote_port: GigabitEthernet2
    - host: R2
      local_port: GigabitEthernet2
      destination_host: R1
      management_ip: 172.16.12.1
      remote_port: GigabitEthernet3
...
```

### How the Test Bundle Is Converted to a Pytest Test

When nuts is executed, pytest converts the test bundles (the yaml files) into tests. During test collection, the custom pytest marker `nuts` uses the data that has been defined in the test bundle mentioned above. 
This annotation is a wrapper around the `pytest.mark.parametrize` annotation and allows the plugin to use the data entries from the test bundle. For each entry in the `test_data` section of the test bundle, the custom marker generates a single test case. To achieve this, the plugin transforms the entries into n-tuples, since `pytest.mark.parametrize` expects a list of n-tuples as input. 

The custom nuts marker takes two arguments: The first argument of the annotation determines the required fields. 
For each entry in `test_data` these fields are extracted and transformed to a tuple considering the correct order.
If any of these fields are not present in an entry of `test_data`, the corresponding test case will be skipped.
A second argument determines optional fields that can also be used in a test case as well - non-present values are passed into the function as `None`.

The following test-run of CDP neighbors for example checks the local port:

```python
class TestNetmikoCdpNeighbors:       
    @pytest.mark.nuts("remote_host,local_port")
    def test_local_port(self, single_result, remote_host, local_port):
        assert single_result.result[remote_host]["local_port"] == local_port        
```


The required fields are `host`, `remote_host` and `local_port` - they must be present in the custom marker, 
but also be provided as argument to the test method itself.

`single_result` uses the `host` field and provides the result that has been processed via the specific context of a test.

### Test classes and their context
Each test module implements a context class to provide module-specific functionality to its tests. This context class is a  `NutsContext` or a subclass of it. 
This guarantees a consistent interface across all tests for test setup and execution. 
Currently, the predefined test classes use [nornir](https://nornir.readthedocs.io/en/latest/) in order to communicate 
with the network devices. Those test classes derive all from a more specific `NornirNutsContext`, 
which provides a nornir instance and nornir-specific helpers. In the example above, it is a class called `CdpNeighborsContext` that derives from `NornirNutsContext`.

If you want to learn more how nuts works but do not have a running network in the background, there's a nuts showcase - an offline test class that displays the basic functionality of nuts. See the [tutorial](https://nuts.readthedocs.io/en/latest/tutorial/firststeps.html) for further information.

## Develop Your Own Test Classes

Nuts is essentially designed as a pytest-plugin and it is possible to add your own, self-written test classes. Dev documentation on how to write your own test classes is planned for a future release.
Still, it is possible to write your own test classes nevertheless, even if we cannot guarantee that upcoming planned refactorings do not introduce breaking changes.

## Community-provided test classes

* [Cisco IOSXE Test Class](https://github.com/briantsaunders/nuts-cisco-iosxe-tests) by [briantsaunders](https://github.com/briantsaunders)
* [nuts-experiments](https://github.com/ubaumann/nuts-experiments) by [ubaumann](https://github.com/ubaumann/)

# Thanks

* [Urs Baumann](https://github.com/ubaumann/), for originating the idea, supervising the development, and serving as the project owner
* Andreas Stalder and David Meister, for developing the first version based on SaltStack in their term project
* Mike Schmid and Janik Schlatter, implemented the first version using Nornir in their term project.
* [Matthias Gabriel](https://github.com/MatthiasGabriel), who laid the foundations of nuts as a pytest plugin.
* [Marco Martinez](https://github.com/marcom4rtinez) and [Severin Grimm](https://github.com/Sevitama), added more test cases and interviewed companies in their term project.
* [Méline Sieber](https://github.com/bytinbit), [Lukas Murer](https://github.com/lucmurer), for maintenance during your working hours at the INS
* [Florian Bruhin (The Compiler)](https://github.com/The-Compiler) for invaluable feedback and advice.

            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/network-unit-testing-system/nuts",
    "name": "nuts",
    "maintainer": "Marco Martinez",
    "docs_url": null,
    "requires_python": ">=3.8.1,<4.0.0",
    "maintainer_email": "marco.maritnez@ost.ch",
    "keywords": "",
    "author": "Lukas Murer, M\u00e9line Sieber, Urs Baumann, Matthias Gabriel, Florian Bruhin",
    "author_email": "",
    "download_url": "https://files.pythonhosted.org/packages/3d/f4/56682a71ca08b0744c5f9c3ea57bf9eb9ffa85a5f9c732da17992d623038/nuts-3.2.0.tar.gz",
    "platform": null,
    "description": "# Network Unit Testing System\n\n## Introduction\n\nThe Network Unit Testing System or \"nuts\" in short draws on the concept of unit tests, known from the domain of programming, and applies it to the domain of networking.\n\nOne major difference between unit tests in programming and network tests is the definition of what a test actually is. \nIn programming, unit tests normally focus on testing edge cases, since the amount of non-edge cases is not definable.\nIn the network testing domain, tests are less about edge cases, but more about testing existing network states with \npre-defined test cases. Such a single test case might be \"can host A reach neighbors X, Y, Z?\" on many different devices. \nThis is what nuts tries to achieve:\nApply test cases based on your pre-defined network topology to your actual network and have the tests confirm the correct state.\n\n## Installation Instructions\n\n### Using pip\n\nRun `pip install nuts` \n\n### Using poetry\n\nNuts uses [poetry](https://python-poetry.org/) as a dependency manager.\n\n1. [Install poetry](https://python-poetry.org/docs/#installation).\n2. Clone this repository.\n3. Run `$ poetry install`\n\n## How It Works: Test Bundles and Test Definitions\n\nThe project relies on the [pytest framework](https://docs.pytest.org/) to setup and execute the tests. \nNuts itself is written as a custom pytest plugin. In the background, [nornir](https://nornir.readthedocs.io/) \nexecutes specific network tasks for the actual tests. This can be extended to use other context as well in the future.\n\nNuts treats the test definition and the so-called test bundle as separate entities. The *test definition* is modeled as a custom `pytest.Class`, and a predefined set of test definitions can be found in the nuts module `base_tests`. New test definitions can be added easily by the user of the plugin.\n\nThe *test bundle* is a file that is parsed by pytest. The file provides data on the desired network state and describes which test definitions should be collected and executed by pytest. \nThe structure of the test bundle should enable people without in-depth python knowledge to add new test bundles or update existing ones to reflect changes in the network. \n\nWhile the readme here is only a short overview, find the [documentation of nuts on readthedocs](https://nuts.readthedocs.io/en/latest/).\n\n### Test Bundle Structure\n\nCurrently only yaml files are supported as test bundles, \nbut other sources such as other file formats or database entries can be considered in later nuts versions.\n\nEach test bundle contains the following structure:\n```yaml\n---\n- test_module: <module that contains the test class> # optional\n  test_class: <name of the test class>\n  label: <label to uniquely identify the test> # optional \n  test_execution: <additional data used to execute the test> # optional\n  test_data: <data used to generate the test cases>\n...\n```\n`test_module`: The full path of the Python module that contains the test class to be used.\nThis value is optional if the test class is registered in `index.py` of the pytest-nuts plugin.\nNote that it can be relevant in which directory `pytest` is started if local test modules are used. Using `test_modules` allows you to write your own test classes. **Note: We currently do not support self-written test modules, since upcoming refactorings might introduce breaking changes.**\n\n`test_class`: The name of the Python class which contains the tests that should be executed.\nNote that currently every test in this class will be executed.\n\n`label`: Additional identifier that can be used to distinguish between multiple occurrences of the same \n test class in a test bundle.\n\n`test_execution`: Data that is exposed as part of the `nuts_parameters` property. \nBy convention, this contains additional information that is passed directly to the nornir task in the background. \nTherefore the key-value pairs must be consistent with the key-value pairs of the specific nornir task. \nAs an example, the test definition `napalm_ping.py` calls a nornir task to execute napalm's ping-command. \nThis allows the additional `max_drop` parameter in `test_execution`, since it is in turn pre-defined by napalm.\n\n`test_data`: Data that is used to parametrize the tests in the test class which have the `pytest.mark.nuts` annotation. It is additionally part of the `nuts_parameters` property.\n\n### Example: CDP Neighbors\nExample of a test bundle for `TestNetmikoCdpNeighbors` which tests that `R1` is a CDP Neighbor of both `R2` and `R3`.\nThis example creates three different tests, one for each entry in the `test_data` list.\n\n```yaml\n---\n- test_module: nuts.base_tests.netmiko_cdp_neighbors\n  test_class: TestNetmikoCdpNeighbors\n  test_data:\n    - host: R1\n      local_port: GigabitEthernet3\n      destination_host: R2\n      management_ip: 172.16.12.2\n      remote_port: GigabitEthernet2\n    - host: R1\n      local_port: GigabitEthernet4\n      destination_host: R3\n      management_ip: 172.16.13.3\n      remote_port: GigabitEthernet2\n    - host: R2\n      local_port: GigabitEthernet2\n      destination_host: R1\n      management_ip: 172.16.12.1\n      remote_port: GigabitEthernet3\n...\n```\n\n### How the Test Bundle Is Converted to a Pytest Test\n\nWhen nuts is executed, pytest converts the test bundles (the yaml files) into tests. During test collection, the custom pytest marker `nuts` uses the data that has been defined in the test bundle mentioned above. \nThis annotation is a wrapper around the `pytest.mark.parametrize` annotation and allows the plugin to use the data entries from the test bundle. For each entry in the `test_data` section of the test bundle, the custom marker generates a single test case. To achieve this, the plugin transforms the entries into n-tuples, since `pytest.mark.parametrize` expects a list of n-tuples as input. \n\nThe custom nuts marker takes two arguments: The first argument of the annotation determines the required fields. \nFor each entry in `test_data` these fields are extracted and transformed to a tuple considering the correct order.\nIf any of these fields are not present in an entry of `test_data`, the corresponding test case will be skipped.\nA second argument determines optional fields that can also be used in a test case as well - non-present values are passed into the function as `None`.\n\nThe following test-run of CDP neighbors for example checks the local port:\n\n```python\nclass TestNetmikoCdpNeighbors:       \n    @pytest.mark.nuts(\"remote_host,local_port\")\n    def test_local_port(self, single_result, remote_host, local_port):\n        assert single_result.result[remote_host][\"local_port\"] == local_port        \n```\n\n\nThe required fields are `host`, `remote_host` and `local_port` - they must be present in the custom marker, \nbut also be provided as argument to the test method itself.\n\n`single_result` uses the `host` field and provides the result that has been processed via the specific context of a test.\n\n### Test classes and their context\nEach test module implements a context class to provide module-specific functionality to its tests. This context class is a  `NutsContext` or a subclass of it. \nThis guarantees a consistent interface across all tests for test setup and execution. \nCurrently, the predefined test classes use [nornir](https://nornir.readthedocs.io/en/latest/) in order to communicate \nwith the network devices. Those test classes derive all from a more specific `NornirNutsContext`, \nwhich provides a nornir instance and nornir-specific helpers. In the example above, it is a class called `CdpNeighborsContext` that derives from `NornirNutsContext`.\n\nIf you want to learn more how nuts works but do not have a running network in the background, there's a nuts showcase - an offline test class that displays the basic functionality of nuts. See the [tutorial](https://nuts.readthedocs.io/en/latest/tutorial/firststeps.html) for further information.\n\n## Develop Your Own Test Classes\n\nNuts is essentially designed as a pytest-plugin and it is possible to add your own, self-written test classes. Dev documentation on how to write your own test classes is planned for a future release.\nStill, it is possible to write your own test classes nevertheless, even if we cannot guarantee that upcoming planned refactorings do not introduce breaking changes.\n\n## Community-provided test classes\n\n* [Cisco IOSXE Test Class](https://github.com/briantsaunders/nuts-cisco-iosxe-tests) by [briantsaunders](https://github.com/briantsaunders)\n* [nuts-experiments](https://github.com/ubaumann/nuts-experiments) by [ubaumann](https://github.com/ubaumann/)\n\n# Thanks\n\n* [Urs Baumann](https://github.com/ubaumann/), for originating the idea, supervising the development, and serving as the project owner\n* Andreas Stalder and David Meister, for developing the first version based on SaltStack in their term project\n* Mike Schmid and Janik Schlatter, implemented the first version using Nornir in their term project.\n* [Matthias Gabriel](https://github.com/MatthiasGabriel), who laid the foundations of nuts as a pytest plugin.\n* [Marco Martinez](https://github.com/marcom4rtinez) and [Severin Grimm](https://github.com/Sevitama), added more test cases and interviewed companies in their term project.\n* [M\u00e9line Sieber](https://github.com/bytinbit), [Lukas Murer](https://github.com/lucmurer), for maintenance during your working hours at the INS\n* [Florian Bruhin (The Compiler)](https://github.com/The-Compiler) for invaluable feedback and advice.\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "Network Unit Testing System",
    "version": "3.2.0",
    "project_urls": {
        "Homepage": "https://github.com/network-unit-testing-system/nuts",
        "Repository": "https://github.com/network-unit-testing-system/nuts"
    },
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "40b9a6d98282bd9f05baa5b5182edabcfe17d61791928ff0ffe6f3ead044df86",
                "md5": "3b0ab9d2267a0814f1477b1b1990d305",
                "sha256": "cdaca15f21b7a699bcd09a1a99aaf963f4a93d610c1215bebf03178b30f9f668"
            },
            "downloads": -1,
            "filename": "nuts-3.2.0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "3b0ab9d2267a0814f1477b1b1990d305",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.8.1,<4.0.0",
            "size": 29927,
            "upload_time": "2023-08-11T15:40:34",
            "upload_time_iso_8601": "2023-08-11T15:40:34.299339Z",
            "url": "https://files.pythonhosted.org/packages/40/b9/a6d98282bd9f05baa5b5182edabcfe17d61791928ff0ffe6f3ead044df86/nuts-3.2.0-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "3df456682a71ca08b0744c5f9c3ea57bf9eb9ffa85a5f9c732da17992d623038",
                "md5": "52cd442a2310ac7224d587e381d03382",
                "sha256": "f73e8687b5cbfceedafb2a52f7a95b831b491b4ebcab290cfcd05848c2a8665f"
            },
            "downloads": -1,
            "filename": "nuts-3.2.0.tar.gz",
            "has_sig": false,
            "md5_digest": "52cd442a2310ac7224d587e381d03382",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.8.1,<4.0.0",
            "size": 24145,
            "upload_time": "2023-08-11T15:40:35",
            "upload_time_iso_8601": "2023-08-11T15:40:35.532372Z",
            "url": "https://files.pythonhosted.org/packages/3d/f4/56682a71ca08b0744c5f9c3ea57bf9eb9ffa85a5f9c732da17992d623038/nuts-3.2.0.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2023-08-11 15:40:35",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "network-unit-testing-system",
    "github_project": "nuts",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "lcname": "nuts"
}
        
Elapsed time: 0.11345s