Netdut: automated software testing for switches using pytest
============================================================
Netdut is a [pytest](https://docs.pytest.org/) plugin which provides infrastructure (e.g. pytest fixtures) which make it easy to write automated tests for network switches.
Features
--------
* Brings the power, maturity and ecosystem of pytest to testing on network switches.
* The `dut` fixture (Device Under Test) providing serial, ssh or [EAPI](https://github.com/arista-eosplus/pyeapi) connectivity for running CLI commands on a network switch.
* Command-line configuration of a hostname and console name.
* Markers for skipping tests based on the device's type or software configuration.
* Compatibility with both Arista's EOS operating system, and the Metamako MOS operating system.
* A pythonic interfaces for writing EOS CLI commands.
Requirements
------------
* Requires python3 >= 3.6, pexpect, pyeapi
* Network devices under test (DUTs) must be accessible via SSH.
Installation
------------
You can install `pytest-netdut` via [pip](https://pypi.org/project/pip/) from [PyPI](https://pypi.org/project):
$ pip install pytest-netdut
The `pytest-netdut` plugin is not auto-loaded upon installation. To enable pytest-netdut for your
tests, add the following to a `conftest.py` file:
```
pytest_plugins = ["pytest_netdut"]
```
or explicitly add it when you run pytest:
```
pytest -p pytest_netdut
```
Usage
-----
### Basics
Several example tests using pytest-netdut are contained in [examples/](examples/). Note that there is a `conftest.py` in that directory which enables pytest-netdut and increases the verbosity of the CLI output.
The DUT must be accessible via SSH to use the SSH or EAPI fixtures (the latter enables and disables the EAPI using SSH). This can usually be achieved using this EOS config:
```text
enable
configure
username admin privilege 15 nopassword
aaa authorization exec default local
```
`test_showver.py` demonstrates some basic examples using the dut fixture. The tests can be
be run using the command below.
```bash
pytest --device=<YOUR_DEVICE_IP_OR_DNS> examples/test_showver.py
```
`test_showver` runs `show version` via EAPI:
```python
def test_showver(dut):
info = dut.eapi.sendcmd("show version")
logging.critical(f"DUT model was: {info['modelName']}")
```
Running this using a device with a hostname of `dmb224` gives some simple output:
### Assertions
Because this is pytest, we get great feedback when tests fail. Tests fail upon assertions. Here we use the test below as an example.
```python
def test_check_version(dut):
info = dut.eapi.sendcmd("show version")
assert info["version"] == "1.0.0"
```
Netdut will start EAPI and use it to run the `show version` command, returning the results as a python dictionary. The assertion then fails, assuming the DUT is not running version 1.0.0 of EOS (in this case it is running an old internal build):
### Running multiple commands
It's rare that a test will execute a single command on the DUT. Netdut allows for multi-command tests. This can be seen in [examples/test_daemon.py](examples/test_daemon.py) where the daemon
is configured:
```python
dut.eapi.sendcmds("""
enable
configure
daemon sleeper
exec /usr/bin/sleep 10
no shutdown
""")
```
In this case the result that is returned is list of the results of each command. It's often useful
to just return the last one:
```python
def daemon_has_started():
daemon_info = dut.eapi.sendcmds(["enable", f"show daemon {sleeper_daemon}"])[-1]
return daemon_info["daemons"][sleeper_daemon]["starttime"] != 0.0
```
### Waiting for results
It's often useful to provide some configuration to the switch, and then wait for the
switch to act (since it's running asynchronously). pytest-netdut provides the
`wait_for` fixture for this purpose. Again, in
[examples/test_daemon.py](examples/test_daemon.py) we use `wait_for` to wait for
the daemon to start and stop.
```python
# daemon_has_started is defined as a local function
# wait_for calls it repeatedly until it's true
wait_for(daemon_has_started, timeout=10.0)
```
### Skipping based on DUT type
Perhaps some tests won't run on every type of device -- `pytest-netdut` provides some skipping mechanisms for this purpose. To skip a test on a particular on a DUT type use the provided
decorator:
```python
@pytest.mark.skip_device_type("DCS-7130.*", reason="Demo")
def test_that_skips_7130(dut):
logging.info("Must not be 7130!")
```
Similar decorators exist for OS type (i.e. EOS-only or MOS-only tests), and can be stacked:
```python
@pytest.mark.eos
@pytest.mark.skip_device_type("DCS-7130.*")
def test_that_only_runs_on_eos_on_7130(dut):
logging.info("Must be EOS on 7130!")
```
OS decorators accept the following keywords:
- min_version (string)
- min_change_number (int)
If both kwargs are specified, min_change_number takes precedence.
```python
@pytest.mark.eos(min_version="4.30.0", min_change_number=3452345)
@pytest.mark.skip_device_type("DCS-7130.*")
def test_that_only_runs_on_eos_on_7130(dut):
logging.info("Must be EOS on 7130!")
```
### Building test harnesses
Pytest's fixture mechanism is very powerful in this scenario, allowing us to set up and tear
down test configurations with low overhead.
```python
@pytest.fixture
def my_test_harness(dut):
dut.eapi.sendcmds(["enable",
"configure",
"banner motd My test is running, hands off!"])
yield
dut.eapi.sendcmds(["enable", "configure", "no banner"])
def test_mytest(my_test_harness):
logging.info("Running my test here")
```
### Translations
In order to reduce test code verbosity and complexity a translator class provides a way to standardize tests to a particular OS.
Both CLI commands and results are standardized to EOS by default.
A particular test can be written with EOS CLI and be run on MOS as well.
Consider the `l1 source` command which differs between EOS and MOS:
```python
@pytest.fixture
def l1_connect(dut):
if dut.ssh.cli_flavor == "eos":
dut.eapi.sendcmds(["enable",
"configure",
"interface Ethernet10",
"l1 source interface Ethernet12"])
elif dut.ssh.cli_flavor == "mos":
dut.eapi.sendcmds(["enable",
"configure",
"interface Ethernet10",
"source Ethernet12"])
```
which can be reduced to:
```python
@pytest.fixture
def l1_connect(dut):
dut.eapi.sendcmds(["enable",
"configure",
"interface Ethernet10",
"l1 source interface Ethernet12"])
```
The translator will try to find a match for each command in the predefined list of regex configuration patterns.
The translator will also process return values; MOS EAPI result keys are camelCased and the translator will convert all keys
to snake_case.
The translator has a predefined set of translations which can be extended by subclassing the Translator class and overriding `config_patterns`.
Return values are processed by the `translate_key` function which must be defined in the subclass.
Set the new translator class via `eapi.set_translator(<class instance>)`.
Contributing
------------
This project is under active use and development. We would appreciate help to improve it,
via pull request. Tests can be run with `make ci` or `tox`.
Docstrings are according to [Google's docstring conventions](https://google.github.io/styleguide/pyguide.html) ([examples](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html)).
License
-------
Distributed under the terms of the [BSD-3](http://opensource.org/licenses/BSD-3-Clause) license, `pytest-netdut` is free and open source software
Issues
------
If you encounter any problems, please [file an issue](https://github.com/dcasnowdon-anet/pytest-netdut/issues) along with a detailed description.
Copyright (c) 2021, David Snowdon
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the name of pytest-netdut nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Raw data
{
"_id": null,
"home_page": "https://github.com/aristanetworks/pytest-netdut",
"name": "pytest-netdut",
"maintainer": "\"Alex Webster\"",
"docs_url": null,
"requires_python": "",
"maintainer_email": "\"alexw@arista.com\"",
"keywords": "pytest,eos",
"author": "\"David Snowdon\"",
"author_email": "\"daves@arista.com\"",
"download_url": "https://files.pythonhosted.org/packages/75/1b/e347c5da28defb499c47ff6974edd0e52af897170e7a9cf9c020a29e81c1/pytest_netdut-0.11.0.tar.gz",
"platform": null,
"description": "Netdut: automated software testing for switches using pytest\n============================================================\n\n\n\n\n\n\nNetdut is a [pytest](https://docs.pytest.org/) plugin which provides infrastructure (e.g. pytest fixtures) which make it easy to write automated tests for network switches.\n\n\nFeatures\n--------\n\n* Brings the power, maturity and ecosystem of pytest to testing on network switches.\n* The `dut` fixture (Device Under Test) providing serial, ssh or [EAPI](https://github.com/arista-eosplus/pyeapi) connectivity for running CLI commands on a network switch.\n* Command-line configuration of a hostname and console name.\n* Markers for skipping tests based on the device's type or software configuration.\n* Compatibility with both Arista's EOS operating system, and the Metamako MOS operating system.\n* A pythonic interfaces for writing EOS CLI commands.\n\n\nRequirements\n------------\n\n* Requires python3 >= 3.6, pexpect, pyeapi\n* Network devices under test (DUTs) must be accessible via SSH.\n\n\nInstallation\n------------\n\nYou can install `pytest-netdut` via [pip](https://pypi.org/project/pip/) from [PyPI](https://pypi.org/project):\n\n $ pip install pytest-netdut\n\nThe `pytest-netdut` plugin is not auto-loaded upon installation. To enable pytest-netdut for your\ntests, add the following to a `conftest.py` file:\n\n```\npytest_plugins = [\"pytest_netdut\"]\n```\n\nor explicitly add it when you run pytest:\n\n```\npytest -p pytest_netdut\n```\n\n\nUsage\n-----\n\n### Basics\n\nSeveral example tests using pytest-netdut are contained in [examples/](examples/). Note that there is a `conftest.py` in that directory which enables pytest-netdut and increases the verbosity of the CLI output.\n\nThe DUT must be accessible via SSH to use the SSH or EAPI fixtures (the latter enables and disables the EAPI using SSH). This can usually be achieved using this EOS config:\n\n```text\n enable\n configure\n username admin privilege 15 nopassword\n aaa authorization exec default local\n```\n\n`test_showver.py` demonstrates some basic examples using the dut fixture. The tests can be\nbe run using the command below.\n\n```bash\npytest --device=<YOUR_DEVICE_IP_OR_DNS> examples/test_showver.py\n```\n\n`test_showver` runs `show version` via EAPI:\n\n```python\ndef test_showver(dut):\n info = dut.eapi.sendcmd(\"show version\")\n logging.critical(f\"DUT model was: {info['modelName']}\")\n```\n\nRunning this using a device with a hostname of `dmb224` gives some simple output:\n\n\n\n\n### Assertions\n\nBecause this is pytest, we get great feedback when tests fail. Tests fail upon assertions. Here we use the test below as an example.\n\n```python\ndef test_check_version(dut):\n info = dut.eapi.sendcmd(\"show version\")\n assert info[\"version\"] == \"1.0.0\"\n```\n\nNetdut will start EAPI and use it to run the `show version` command, returning the results as a python dictionary. The assertion then fails, assuming the DUT is not running version 1.0.0 of EOS (in this case it is running an old internal build):\n\n\n\n\n### Running multiple commands\n\nIt's rare that a test will execute a single command on the DUT. Netdut allows for multi-command tests. This can be seen in [examples/test_daemon.py](examples/test_daemon.py) where the daemon\nis configured:\n\n```python\ndut.eapi.sendcmds(\"\"\"\n enable\n configure\n daemon sleeper\n exec /usr/bin/sleep 10\n no shutdown\n \"\"\")\n```\n\nIn this case the result that is returned is list of the results of each command. It's often useful\nto just return the last one:\n\n```python\ndef daemon_has_started():\n daemon_info = dut.eapi.sendcmds([\"enable\", f\"show daemon {sleeper_daemon}\"])[-1]\n return daemon_info[\"daemons\"][sleeper_daemon][\"starttime\"] != 0.0\n```\n\n### Waiting for results\n\nIt's often useful to provide some configuration to the switch, and then wait for the\nswitch to act (since it's running asynchronously). pytest-netdut provides the\n`wait_for` fixture for this purpose. Again, in\n[examples/test_daemon.py](examples/test_daemon.py) we use `wait_for` to wait for\nthe daemon to start and stop.\n\n```python\n# daemon_has_started is defined as a local function\n# wait_for calls it repeatedly until it's true\nwait_for(daemon_has_started, timeout=10.0)\n```\n\n### Skipping based on DUT type\n\nPerhaps some tests won't run on every type of device -- `pytest-netdut` provides some skipping mechanisms for this purpose. To skip a test on a particular on a DUT type use the provided\ndecorator:\n\n```python\n@pytest.mark.skip_device_type(\"DCS-7130.*\", reason=\"Demo\")\ndef test_that_skips_7130(dut):\n logging.info(\"Must not be 7130!\")\n```\n\nSimilar decorators exist for OS type (i.e. EOS-only or MOS-only tests), and can be stacked:\n```python\n@pytest.mark.eos\n@pytest.mark.skip_device_type(\"DCS-7130.*\")\ndef test_that_only_runs_on_eos_on_7130(dut):\n logging.info(\"Must be EOS on 7130!\")\n```\n\nOS decorators accept the following keywords:\n - min_version (string)\n - min_change_number (int)\n\nIf both kwargs are specified, min_change_number takes precedence.\n```python\n@pytest.mark.eos(min_version=\"4.30.0\", min_change_number=3452345)\n@pytest.mark.skip_device_type(\"DCS-7130.*\")\ndef test_that_only_runs_on_eos_on_7130(dut):\n logging.info(\"Must be EOS on 7130!\")\n```\n\n### Building test harnesses\n\nPytest's fixture mechanism is very powerful in this scenario, allowing us to set up and tear\ndown test configurations with low overhead.\n\n```python\n@pytest.fixture\ndef my_test_harness(dut):\n dut.eapi.sendcmds([\"enable\",\n \"configure\",\n \"banner motd My test is running, hands off!\"])\n yield\n dut.eapi.sendcmds([\"enable\", \"configure\", \"no banner\"])\n\ndef test_mytest(my_test_harness):\n logging.info(\"Running my test here\")\n```\n\n### Translations\n\nIn order to reduce test code verbosity and complexity a translator class provides a way to standardize tests to a particular OS.\nBoth CLI commands and results are standardized to EOS by default.\n\nA particular test can be written with EOS CLI and be run on MOS as well.\nConsider the `l1 source` command which differs between EOS and MOS:\n\n```python\n@pytest.fixture\ndef l1_connect(dut):\n if dut.ssh.cli_flavor == \"eos\":\n dut.eapi.sendcmds([\"enable\",\n \"configure\",\n \"interface Ethernet10\",\n \"l1 source interface Ethernet12\"])\n elif dut.ssh.cli_flavor == \"mos\":\n dut.eapi.sendcmds([\"enable\",\n \"configure\",\n \"interface Ethernet10\",\n \"source Ethernet12\"])\n```\n\nwhich can be reduced to:\n\n```python\n@pytest.fixture\ndef l1_connect(dut):\n dut.eapi.sendcmds([\"enable\",\n \"configure\",\n \"interface Ethernet10\",\n \"l1 source interface Ethernet12\"])\n```\n\nThe translator will try to find a match for each command in the predefined list of regex configuration patterns.\n\nThe translator will also process return values; MOS EAPI result keys are camelCased and the translator will convert all keys\nto snake_case.\n\nThe translator has a predefined set of translations which can be extended by subclassing the Translator class and overriding `config_patterns`.\nReturn values are processed by the `translate_key` function which must be defined in the subclass.\n\nSet the new translator class via `eapi.set_translator(<class instance>)`.\n\nContributing\n------------\nThis project is under active use and development. We would appreciate help to improve it,\nvia pull request. Tests can be run with `make ci` or `tox`.\n\nDocstrings are according to [Google's docstring conventions](https://google.github.io/styleguide/pyguide.html) ([examples](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html)).\n\n\n\nLicense\n-------\n\nDistributed under the terms of the [BSD-3](http://opensource.org/licenses/BSD-3-Clause) license, `pytest-netdut` is free and open source software\n\n\nIssues\n------\n\nIf you encounter any problems, please [file an issue](https://github.com/dcasnowdon-anet/pytest-netdut/issues) along with a detailed description.\n\nCopyright (c) 2021, David Snowdon\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or without\nmodification, are permitted provided that the following conditions are met:\n\n* Redistributions of source code must retain the above copyright notice, this\n list of conditions and the following disclaimer.\n\n* Redistributions in binary form must reproduce the above copyright notice,\n this list of conditions and the following disclaimer in the documentation\n and/or other materials provided with the distribution.\n\n* Neither the name of pytest-netdut nor the names of its\n contributors may be used to endorse or promote products derived from\n this software without specific prior written permission.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\nAND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\nIMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\nDISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE\nFOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL\nDAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR\nSERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\nCAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,\nOR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\nOF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n",
"bugtrack_url": null,
"license": "\"BSD 3-Clause License\"",
"summary": "\"Automated software testing for switches using pytest\"",
"version": "0.11.0",
"project_urls": {
"Documentation": "https://github.com/aristanetworks/pytest-netdut/",
"Homepage": "https://github.com/aristanetworks/pytest-netdut",
"Source": "https://github.com/aristanetworks/pytest-netdut/",
"Tracker": "https://github.com/aristanetworks/pytest-netdut/issues"
},
"split_keywords": [
"pytest",
"eos"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "5a80b07bf52c1ec07662fc1c487bfa45e8424e803d21cca31f1045dacd0e0df1",
"md5": "3498800eeecdf360b9b8cdb67145a74b",
"sha256": "a97de522bc33098062d480a43e181f25a0b01c9402797ddc1ebbf0425427659d"
},
"downloads": -1,
"filename": "pytest_netdut-0.11.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3498800eeecdf360b9b8cdb67145a74b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 20284,
"upload_time": "2024-03-07T03:46:46",
"upload_time_iso_8601": "2024-03-07T03:46:46.728276Z",
"url": "https://files.pythonhosted.org/packages/5a/80/b07bf52c1ec07662fc1c487bfa45e8424e803d21cca31f1045dacd0e0df1/pytest_netdut-0.11.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "751be347c5da28defb499c47ff6974edd0e52af897170e7a9cf9c020a29e81c1",
"md5": "69dfca1602b1e89962e39ae7ce0e5048",
"sha256": "dc511278def6f65fdc9902e4b64ee05da96404a1eae4f41ee6983e72dc290bc4"
},
"downloads": -1,
"filename": "pytest_netdut-0.11.0.tar.gz",
"has_sig": false,
"md5_digest": "69dfca1602b1e89962e39ae7ce0e5048",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 233352,
"upload_time": "2024-03-07T03:46:48",
"upload_time_iso_8601": "2024-03-07T03:46:48.944742Z",
"url": "https://files.pythonhosted.org/packages/75/1b/e347c5da28defb499c47ff6974edd0e52af897170e7a9cf9c020a29e81c1/pytest_netdut-0.11.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-07 03:46:48",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "aristanetworks",
"github_project": "pytest-netdut",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "pytest-netdut"
}
JSON
