# Py.test plugin for validating Jupyter notebooks
[](https://github.com/computationalmodelling/nbval/actions/workflows/tests.yml)
[](https://pypi.python.org/pypi/nbval)
[](https://nbval.readthedocs.io/)
The plugin adds functionality to py.test to recognise and collect Jupyter
notebooks. The intended purpose of the tests is to determine whether execution
of the stored inputs match the stored outputs of the `.ipynb` file. Whilst also
ensuring that the notebooks are running without errors.
The tests were designed to ensure that Jupyter notebooks (especially those for
reference and documentation), are executing consistently.
Each cell is taken as a test, a cell that doesn't reproduce the expected
output will fail.
See [`docs/source/index.ipynb`](http://nbviewer.jupyter.org/github/computationalmodelling/nbval/blob/HEAD/docs/source/index.ipynb) for the full documentation.
## Installation
Available on PyPi:
pip install nbval
or install the latest version from cloning the repository and running:
pip install .
from the main directory. To uninstall:
pip uninstall nbval
## How it works
The extension looks through every cell that contains code in an IPython notebook
and then the `py.test` system compares the outputs stored in the notebook
with the outputs of the cells when they are executed. Thus, the notebook itself is
used as a testing function.
The output lines when executing the notebook can be sanitized passing an
extra option and file, when calling the `py.test` command. This file
is a usual configuration file for the `ConfigParser` library.
Regarding the execution, roughly, the script initiates an
IPython Kernel with a `shell` and
an `iopub` sockets. The `shell` is needed to execute the cells in
the notebook (it sends requests to the Kernel) and the `iopub` provides
an interface to get the messages from the outputs. The contents
of the messages obtained from the Kernel are organised in dictionaries
with different information, such as time stamps of executions,
cell data types, cell types, the status of the Kernel, username, etc.
In general, the functionality of the IPython notebook system is
quite complex, but a detailed explanation of the messages
and how the system works, can be found here
https://jupyter-client.readthedocs.io/en/latest/messaging.html#messaging
## Execution
To execute this plugin, you need to execute `py.test` with the `nbval` flag
to differentiate the testing from the usual python files:
py.test --nbval
You can also specify `--nbval-lax`, which runs notebooks and checks for
errors, but only compares the outputs of cells with a `#NBVAL_CHECK_OUTPUT`
marker comment.
py.test --nbval-lax
The commands above will execute all the `.ipynb` files and 'pytest' tests in the current folder.
Specify `-p no:python` if you would like to execute notebooks only. Alternatively, you can execute a specific notebook:
py.test --nbval my_notebook.ipynb
By default, each `.ipynb` file will be executed using the kernel
specified in its metadata. You can override this behavior by passing
either `--nbval-kernel-name mykernel` to run all the notebooks using
`mykernel`, or `--current-env` to use a kernel in the same environment
in which pytest itself was launched.
If the output lines are going to be sanitized, an extra flag, `--nbval-sanitize-with`
together with the path to a confguration file with regex expressions, must be passed,
i.e.
py.test --nbval my_notebook.ipynb --nbval-sanitize-with path/to/my_sanitize_file
where `my_sanitize_file` has the following structure.
```
[Section1]
regex: [a-z]*
replace: abcd
regex: [1-9]*
replace: 0000
[Section2]
regex: foo
replace: bar
```
The `regex` option contains the expression that is going to be matched in the outputs, and
`replace` is the string that will replace the `regex` match. Currently, the section
names do not have any meaning or influence in the testing system, it will take
all the sections and replace the corresponding options.
### Coverage
To use notebooks to generate coverage for imported code, use the pytest-cov plugin.
nbval should automatically detect the relevant options and configure itself with it.
### Parallel execution
nbval is compatible with the pytest-xdist plugin for parallel running of tests. It does
however require the use of the `--dist loadscope` flag to ensure that all cells of one
notebook are run on the same kernel.
## Documentation
The narrative documentation for nbval can be found at https://nbval.readthedocs.io.
## Help
The `py.test` system help can be obtained with `py.test -h`, which will
show all the flags that can be passed to the command, such as the
verbose `-v` option. Nbval's options can be found under the
`Jupyter Notebook validation` section.
## Acknowledgements
This plugin was inspired by Andrea Zonca's py.test plugin for collecting unit
tests in the IPython notebooks (https://github.com/zonca/pytest-ipynb).
The original prototype was based on the template in
https://gist.github.com/timo/2621679 and the code of a testing system
for notebooks https://gist.github.com/minrk/2620735 which we
integrated and mixed with the `py.test` system.
We acknowledge financial support from
- OpenDreamKit Horizon 2020 European Research Infrastructures project (#676541), http://opendreamkit.org
- EPSRC's Centre for Doctoral Training in Next Generation
Computational Modelling, http://ngcm.soton.ac.uk (#EP/L015382/1) and
EPSRC's Doctoral Training Centre in Complex System Simulation
((EP/G03690X/1),
- The Gordon and Betty Moore Foundation through Grant GBMF #4856, by the
Alfred P. Sloan Foundation and by the Helmsley Trust.
## Authors
2014 - 2017 David Cortes-Ortuno, Oliver Laslett, T. Kluyver, Vidar
Fauske, Maximilian Albert, MinRK, Ondrej Hovorka, Hans Fangohr
Raw data
{
"_id": null,
"home_page": "https://github.com/computationalmodelling/nbval",
"name": "nbval",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7, <4",
"maintainer_email": "",
"keywords": "",
"author": "Laslett, Cortes, Fauske, Kluyver, Pepper, Fangohr",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/28/be/22bd64d09e0cb53258f83b6fc455f05f18a78e3e5c109ccb6af42f1f49a2/nbval-0.11.0.tar.gz",
"platform": null,
"description": "# Py.test plugin for validating Jupyter notebooks\n\n[](https://github.com/computationalmodelling/nbval/actions/workflows/tests.yml)\n[](https://pypi.python.org/pypi/nbval)\n[](https://nbval.readthedocs.io/)\n\nThe plugin adds functionality to py.test to recognise and collect Jupyter\nnotebooks. The intended purpose of the tests is to determine whether execution\nof the stored inputs match the stored outputs of the `.ipynb` file. Whilst also\nensuring that the notebooks are running without errors.\n\nThe tests were designed to ensure that Jupyter notebooks (especially those for\nreference and documentation), are executing consistently.\n\nEach cell is taken as a test, a cell that doesn't reproduce the expected\noutput will fail.\n\nSee [`docs/source/index.ipynb`](http://nbviewer.jupyter.org/github/computationalmodelling/nbval/blob/HEAD/docs/source/index.ipynb) for the full documentation.\n\n## Installation\nAvailable on PyPi:\n\n pip install nbval\n\nor install the latest version from cloning the repository and running:\n\n pip install .\n\nfrom the main directory. To uninstall:\n\n pip uninstall nbval\n\n\n## How it works\nThe extension looks through every cell that contains code in an IPython notebook\nand then the `py.test` system compares the outputs stored in the notebook\nwith the outputs of the cells when they are executed. Thus, the notebook itself is\nused as a testing function.\nThe output lines when executing the notebook can be sanitized passing an\nextra option and file, when calling the `py.test` command. This file\nis a usual configuration file for the `ConfigParser` library.\n\nRegarding the execution, roughly, the script initiates an\nIPython Kernel with a `shell` and\nan `iopub` sockets. The `shell` is needed to execute the cells in\nthe notebook (it sends requests to the Kernel) and the `iopub` provides\nan interface to get the messages from the outputs. The contents\nof the messages obtained from the Kernel are organised in dictionaries\nwith different information, such as time stamps of executions,\ncell data types, cell types, the status of the Kernel, username, etc.\n\nIn general, the functionality of the IPython notebook system is\nquite complex, but a detailed explanation of the messages\nand how the system works, can be found here\n\nhttps://jupyter-client.readthedocs.io/en/latest/messaging.html#messaging\n\n## Execution\nTo execute this plugin, you need to execute `py.test` with the `nbval` flag\nto differentiate the testing from the usual python files:\n\n py.test --nbval\n\nYou can also specify `--nbval-lax`, which runs notebooks and checks for\nerrors, but only compares the outputs of cells with a `#NBVAL_CHECK_OUTPUT`\nmarker comment.\n\n py.test --nbval-lax\n\nThe commands above will execute all the `.ipynb` files and 'pytest' tests in the current folder.\nSpecify `-p no:python` if you would like to execute notebooks only. Alternatively, you can execute a specific notebook:\n\n py.test --nbval my_notebook.ipynb\n\nBy default, each `.ipynb` file will be executed using the kernel\nspecified in its metadata. You can override this behavior by passing\neither `--nbval-kernel-name mykernel` to run all the notebooks using\n`mykernel`, or `--current-env` to use a kernel in the same environment\nin which pytest itself was launched.\n\nIf the output lines are going to be sanitized, an extra flag, `--nbval-sanitize-with`\ntogether with the path to a confguration file with regex expressions, must be passed,\ni.e.\n\n py.test --nbval my_notebook.ipynb --nbval-sanitize-with path/to/my_sanitize_file\n\nwhere `my_sanitize_file` has the following structure.\n\n```\n[Section1]\nregex: [a-z]*\nreplace: abcd\n\nregex: [1-9]*\nreplace: 0000\n\n[Section2]\nregex: foo\nreplace: bar\n```\n\nThe `regex` option contains the expression that is going to be matched in the outputs, and\n`replace` is the string that will replace the `regex` match. Currently, the section\nnames do not have any meaning or influence in the testing system, it will take\nall the sections and replace the corresponding options.\n\n\n### Coverage\n\nTo use notebooks to generate coverage for imported code, use the pytest-cov plugin.\nnbval should automatically detect the relevant options and configure itself with it.\n\n\n### Parallel execution\n\nnbval is compatible with the pytest-xdist plugin for parallel running of tests. It does\nhowever require the use of the `--dist loadscope` flag to ensure that all cells of one\nnotebook are run on the same kernel.\n\n## Documentation\n\nThe narrative documentation for nbval can be found at https://nbval.readthedocs.io.\n\n## Help\nThe `py.test` system help can be obtained with `py.test -h`, which will\nshow all the flags that can be passed to the command, such as the\nverbose `-v` option. Nbval's options can be found under the\n`Jupyter Notebook validation` section.\n\n\n## Acknowledgements\nThis plugin was inspired by Andrea Zonca's py.test plugin for collecting unit\ntests in the IPython notebooks (https://github.com/zonca/pytest-ipynb).\n\nThe original prototype was based on the template in\nhttps://gist.github.com/timo/2621679 and the code of a testing system\nfor notebooks https://gist.github.com/minrk/2620735 which we\nintegrated and mixed with the `py.test` system.\n\nWe acknowledge financial support from\n\n- OpenDreamKit Horizon 2020 European Research Infrastructures project (#676541), http://opendreamkit.org\n\n- EPSRC's Centre for Doctoral Training in Next Generation\n Computational Modelling, http://ngcm.soton.ac.uk (#EP/L015382/1) and\n EPSRC's Doctoral Training Centre in Complex System Simulation\n ((EP/G03690X/1),\n\n- The Gordon and Betty Moore Foundation through Grant GBMF #4856, by the\n Alfred P. Sloan Foundation and by the Helmsley Trust.\n\n\n## Authors\n\n2014 - 2017 David Cortes-Ortuno, Oliver Laslett, T. Kluyver, Vidar\nFauske, Maximilian Albert, MinRK, Ondrej Hovorka, Hans Fangohr\n",
"bugtrack_url": null,
"license": "",
"summary": "A py.test plugin to validate Jupyter notebooks",
"version": "0.11.0",
"project_urls": {
"Homepage": "https://github.com/computationalmodelling/nbval"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2c5ceb1e3ce54c4e94c7734b3831756c63f21badb3de91a98d77b9e23c0ca76a",
"md5": "06670f349623813934225abbd0f29f03",
"sha256": "307aecc866c9a1e8a13bb5bbb008a702bacfda2394dff6fe504a3108a58042a0"
},
"downloads": -1,
"filename": "nbval-0.11.0-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "06670f349623813934225abbd0f29f03",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": ">=3.7, <4",
"size": 24013,
"upload_time": "2024-03-04T14:36:57",
"upload_time_iso_8601": "2024-03-04T14:36:57.126881Z",
"url": "https://files.pythonhosted.org/packages/2c/5c/eb1e3ce54c4e94c7734b3831756c63f21badb3de91a98d77b9e23c0ca76a/nbval-0.11.0-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "28be22bd64d09e0cb53258f83b6fc455f05f18a78e3e5c109ccb6af42f1f49a2",
"md5": "81a36e6b98a3c04c54bb311c582f2504",
"sha256": "77c95797607b0a968babd2597ee3494102d25c3ad37435debbdac0e46e379094"
},
"downloads": -1,
"filename": "nbval-0.11.0.tar.gz",
"has_sig": false,
"md5_digest": "81a36e6b98a3c04c54bb311c582f2504",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7, <4",
"size": 62718,
"upload_time": "2024-03-04T14:36:58",
"upload_time_iso_8601": "2024-03-04T14:36:58.256896Z",
"url": "https://files.pythonhosted.org/packages/28/be/22bd64d09e0cb53258f83b6fc455f05f18a78e3e5c109ccb6af42f1f49a2/nbval-0.11.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-04 14:36:58",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "computationalmodelling",
"github_project": "nbval",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"appveyor": true,
"lcname": "nbval"
}
JSON
