| Name | nbmake JSON |
| Version |
1.5.5
JSON |
| download |
| home_page | None |
| Summary | Pytest plugin for testing notebooks |
| upload_time | 2024-12-23 18:33:46 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.8.0 |
| license | Apache-2.0 |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# nbmake
[](https://discord.gg/QFjCpMjqRY)
[](https://codecov.io/gh/treebeardtech/nbmake)
[](https://pypi.org/project/nbmake)
[](https://pypi.org/project/nbmake)
[](https://pypi.org/project/nbmake)
**What?** Pytest plugin for testing and releasing notebook documentation
**Why?** To raise the quality of scientific material through better automation
**Who is this for?** Research/Machine Learning Software Engineers who maintain packages/teaching materials with documentation written in notebooks.
## Functionality
1. Executes notebooks using pytest and nbclient, allowing parallel notebook testing
2. Optionally writes back to the repo, allowing faster building of [nbsphinx](https://github.com/spatialaudio/nbsphinx) or [jupyter book](https://github.com/executablebooks/jupyter-book) docs
## Quick Start
If you have a notebook that runs interactively using an ipython kernel,
you can try testing it automatically as follows:
```sh
pip install pytest nbmake
pytest --nbmake **/*ipynb
```
## Configure Cell Timeouts
You can configure the cell timeout with the following pytest flag:
```sh
pytest --nbmake --nbmake-timeout=3000 # allows each cell 3000 seconds to finish
```
## Allow Errors For a Whole Notebook
This configuration must be placed in the notebook's **top-level metadata** (not cell-level metadata).
Your notebook should look like this:
```json
{
"cells": [ ... ],
"metadata": {
"kernelspec": { ... },
"execution": {
"allow_errors": true,
"timeout": 300
}
}
}
```
## Allow a Cell to Throw an Exception
A cell with the following metadata can throw an exception without failing the test:
```json
"metadata": {
"tags": [
"raises-exception"
]
}
```
## Ignore a Code Cell
A cell with the following metadata will not be executed by nbmake
```json
{
"language": "python",
"custom": {
"metadata": {
"tags": [
"skip-execution"
]
}
}
}
```
## Override Notebook Kernels when Testing
Regardless of the kernel configured in the notebook JSON, you can force nbmake to use a specific kernel when testing:
```
pytest --nbmake --nbmake-kernel=mycustomkernel
```
## Add Missing Jupyter Kernel to Your CI Environment
If you are not using the flag above and are using a kernel name other than the default ‘python3’, you will see an error message when executing your notebooks in a fresh CI environment: `Error - No such kernel: 'mycustomkernel'`
Use ipykernel to install the custom kernel:
```sh
python -m ipykernel install --user --name mycustomkernel
```
If you are using another language such as c++ in your notebooks, you may have a different process for installing your kernel.
## Parallelisation
For repos containing a large number of notebooks that run slowly, you can run each notebook
in parallel using `pytest-xdist`.
```sh
pip install pytest-xdist
pytest --nbmake -n=auto
```
It is also possible to parallelise at a CI-level using strategies, see [example](https://github.com/LabForComputationalVision/plenoptic/blob/master/.github/workflows/treebeard.yml)
### Build Jupyter Books Faster
Using xdist and the `--overwrite` flag let you build a large jupyter book repo faster:
```sh
pytest --nbmake --overwrite -n=auto examples
jb build examples
```
## Find missing imports in a directory of failing notebooks
It's not always feasible to get notebooks running from top to bottom from the start.
You can however, use nbmake to check that there are no `ModuleNotFoundError`s:
```sh
pytest \
--nbmake \
--nbmake-find-import-errors \ # Ignore all errors except ModuleNotFoundError
--nbmake-timeout=20 # Skip past cells longer than 20s
```
## Mock out variables to simplify testing
If your notebook runs a training process that takes a long time to run, you can use nbmake's
mocking feature to overwrite variables after a cell runs:
```json
{
"cells": [
...,
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbmake": {
"mock": {
// these keys will override global variables after this cell runs
"epochs": 2,
"config": "/test/config.json",
"args": {
"env": "test"
}
}
}
},
"outputs": [],
"source": [
"epochs = 10\n",
"..."
]
},
...
],
...
}
```
## Run test logic after a cell executes
You can fetch CI secrets and run assertions after any cell by putting scripts in the cell metadata under `nbmake.post_cell_execute`:
```json
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbmake": {
"post_cell_execute": [
"y = 3",
"z = x+y"
]
}
},
"outputs": [],
"source": [
"x = 1\n",
"y = 2\n",
"z = 0\n",
"# this cell has a post_cell_execute that assigns y and z"
]
},
```
## Advice on Usage
nbmake is best used in a scenario where you use the ipynb files only for development. Consumption of notebooks is primarily done via a docs site, built through jupyter book, nbsphinx, or some other means. If using one of these tools, you are able to write assertion code in cells which will be [hidden from readers](https://jupyterbook.org/interactive/hiding.html).
### Pre-commit
Treating notebooks like source files lets you keep your repo minimal. Some tools, such as plotly may drop several megabytes of javascript in your output cells, as a result, stripping out notebooks on pre-commit is advisable:
```
# .pre-commit-config.yaml
repos:
- repo: https://github.com/kynan/nbstripout
rev: master
hooks:
- id: nbstripout
```
See https://pre-commit.com/ for more...
## Disable Nbmake
Implicitly:
```
pytest
```
Explicitly:
```
pytest -p no:nbmake
```
## See Also:
* A more in-depth [intro to nbmake](https://semaphoreci.com/blog/test-jupyter-notebooks-with-pytest-and-nbmake) running on Semaphore CI
* [nbmake action](https://github.com/treebeardtech/treebeard)
* [pytest](https://pytest.org/)
* [jupyter book](https://github.com/executablebooks/jupyter-book)
* [jupyter cache](https://github.com/executablebooks/jupyter-cache)
* [MyST-NB](https://github.com/executablebooks/MyST-NB)
Raw data
{
"_id": null,
"home_page": null,
"name": "nbmake",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8.0",
"maintainer_email": null,
"keywords": null,
"author": null,
"author_email": "alex-treebeard <alex@treebeard.io>",
"download_url": "https://files.pythonhosted.org/packages/43/9a/aae201cee5639e1d562b3843af8fd9f8d018bb323e776a2b973bdd5fc64b/nbmake-1.5.5.tar.gz",
"platform": null,
"description": "# nbmake\n[](https://discord.gg/QFjCpMjqRY)\n[](https://codecov.io/gh/treebeardtech/nbmake)\n[](https://pypi.org/project/nbmake)\n[](https://pypi.org/project/nbmake)\n[](https://pypi.org/project/nbmake)\n\n**What?** Pytest plugin for testing and releasing notebook documentation\n\n**Why?** To raise the quality of scientific material through better automation\n\n**Who is this for?** Research/Machine Learning Software Engineers who maintain packages/teaching materials with documentation written in notebooks.\n\n## Functionality\n\n1. Executes notebooks using pytest and nbclient, allowing parallel notebook testing\n2. Optionally writes back to the repo, allowing faster building of [nbsphinx](https://github.com/spatialaudio/nbsphinx) or [jupyter book](https://github.com/executablebooks/jupyter-book) docs\n\n## Quick Start\n\nIf you have a notebook that runs interactively using an ipython kernel,\nyou can try testing it automatically as follows:\n\n```sh\npip install pytest nbmake\npytest --nbmake **/*ipynb\n```\n\n## Configure Cell Timeouts\n\nYou can configure the cell timeout with the following pytest flag:\n\n```sh\npytest --nbmake --nbmake-timeout=3000 # allows each cell 3000 seconds to finish\n```\n\n## Allow Errors For a Whole Notebook\n\nThis configuration must be placed in the notebook's **top-level metadata** (not cell-level metadata).\n\nYour notebook should look like this:\n\n```json\n{\n \"cells\": [ ... ],\n \"metadata\": {\n \"kernelspec\": { ... },\n \"execution\": {\n \"allow_errors\": true,\n \"timeout\": 300\n }\n }\n}\n```\n\n## Allow a Cell to Throw an Exception\n\nA cell with the following metadata can throw an exception without failing the test:\n\n```json\n \"metadata\": {\n \"tags\": [\n \"raises-exception\"\n ]\n }\n```\n\n## Ignore a Code Cell\n\nA cell with the following metadata will not be executed by nbmake\n\n```json\n{\n \"language\": \"python\",\n \"custom\": {\n \"metadata\": {\n \"tags\": [\n \"skip-execution\"\n ]\n }\n }\n}\n```\n\n## Override Notebook Kernels when Testing\n\nRegardless of the kernel configured in the notebook JSON, you can force nbmake to use a specific kernel when testing:\n\n```\npytest --nbmake --nbmake-kernel=mycustomkernel\n```\n\n## Add Missing Jupyter Kernel to Your CI Environment\n\nIf you are not using the flag above and are using a kernel name other than the default \u2018python3\u2019, you will see an error message when executing your notebooks in a fresh CI environment: `Error - No such kernel: 'mycustomkernel'`\n\nUse ipykernel to install the custom kernel:\n\n```sh\npython -m ipykernel install --user --name mycustomkernel\n```\n\nIf you are using another language such as c++ in your notebooks, you may have a different process for installing your kernel.\n\n## Parallelisation\n\nFor repos containing a large number of notebooks that run slowly, you can run each notebook\nin parallel using `pytest-xdist`.\n\n```sh\npip install pytest-xdist\n\npytest --nbmake -n=auto\n```\n\nIt is also possible to parallelise at a CI-level using strategies, see [example](https://github.com/LabForComputationalVision/plenoptic/blob/master/.github/workflows/treebeard.yml)\n\n### Build Jupyter Books Faster\n\nUsing xdist and the `--overwrite` flag let you build a large jupyter book repo faster:\n\n```sh\npytest --nbmake --overwrite -n=auto examples\njb build examples\n```\n\n## Find missing imports in a directory of failing notebooks\n\nIt's not always feasible to get notebooks running from top to bottom from the start.\n\nYou can however, use nbmake to check that there are no `ModuleNotFoundError`s:\n\n```sh\npytest \\\n --nbmake \\\n --nbmake-find-import-errors \\ # Ignore all errors except ModuleNotFoundError\n --nbmake-timeout=20 # Skip past cells longer than 20s\n```\n\n## Mock out variables to simplify testing\n\nIf your notebook runs a training process that takes a long time to run, you can use nbmake's\nmocking feature to overwrite variables after a cell runs:\n\n```json\n{\n \"cells\": [\n ...,\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"nbmake\": {\n \"mock\": {\n // these keys will override global variables after this cell runs\n \"epochs\": 2,\n \"config\": \"/test/config.json\",\n \"args\": {\n \"env\": \"test\"\n }\n }\n }\n },\n \"outputs\": [],\n \"source\": [\n \"epochs = 10\\n\",\n \"...\"\n ]\n },\n ...\n ],\n ...\n}\n```\n\n## Run test logic after a cell executes\n\nYou can fetch CI secrets and run assertions after any cell by putting scripts in the cell metadata under `nbmake.post_cell_execute`:\n\n```json\n{\n \"cells\": [\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"nbmake\": {\n \"post_cell_execute\": [\n \"y = 3\",\n \"z = x+y\"\n ]\n }\n },\n \"outputs\": [],\n \"source\": [\n \"x = 1\\n\",\n \"y = 2\\n\",\n \"z = 0\\n\",\n \"# this cell has a post_cell_execute that assigns y and z\"\n ]\n },\n```\n\n## Advice on Usage\n\nnbmake is best used in a scenario where you use the ipynb files only for development. Consumption of notebooks is primarily done via a docs site, built through jupyter book, nbsphinx, or some other means. If using one of these tools, you are able to write assertion code in cells which will be [hidden from readers](https://jupyterbook.org/interactive/hiding.html).\n\n### Pre-commit\n\nTreating notebooks like source files lets you keep your repo minimal. Some tools, such as plotly may drop several megabytes of javascript in your output cells, as a result, stripping out notebooks on pre-commit is advisable:\n\n```\n# .pre-commit-config.yaml\nrepos:\n - repo: https://github.com/kynan/nbstripout\n rev: master\n hooks:\n - id: nbstripout\n```\n\nSee https://pre-commit.com/ for more...\n\n## Disable Nbmake\n\nImplicitly:\n```\npytest\n```\n\nExplicitly:\n```\npytest -p no:nbmake\n```\n\n## See Also:\n\n* A more in-depth [intro to nbmake](https://semaphoreci.com/blog/test-jupyter-notebooks-with-pytest-and-nbmake) running on Semaphore CI\n* [nbmake action](https://github.com/treebeardtech/treebeard)\n* [pytest](https://pytest.org/)\n* [jupyter book](https://github.com/executablebooks/jupyter-book)\n* [jupyter cache](https://github.com/executablebooks/jupyter-cache)\n* [MyST-NB](https://github.com/executablebooks/MyST-NB)\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Pytest plugin for testing notebooks",
"version": "1.5.5",
"project_urls": {
"Homepage": "https://github.com/treebeardtech/nbmake"
},
"split_keywords": [],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "ebbeb257e12f9710819fde40adc972578bee6b72c5992da1bc8369bef2597756",
"md5": "5ea654a06161f9f90d1ba451a962ce60",
"sha256": "c6fbe6e48b60cacac14af40b38bf338a3b88f47f085c54ac5b8639ff0babaf4b"
},
"downloads": -1,
"filename": "nbmake-1.5.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5ea654a06161f9f90d1ba451a962ce60",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8.0",
"size": 12818,
"upload_time": "2024-12-23T18:33:44",
"upload_time_iso_8601": "2024-12-23T18:33:44.566087Z",
"url": "https://files.pythonhosted.org/packages/eb/be/b257e12f9710819fde40adc972578bee6b72c5992da1bc8369bef2597756/nbmake-1.5.5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "439aaae201cee5639e1d562b3843af8fd9f8d018bb323e776a2b973bdd5fc64b",
"md5": "f75c5e80444803e4128cc18b8761e659",
"sha256": "239dc868ea13a7c049746e2aba2c229bd0f6cdbc6bfa1d22f4c88638aa4c5f5c"
},
"downloads": -1,
"filename": "nbmake-1.5.5.tar.gz",
"has_sig": false,
"md5_digest": "f75c5e80444803e4128cc18b8761e659",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8.0",
"size": 85929,
"upload_time": "2024-12-23T18:33:46",
"upload_time_iso_8601": "2024-12-23T18:33:46.774069Z",
"url": "https://files.pythonhosted.org/packages/43/9a/aae201cee5639e1d562b3843af8fd9f8d018bb323e776a2b973bdd5fc64b/nbmake-1.5.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-23 18:33:46",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "treebeardtech",
"github_project": "nbmake",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "nbmake"
}
