[](./LICENSE) [](https://github.com/outerbounds/metaflow-card-notebook/actions/workflows/test.yaml) [](https://pypi.org/project/metaflow-card-notebook/) [](http://slack.outerbounds.co/)
# metaflow-card-notebook
> Use `@card(type='notebook')` to programatically run & render notebooks in your flows.
<!-- TOC -->
- [metaflow-card-notebook](#metaflow-card-notebook)
- [Installation](#installation)
- [Motivation](#motivation)
- [Usage](#usage)
- [Step 1: Prepare your notebook](#step-1-prepare-your-notebook)
- [Step 2: Prepare your flow with the notebook card](#step-2-prepare-your-flow-with-the-notebook-card)
- [Step 3: Prototype the rest of your notebook](#step-3-prototype-the-rest-of-your-notebook)
- [Step 4: Test the card](#step-4-test-the-card)
- [Customized Rendering](#customized-rendering)
- [Common Issues](#common-issues)
- [Papermill Arguments](#papermill-arguments)
- [Remote Execution](#remote-execution)
- [Dependency Management](#dependency-management)
- [Including Notebook Files In The Context](#including-notebook-files-in-the-context)
- [Examples](#examples)
<!-- /TOC -->
# Installation
```bash
pip install metaflow-card-notebook
```
# Motivation
You may have seen [this series of blog posts](https://netflixtechblog.com/notebook-innovation-591ee3221233) that have been written about Notebook Infrastructure at Netflix. Of particular interest is how notebooks [are programmatically run](https://netflixtechblog.com/scheduling-notebooks-348e6c14cfd6), often in DAGs, to generate reports and dashboards:
| Parameterized Execution of Notebooks | Notebooks in DAGs | Dependency Management & Scheduling |
| :----------------------------------: | :-----------------: | :--------------------------------: |
|  |  |  |
This way of generating reports and dashboards is very compelling, as it lets data scientists create content using environments and tools that they are familiar with. **With `@card(type='notebook')` you can programmatically run and render notebooks as part of a DAG.** This card allows you to accomplish the following:
- Run notebook(s) programmatically in your Metaflow DAGs.
- Access data from any step in your DAG so you can visualize it or otherwise use it to generate reports in a notebook.
- Render your notebooks as reports or model cards that can be embedded in various apps.
- Inject custom parameters into your notebook for execution.
- Ensure that notebook outputs are reproducible.
Additionally, you can use all of the features of Metaflow to manage the execution of notebooks, for example:
- Managing dependencies (ex: `@conda`)
- Requesting compute (ex: `@resources`)
- Parallel execution (ex: `foreach`)
- etc.
Here is an example of a dashboard generated by a notebook card:
>  _you can see real examples of flows that generate these dashboards in [examples](examples/)_
# Usage
## Step 1: Prepare your notebook
The notebook card injects the following five variables into your notebook:
1. `run_id`
2. `step name`
3. `task_id`
4. `flow_name`
5. `pathspec`
You can use these variables to retrieve the data you need from a flow. It is recommended that the first cell in your notebook defines these variables and that [you designate this cell with the tag "parameters"](https://papermill.readthedocs.io/en/latest/usage-parameterize.html).
For example of this, see [tests/nbflow.ipynb](tests/nbflow.ipynb):
> Note: in the example notebook these variables are set to `None` however, you can set these variables to real values based on flows that have been previously executed for prototyping.
## Step 2: Prepare your flow with the notebook card
You can render cards from notebooks using the `@card(type='notebook')` decorator on a step. For example, in [tests/nbflow.py](tests/nbflow.py), the notebook [tests/nbflow.ipynb](tests/nbflow.ipynb) is run and rendered programatically:
```python
from metaflow import step, current, FlowSpec, Parameter, card
class NBFlow(FlowSpec):
exclude_nb_input = Parameter('exclude_nb_input', default=True, type=bool)
@step
def start(self):
self.data_for_notebook = "I Will Print Myself From A Notebook"
self.next(self.end)
@card(type='notebook')
@step
def end(self):
self.nb_options_dict = dict(input_path='nbflow.ipynb', exclude_input=self.exclude_nb_input)
if __name__ == '__main__':
NBFlow()
```
Note how the `start` step stores some data that we want to access from a notebook later. We will discuss how to access this data from a notebook in the next step.
By default, a step that is decorated with `@card(type='notebook')` expects the variable `nb_options_dict` to be defined in the step. This variable is a dictionary of arguments that is passed to [papermill.execute.notebook](https://papermill.readthedocs.io/en/latest/reference/papermill-workflow.html#module-papermill.execute). Only the `input_path` argument is required. If `output_path` is absent, this is automatically set to `_rendered_<run_id>_<step_name>_<task_id>_<your_input_notebook_name>.ipynb`.
Furthermore, the `exclude_input` is an additional boolean argument that specifies whether or not to show our hide cell outputs, which is `False` by default.
## Step 3: Prototype the rest of your notebook
Recall that the `run_id`, `step_name`, `task_id`, `flow_name` and `pathspec` are injected into the notebook. We can access this in a notebook using [Metaflow's utlities for inspecting Flows and Results](https://docs.metaflow.org/metaflow/client). We demonstrate this in [tests/nbflow.ipynb](tests/nbflow.ipynb):
Some notes about this notebook:
- We recommend printing the variables injected into the notebook. This can help with debugging and provide an easy to locate lineage.
- We demonstrate how to access your flow's data via a `Step` or a `Task` object. You can read more about the relationship between these objects [in these docs](https://docs.metaflow.org/metaflow/client). In short, a `Task` is a child of a `Step` because a `Step` can have many tasks (for example if you use a `foreach` construct for parallelism).
- We recommend executing a run manually and prototyping the notebook by temporarily supplying the `run_id`, `flow_name`, etc to achieve the desired result.
## Step 4: Test the card
To test the card in the example outlined above, you must first run the flow (the parenthesis allows the commands to run in a subshell):
```bash
(cd tests && python nbflow.py run)
```
Then, render the card
```bash
(cd tests && python nbflow.py card view end)
```
By default, the cell inputs are hidden when the card is rendered. For learning purposes, it can be useful to render the card with the inputs to validate how the card is executed. You can do this by setting the `exclude_nb_input` parameter to `False` that was defined in the flow:
```bash
(cd tests && python nbflow.py run --exclude_nb_input=False && python nbflow.py card view end)
```
# Customized Rendering
The `@card(type='notebook')` is an opinionated way to execute and render notebooks with the tradeoff of requiring significantly less code. While some customization is possible by passing the appropriate arguments to `nb_options_dict` as listed in [papermill.execute.notebook](https://papermill.readthedocs.io/en/latest/reference/papermill-workflow.html#module-papermill.execute), you can achieve more fine-grained control by executing and rendering the notebook yourself and using the [html card](https://github.com/outerbounds/metaflow-card-html). We show an example of this in [examples/deep_learning/dl_flow.py](examples/deep_learning/dl_flow.py):
```py
@card(type='html')
@step
def nb_manual(self):
"""
Run & Render Jupyter Notebook Manually With The HTML Card.
Using the html card provides you greater control over notebook execution and rendering.
"""
import papermill as pm
output_nb_path = 'notebooks/rendered_Evaluate.ipynb'
output_html_path = output_nb_path.replace('.ipynb', '.html')
pm.execute_notebook('notebooks/Evaluate.ipynb',
output_nb_path,
parameters=dict(run_id=current.run_id,
flow_name=current.flow_name,)
)
run(f'jupyter nbconvert --to html --no-input --no-prompt {output_nb_path}')
with open(output_html_path, 'r') as f:
self.html = f.read()
self.next(self.end)
```
You can run the following command in your terminal the see output of this step(may take several minutes):
```bash
(cd example && python dl_flow.py run && python dl_flow.py card view nb_manual)
```
# Common Issues
## Papermill Arguments
Many issues can be resolved by providing the right arguments to [papermill.execute.notebook](https://papermill.readthedocs.io/en/latest/reference/papermill-workflow.html#module-papermill.execute). Below are some common issues and examples of how to resolve them:
1. **Kernel Name**: The name of the python kernel you use locally may be different from your remote execution environment. By default, papermill will attempt to find a kernel name in the metadata of your notebook, which is often automatically created when you select a kernel while running a notebook. You can use the `kernel_name` argument to specify a kernel. Below is an example:
```python
@card(type='notebook')
@step
def end(self):
self.nb_options_dict = dict(input_path='nbflow.ipynb', kernel_name='Python3')
```
2. **Working Directory**: The working directory may be important when your notebook is executed, especially if your notebooks rely on certain files or other assets. You can set the working directory the notebook is executed in with the `cwd` argument, for example, to set the working directory to `data/`:
```python
@card(type='notebook')
@step
def end(self):
self.nb_options_dict = dict(input_path='nbflow.ipynb', cwd='data/')
```
## Remote Execution
### Dependency Management
If you are running your flow remotely, for example [with `@batch`](https://docs.metaflow.org/metaflow/scaling#using-aws-batch-selectively-with-batch-decorator), you must remember to include the dependencies for this notebook card itself! One way to do this is using `pip` as illustrated below:
<!-- [[[cog
import cog
from configparser import ConfigParser
config = ConfigParser(delimiters=['='])
config.read('settings.ini')
cfg = config['DEFAULT']
requirements = cfg.get('requirements')
code_block = f"""
```python
@card(type='notebook')
@step
def end(self):
import os, sys
os.system(f"sys.executable -m pip {requirements}")
self.nb_options_dict = dict(input_path='nbflow.ipynb')
```
"""
cog.outl(code_block)
]]] -->
```python
@card(type='notebook')
@step
def end(self):
import os, sys
os.system(f"sys.executable -m pip ipykernel>=6.4.1 papermill>=2.3.3 nbconvert>=6.4.1 nbformat>=5.1.3")
self.nb_options_dict = dict(input_path='nbflow.ipynb')
```
<!-- [[[end]]] -->
Note: You can omit the `pip install` step above if your environment already includes all the dependendencies in your target environment listed in [settings.ini](/settings.ini). If you do omit `pip install`, make sure that you pin the correct version numbers as well.
### Including Notebook Files In The Context
If you are running steps remotely, you must ensure that your notebooks are uploaded to the remote environment with the cli argument `--package-suffixes=".ipynb"` For example, to execute [examples/deep_learning/dl_flow.py](examples/deep_learning/dl_flow.py) with this argument:
```bash
(cd example && python dl_flow.py --package-suffixes=".ipynb" run)
```
# Examples
We provide several examples of flows that contain the notebook card in [examples/](examples/).
Raw data
{
"_id": null,
"home_page": null,
"name": "metaflow-card-notebook",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": null,
"author": "Hamel Husain",
"author_email": "hamel@outerbounds.com",
"download_url": "https://files.pythonhosted.org/packages/75/11/1de62371bb58ea7363f9ae857b11f7cde65415e31aa26591ce3cf234b4f1/metaflow-card-notebook-1.0.8.tar.gz",
"platform": null,
"description": "[](./LICENSE) [](https://github.com/outerbounds/metaflow-card-notebook/actions/workflows/test.yaml) [](https://pypi.org/project/metaflow-card-notebook/) [](http://slack.outerbounds.co/)\n\n# metaflow-card-notebook\n\n> Use `@card(type='notebook')` to programatically run & render notebooks in your flows.\n\n<!-- TOC -->\n\n- [metaflow-card-notebook](#metaflow-card-notebook)\n- [Installation](#installation)\n- [Motivation](#motivation)\n- [Usage](#usage)\n - [Step 1: Prepare your notebook](#step-1-prepare-your-notebook)\n - [Step 2: Prepare your flow with the notebook card](#step-2-prepare-your-flow-with-the-notebook-card)\n - [Step 3: Prototype the rest of your notebook](#step-3-prototype-the-rest-of-your-notebook)\n - [Step 4: Test the card](#step-4-test-the-card)\n- [Customized Rendering](#customized-rendering)\n- [Common Issues](#common-issues)\n - [Papermill Arguments](#papermill-arguments)\n - [Remote Execution](#remote-execution)\n - [Dependency Management](#dependency-management)\n - [Including Notebook Files In The Context](#including-notebook-files-in-the-context)\n- [Examples](#examples)\n\n<!-- /TOC -->\n\n# Installation\n\n```bash\npip install metaflow-card-notebook\n```\n\n# Motivation\n\nYou may have seen [this series of blog posts](https://netflixtechblog.com/notebook-innovation-591ee3221233) that have been written about Notebook Infrastructure at Netflix. Of particular interest is how notebooks [are programmatically run](https://netflixtechblog.com/scheduling-notebooks-348e6c14cfd6), often in DAGs, to generate reports and dashboards:\n\n| Parameterized Execution of Notebooks | Notebooks in DAGs | Dependency Management & Scheduling |\n| :----------------------------------: | :-----------------: | :--------------------------------: |\n|  |  |  |\n\nThis way of generating reports and dashboards is very compelling, as it lets data scientists create content using environments and tools that they are familiar with. **With `@card(type='notebook')` you can programmatically run and render notebooks as part of a DAG.** This card allows you to accomplish the following:\n\n- Run notebook(s) programmatically in your Metaflow DAGs.\n- Access data from any step in your DAG so you can visualize it or otherwise use it to generate reports in a notebook.\n- Render your notebooks as reports or model cards that can be embedded in various apps.\n- Inject custom parameters into your notebook for execution.\n- Ensure that notebook outputs are reproducible.\n\nAdditionally, you can use all of the features of Metaflow to manage the execution of notebooks, for example:\n\n- Managing dependencies (ex: `@conda`)\n- Requesting compute (ex: `@resources`)\n- Parallel execution (ex: `foreach`)\n- etc.\n\nHere is an example of a dashboard generated by a notebook card:\n\n>  _you can see real examples of flows that generate these dashboards in [examples](examples/)_\n\n# Usage\n\n## Step 1: Prepare your notebook\n\nThe notebook card injects the following five variables into your notebook:\n\n1. `run_id`\n2. `step name`\n3. `task_id`\n4. `flow_name`\n5. `pathspec`\n\nYou can use these variables to retrieve the data you need from a flow. It is recommended that the first cell in your notebook defines these variables and that [you designate this cell with the tag \"parameters\"](https://papermill.readthedocs.io/en/latest/usage-parameterize.html).\n\nFor example of this, see [tests/nbflow.ipynb](tests/nbflow.ipynb):\n\n\n\n\n\n> Note: in the example notebook these variables are set to `None` however, you can set these variables to real values based on flows that have been previously executed for prototyping.\n\n## Step 2: Prepare your flow with the notebook card\n\nYou can render cards from notebooks using the `@card(type='notebook')` decorator on a step. For example, in [tests/nbflow.py](tests/nbflow.py), the notebook [tests/nbflow.ipynb](tests/nbflow.ipynb) is run and rendered programatically:\n\n```python\nfrom metaflow import step, current, FlowSpec, Parameter, card\n\nclass NBFlow(FlowSpec):\n\n exclude_nb_input = Parameter('exclude_nb_input', default=True, type=bool)\n\n @step\n def start(self):\n self.data_for_notebook = \"I Will Print Myself From A Notebook\"\n self.next(self.end)\n\n @card(type='notebook')\n @step\n def end(self):\n self.nb_options_dict = dict(input_path='nbflow.ipynb', exclude_input=self.exclude_nb_input)\n\nif __name__ == '__main__':\n NBFlow()\n```\n\nNote how the `start` step stores some data that we want to access from a notebook later. We will discuss how to access this data from a notebook in the next step.\n\nBy default, a step that is decorated with `@card(type='notebook')` expects the variable `nb_options_dict` to be defined in the step. This variable is a dictionary of arguments that is passed to [papermill.execute.notebook](https://papermill.readthedocs.io/en/latest/reference/papermill-workflow.html#module-papermill.execute). Only the `input_path` argument is required. If `output_path` is absent, this is automatically set to `_rendered_<run_id>_<step_name>_<task_id>_<your_input_notebook_name>.ipynb`.\n\nFurthermore, the `exclude_input` is an additional boolean argument that specifies whether or not to show our hide cell outputs, which is `False` by default.\n\n## Step 3: Prototype the rest of your notebook\n\nRecall that the `run_id`, `step_name`, `task_id`, `flow_name` and `pathspec` are injected into the notebook. We can access this in a notebook using [Metaflow's utlities for inspecting Flows and Results](https://docs.metaflow.org/metaflow/client). We demonstrate this in [tests/nbflow.ipynb](tests/nbflow.ipynb):\n\n\n\nSome notes about this notebook:\n\n- We recommend printing the variables injected into the notebook. This can help with debugging and provide an easy to locate lineage.\n- We demonstrate how to access your flow's data via a `Step` or a `Task` object. You can read more about the relationship between these objects [in these docs](https://docs.metaflow.org/metaflow/client). In short, a `Task` is a child of a `Step` because a `Step` can have many tasks (for example if you use a `foreach` construct for parallelism).\n- We recommend executing a run manually and prototyping the notebook by temporarily supplying the `run_id`, `flow_name`, etc to achieve the desired result.\n\n## Step 4: Test the card\n\nTo test the card in the example outlined above, you must first run the flow (the parenthesis allows the commands to run in a subshell):\n\n```bash\n(cd tests && python nbflow.py run)\n```\n\nThen, render the card\n\n```bash\n(cd tests && python nbflow.py card view end)\n```\n\nBy default, the cell inputs are hidden when the card is rendered. For learning purposes, it can be useful to render the card with the inputs to validate how the card is executed. You can do this by setting the `exclude_nb_input` parameter to `False` that was defined in the flow:\n\n```bash\n(cd tests && python nbflow.py run --exclude_nb_input=False && python nbflow.py card view end)\n```\n\n# Customized Rendering\n\nThe `@card(type='notebook')` is an opinionated way to execute and render notebooks with the tradeoff of requiring significantly less code. While some customization is possible by passing the appropriate arguments to `nb_options_dict` as listed in [papermill.execute.notebook](https://papermill.readthedocs.io/en/latest/reference/papermill-workflow.html#module-papermill.execute), you can achieve more fine-grained control by executing and rendering the notebook yourself and using the [html card](https://github.com/outerbounds/metaflow-card-html). We show an example of this in [examples/deep_learning/dl_flow.py](examples/deep_learning/dl_flow.py):\n\n```py\n @card(type='html')\n @step\n def nb_manual(self):\n \"\"\"\n Run & Render Jupyter Notebook Manually With The HTML Card.\n\n Using the html card provides you greater control over notebook execution and rendering.\n \"\"\"\n import papermill as pm\n output_nb_path = 'notebooks/rendered_Evaluate.ipynb'\n output_html_path = output_nb_path.replace('.ipynb', '.html')\n\n pm.execute_notebook('notebooks/Evaluate.ipynb',\n output_nb_path,\n parameters=dict(run_id=current.run_id,\n flow_name=current.flow_name,)\n )\n run(f'jupyter nbconvert --to html --no-input --no-prompt {output_nb_path}')\n with open(output_html_path, 'r') as f:\n self.html = f.read()\n self.next(self.end)\n```\n\nYou can run the following command in your terminal the see output of this step(may take several minutes):\n\n```bash\n(cd example && python dl_flow.py run && python dl_flow.py card view nb_manual)\n```\n\n# Common Issues\n\n## Papermill Arguments\n\nMany issues can be resolved by providing the right arguments to [papermill.execute.notebook](https://papermill.readthedocs.io/en/latest/reference/papermill-workflow.html#module-papermill.execute). Below are some common issues and examples of how to resolve them:\n\n1. **Kernel Name**: The name of the python kernel you use locally may be different from your remote execution environment. By default, papermill will attempt to find a kernel name in the metadata of your notebook, which is often automatically created when you select a kernel while running a notebook. You can use the `kernel_name` argument to specify a kernel. Below is an example:\n\n```python\n @card(type='notebook')\n @step\n def end(self):\n self.nb_options_dict = dict(input_path='nbflow.ipynb', kernel_name='Python3')\n```\n\n2. **Working Directory**: The working directory may be important when your notebook is executed, especially if your notebooks rely on certain files or other assets. You can set the working directory the notebook is executed in with the `cwd` argument, for example, to set the working directory to `data/`:\n\n```python\n @card(type='notebook')\n @step\n def end(self):\n self.nb_options_dict = dict(input_path='nbflow.ipynb', cwd='data/')\n```\n\n## Remote Execution\n\n### Dependency Management\n\nIf you are running your flow remotely, for example [with `@batch`](https://docs.metaflow.org/metaflow/scaling#using-aws-batch-selectively-with-batch-decorator), you must remember to include the dependencies for this notebook card itself! One way to do this is using `pip` as illustrated below:\n\n<!-- [[[cog\nimport cog\nfrom configparser import ConfigParser\nconfig = ConfigParser(delimiters=['='])\nconfig.read('settings.ini')\ncfg = config['DEFAULT']\nrequirements = cfg.get('requirements')\n\ncode_block = f\"\"\"\n```python\n @card(type='notebook')\n @step\n def end(self):\n import os, sys\n os.system(f\"sys.executable -m pip {requirements}\")\n self.nb_options_dict = dict(input_path='nbflow.ipynb')\n```\n\"\"\"\ncog.outl(code_block)\n]]] -->\n\n```python\n @card(type='notebook')\n @step\n def end(self):\n import os, sys\n os.system(f\"sys.executable -m pip ipykernel>=6.4.1 papermill>=2.3.3 nbconvert>=6.4.1 nbformat>=5.1.3\")\n self.nb_options_dict = dict(input_path='nbflow.ipynb')\n```\n\n<!-- [[[end]]] -->\n\nNote: You can omit the `pip install` step above if your environment already includes all the dependendencies in your target environment listed in [settings.ini](/settings.ini). If you do omit `pip install`, make sure that you pin the correct version numbers as well.\n\n### Including Notebook Files In The Context\n\nIf you are running steps remotely, you must ensure that your notebooks are uploaded to the remote environment with the cli argument `--package-suffixes=\".ipynb\"` For example, to execute [examples/deep_learning/dl_flow.py](examples/deep_learning/dl_flow.py) with this argument:\n\n```bash\n(cd example && python dl_flow.py --package-suffixes=\".ipynb\" run)\n```\n\n# Examples\n\nWe provide several examples of flows that contain the notebook card in [examples/](examples/).\n",
"bugtrack_url": null,
"license": "Apache Software License 2.0",
"summary": "Render Jupyter Notebooks in Metaflow Cards",
"version": "1.0.8",
"project_urls": null,
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "86610c2a9968e16021d188096991420f8f773b8b0986d0703c76a59b5d20a120",
"md5": "5603457816091e0ab18b3efb429e831f",
"sha256": "0f3f7283951add212305450821f3496fb31ab255c9e082cc29ad1d08a940f596"
},
"downloads": -1,
"filename": "metaflow_card_notebook-1.0.8-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "5603457816091e0ab18b3efb429e831f",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": null,
"size": 11145,
"upload_time": "2024-09-26T21:23:52",
"upload_time_iso_8601": "2024-09-26T21:23:52.436653Z",
"url": "https://files.pythonhosted.org/packages/86/61/0c2a9968e16021d188096991420f8f773b8b0986d0703c76a59b5d20a120/metaflow_card_notebook-1.0.8-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "75111de62371bb58ea7363f9ae857b11f7cde65415e31aa26591ce3cf234b4f1",
"md5": "11400c3fcfae66bf2b006322ab50e376",
"sha256": "e06cb2acdd9775ecd242f554e0de793b1a7a38011ac10282edb48ba3b81a3df8"
},
"downloads": -1,
"filename": "metaflow-card-notebook-1.0.8.tar.gz",
"has_sig": false,
"md5_digest": "11400c3fcfae66bf2b006322ab50e376",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 10868,
"upload_time": "2024-09-26T21:23:53",
"upload_time_iso_8601": "2024-09-26T21:23:53.719642Z",
"url": "https://files.pythonhosted.org/packages/75/11/1de62371bb58ea7363f9ae857b11f7cde65415e31aa26591ce3cf234b4f1/metaflow-card-notebook-1.0.8.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-26 21:23:53",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "metaflow-card-notebook"
}
JSON