# jupyterlab-pachyderm
[](https://circleci.com/gh/pachyderm/jupyterlab-pachyderm/tree/main)
[](https://pypi.org/project/jupyterlab-pachyderm)
A JupyterLab extension for integrations with Pachyderm.
This extension is composed of a Python package named `jupyterlab_pachyderm`
for the server extension and a NPM package named `jupyterlab-pachyderm`
for the frontend extension.
# Requirements
- JupyterLab >= 3.0
- Python >=3.8,<4
- [pyenv](https://github.com/pyenv/pyenv) is a great way to manage and install different versions of python. You can check which version you are using by running `pyenv versions`. Our Python extension is built to be compatible with Python versions 3.8 to 3.10. Therefore, it is best to run the lowest version (3.8.x) for highest compatibility.
- Node
- If you are using [nvm](https://github.com/nvm-sh/nvm) first run `nvm install`. This will install and switch the version of node to the one defined in the `.nvmrc`. If you are upgrading the version of node used in the project, please check and make sure that the versions defined in the `.nvmrc`.
# Development Environment
There are two ways you can work on this project. One is setting up a local python virtual environment and the other is using a docker image developing on a virtual machine.
## Dev Container Workflow
Building and running the extension in a Docker container allows us to use mount, which is not possible in recent versions of macOS.
Assuming that you are running a Pachyderm instance somewhere, and that you can port-forward `pachd`
```
kubectl port-forward service/pachd 30650:30650
```
Start a bash session in the `pachyderm/notebooks-user` container from the top-level `jupyterlab-pachyderm` repo.
```
docker run --name jupyterlab_pachyderm_frontend_dev \
-p 8888:8888 \
-it -e GRANT_SUDO=yes --user root \
--device /dev/fuse --privileged \
-v $(pwd):/home/jovyan/extension-wd \
-w /home/jovyan/extension-wd \
pachyderm/notebooks-user:<latest master SHA> \
bash
```
If you are running the frontend container on Linux, and want to be able to talk to pachd in minikube on `grpc://localhost:30650`, use `--net=host` at the start of the the `docker run` command as shown above. If you are on macOS, you will want to remove `--net=host` as it is not supported. On macOS you can specify `grpc://host.docker.internal:30650` to get the same effect.
Install the project in editable mode, and start JupyterLab
```
pip install -e ".[dev]"
jupyter labextension develop --overwrite
# Server extension must be manually installed in develop mode, for example
jupyter server extension enable jupyterlab_pachyderm
jupyter lab --allow-root
```
Open another bash inside the same container:
```
docker exec -it <container-id> bash
```
Within container run:
```
npm run watch
```
Note: Once run above, you can start the dev container again in future sessions using
```
docker start <container id>
```
and then running
```
docker exec -it <container id> bash
```
Iterating on the mount server, from inside a `pachyderm` checkout:
```
CGO_ENABLED=0 make install
docker cp /home/luke/gocode/bin/pachctl jupyterlab_pachyderm_frontend_dev:/usr/local/bin/pachctl
docker exec -ti jupyterlab_pachyderm_frontend_dev pkill -f pachctl
```
## Local Virtual Environment Setup
When developing in python, it is good practice to set up a virtual environment. A simple guid to set up a virtual environment is as follows:
create a virtual environment using venv
`python -m venv venv`
Activate the environment
`source venv/bin/activate`
When you are done using the environment you can close your shell or deactivate the environment:
`deactivate`
## Development install
Note: You will need NodeJS to build the extension package.
```bash
# Clone the repo to your local environment
# Change directory to the jupyterlab-pachyderm directory (top level of repo)
# Make sure you are using a virtual environment
# Install package in development mode
pip install -e ".[dev]"
# Link your development version of the extension with JupyterLab
jupyter labextension develop . --overwrite
# Server extension must be manually installed in develop mode
jupyter server extension enable jupyterlab_pachyderm
# Rebuild extension Typescript source after making changes
npm run build
```
You can watch the source directory and run JupyterLab at the same time in different terminals to watch for changes in the extension's source and automatically rebuild the extension.
```bash
# Watch the source directory in one terminal, automatically rebuilding when needed
npm run watch
# Run JupyterLab in another terminal
jupyter lab --allow-root --ip=0.0.0.0
```
With the watch command running, every saved change will immediately be built locally and available in your running JupyterLab. Refresh JupyterLab to load the change in your browser (you may need to wait several seconds for the extension to be rebuilt).
By default, the `npm run build` command generates the source maps for this extension to make it easier to debug using the browser dev tools. To also generate source maps for the JupyterLab core extensions, you can run the following command:
```bash
jupyter lab build --minimize=False
```
Note that to enable debug logging for `jupyter` (which also enables it for our extension backend, a python plugin to the JupyterLab server), you can run:
```bash
jupyter lab --debug
```
## Development uninstall
```bash
# Server extension must be manually disabled in develop mode
jupyter server extension disable jupyterlab_pachyderm
pip uninstall jupyterlab_pachyderm
```
In development mode, you will also need to remove the symlink created by `jupyter labextension develop`
command. To find its location, you can run `jupyter labextension list` to figure out where the `labextensions`
folder is located. Then you can remove the symlink named `jupyterlab-pachyderm` within that folder.
## Locally building the docker image
Useful if iterating on the Dockerfile locally or iterating on changes to a version of mount-server.
Create & activate venv:
```
python3 -m venv venv
source venv/bin/activate
```
Build `dist` directory:
```
python -m pip install --upgrade pip
python -m pip install -r ci-requirements.txt
python -m build
```
Build docker image:
```
export PACHCTL_VERSION=aaa7434c714fab6130c3982ebdaa8f279bd850c2 # or whichever version you want
docker build --build-arg PACHCTL_VERSION=$PACHCTL_VERSION -t pachyderm/notebooks-user:dev .
docker run -ti -p 8888:8888 pachyderm/notebooks-user:dev
```
Navigate to the URL that's printed out by the docker container, then change `tree` to `lab` in your browser's address bar.
## Install
To install the extension, execute:
```bash
pip install jupyterlab_pachyderm
```
## Uninstall
To remove the extension, execute:
```bash
pip uninstall jupyterlab_pachyderm
```
## Server extension
First make sure the server extension is enabled:
```
jupyter server extension list 2>&1 | grep -ie "jupyterlab_pachyderm.*OK"
```
# Install Jupyterhub locally (Sidecar mode testing)
```
helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/
helm repo update
helm install jhub jupyterhub/jupyterhub --version 2.0.0 --values scripts/jhub-dev-helm-values.yml
```
Note: Use any username and no password to login
### API endpoints
```
GET /repos # returns a list of all repos/branches
GET /mounts # returns a list of all active mounts and unmounted repos/branches
PUT /_mount # mounts a single repo
PUT /_unmount # unmounts a single repo
PUT /_commit # commits any changes to the repo
PUT /_unmount_all # unmounts all repos
PUT /datums/_mount # mounts first datum of given input spec
PUT /datums/_next # mounts next datum
PUT /datums/_prev # mounts prev datum
GET /datums # returns info on mounted datum
```
The servers-side extension extends jupyter server, so it automatically starts as part of `jupyter lab`.
API endpoints can be accessed via `localhost:8888/pachyderm/v2`
The frontend can access the endpoints via `/v2`, for example:
```js
requestAPI<any>('/v2/repos')
.then(data => {
console.log(data);
})
.catch(reason => {
console.error(reason);
});
```
You can also access it via `localhost:8888/v2`
## SVG Images
We are leveraging [svgr](https://react-svgr.com/) to simplify the use of non icon svgs in in the project. Svg images that are to be converted live in the `svg-images` folder and get output to the `src/utils/components/Svgs` folder. If you want to add a new image to the project simply add the svg to the `svg-images` folder and run `npm run build:svg`. We have spent some time trying to get svgr to work through the `@svgr/webpack` plugin but have not been successful as of yet.
# Project Structure
Jupyter extensions are composed of several plugins. These plugins can be selectively enabled or disabled. Because of this we have decided separate the functionality in the extension using plugins. Plugins exported in this extension are as follows.
### Hub
This plugin contains custom styling and other features only used by hub. By default this extension is disabled.
### Mount
This plugin contains the mount feature currently under development.
## Plugin settings
You can disable certain plugins by specifying the following config data in the `<jupyter_config_path>/labconfig/page_config.json`:
```
{
"disabledExtensions": {
"jupyterlab-pachyderm:examples": true
}
}
```
You can check your config paths by running `jupyter --paths`. Setting this config file is not part of the built extension and needs to be done by the user.
Adding the following to the package.json in the `jupyterlab` object will disable the examples plugin by default.
```
"disabledExtensions": ["jupyterlab-pachyderm:examples"]
```
So we can build the extension with hub features turned off and override the setting for hub.
## Troubleshoot
If you are seeing the frontend extension, but it is not working, check
that the server extension is enabled:
```bash
jupyter server extension list
```
If the server extension is installed and enabled, but you are not seeing
the frontend extension, check the frontend extension is installed:
```bash
jupyter labextension list
```
Raw data
{
"_id": null,
"home_page": "https://github.com/pachyderm/pachyderm",
"name": "jupyterlab-pachyderm",
"maintainer": null,
"docs_url": null,
"requires_python": "<4,>=3.8",
"maintainer_email": null,
"keywords": "jupyter, jupyterlab, jupyterlab-extension",
"author": null,
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/0e/b4/c77ca5ba89b20ebf28e8526c1d16dfa6257bd80986af297097332ff476c8/jupyterlab_pachyderm-2.9.5.tar.gz",
"platform": "Linux",
"description": "# jupyterlab-pachyderm\n\n[](https://circleci.com/gh/pachyderm/jupyterlab-pachyderm/tree/main)\n[](https://pypi.org/project/jupyterlab-pachyderm)\n\nA JupyterLab extension for integrations with Pachyderm.\n\nThis extension is composed of a Python package named `jupyterlab_pachyderm`\nfor the server extension and a NPM package named `jupyterlab-pachyderm`\nfor the frontend extension.\n\n# Requirements\n\n- JupyterLab >= 3.0\n\n- Python >=3.8,<4\n - [pyenv](https://github.com/pyenv/pyenv) is a great way to manage and install different versions of python. You can check which version you are using by running `pyenv versions`. Our Python extension is built to be compatible with Python versions 3.8 to 3.10. Therefore, it is best to run the lowest version (3.8.x) for highest compatibility.\n\n- Node\n - If you are using [nvm](https://github.com/nvm-sh/nvm) first run `nvm install`. This will install and switch the version of node to the one defined in the `.nvmrc`. If you are upgrading the version of node used in the project, please check and make sure that the versions defined in the `.nvmrc`.\n\n\n# Development Environment\n\nThere are two ways you can work on this project. One is setting up a local python virtual environment and the other is using a docker image developing on a virtual machine.\n\n## Dev Container Workflow\n\nBuilding and running the extension in a Docker container allows us to use mount, which is not possible in recent versions of macOS.\n\nAssuming that you are running a Pachyderm instance somewhere, and that you can port-forward `pachd`\n\n```\nkubectl port-forward service/pachd 30650:30650\n```\n\nStart a bash session in the `pachyderm/notebooks-user` container from the top-level `jupyterlab-pachyderm` repo.\n\n```\ndocker run --name jupyterlab_pachyderm_frontend_dev \\\n -p 8888:8888 \\\n -it -e GRANT_SUDO=yes --user root \\\n --device /dev/fuse --privileged \\\n -v $(pwd):/home/jovyan/extension-wd \\\n -w /home/jovyan/extension-wd \\\n pachyderm/notebooks-user:<latest master SHA> \\\n bash\n```\n\nIf you are running the frontend container on Linux, and want to be able to talk to pachd in minikube on `grpc://localhost:30650`, use `--net=host` at the start of the the `docker run` command as shown above. If you are on macOS, you will want to remove `--net=host` as it is not supported. On macOS you can specify `grpc://host.docker.internal:30650` to get the same effect.\n\nInstall the project in editable mode, and start JupyterLab\n\n```\npip install -e \".[dev]\"\njupyter labextension develop --overwrite\n\n# Server extension must be manually installed in develop mode, for example\njupyter server extension enable jupyterlab_pachyderm\n\njupyter lab --allow-root\n```\n\nOpen another bash inside the same container:\n\n```\ndocker exec -it <container-id> bash\n```\n\nWithin container run:\n\n```\nnpm run watch\n```\n\nNote: Once run above, you can start the dev container again in future sessions using\n\n```\ndocker start <container id>\n```\n\nand then running\n\n```\ndocker exec -it <container id> bash\n```\n\nIterating on the mount server, from inside a `pachyderm` checkout:\n\n```\nCGO_ENABLED=0 make install\ndocker cp /home/luke/gocode/bin/pachctl jupyterlab_pachyderm_frontend_dev:/usr/local/bin/pachctl\ndocker exec -ti jupyterlab_pachyderm_frontend_dev pkill -f pachctl\n```\n\n## Local Virtual Environment Setup \nWhen developing in python, it is good practice to set up a virtual environment. A simple guid to set up a virtual environment is as follows:\ncreate a virtual environment using venv\n\n`python -m venv venv`\n\nActivate the environment\n\n`source venv/bin/activate`\n\nWhen you are done using the environment you can close your shell or deactivate the environment: \n`deactivate`\n\n## Development install\n\nNote: You will need NodeJS to build the extension package.\n\n```bash\n# Clone the repo to your local environment\n# Change directory to the jupyterlab-pachyderm directory (top level of repo)\n# Make sure you are using a virtual environment\n# Install package in development mode\npip install -e \".[dev]\"\n# Link your development version of the extension with JupyterLab\njupyter labextension develop . --overwrite\n# Server extension must be manually installed in develop mode\njupyter server extension enable jupyterlab_pachyderm\n# Rebuild extension Typescript source after making changes\nnpm run build\n```\n\nYou can watch the source directory and run JupyterLab at the same time in different terminals to watch for changes in the extension's source and automatically rebuild the extension.\n\n```bash\n# Watch the source directory in one terminal, automatically rebuilding when needed\nnpm run watch\n# Run JupyterLab in another terminal\njupyter lab --allow-root --ip=0.0.0.0\n```\n\nWith the watch command running, every saved change will immediately be built locally and available in your running JupyterLab. Refresh JupyterLab to load the change in your browser (you may need to wait several seconds for the extension to be rebuilt).\n\nBy default, the `npm run build` command generates the source maps for this extension to make it easier to debug using the browser dev tools. To also generate source maps for the JupyterLab core extensions, you can run the following command:\n\n```bash\njupyter lab build --minimize=False\n```\n\nNote that to enable debug logging for `jupyter` (which also enables it for our extension backend, a python plugin to the JupyterLab server), you can run:\n\n```bash\njupyter lab --debug\n```\n\n## Development uninstall\n\n```bash\n# Server extension must be manually disabled in develop mode\njupyter server extension disable jupyterlab_pachyderm\npip uninstall jupyterlab_pachyderm\n```\n\nIn development mode, you will also need to remove the symlink created by `jupyter labextension develop`\ncommand. To find its location, you can run `jupyter labextension list` to figure out where the `labextensions`\nfolder is located. Then you can remove the symlink named `jupyterlab-pachyderm` within that folder.\n\n## Locally building the docker image\n\nUseful if iterating on the Dockerfile locally or iterating on changes to a version of mount-server.\n\nCreate & activate venv:\n```\npython3 -m venv venv\nsource venv/bin/activate\n```\n\nBuild `dist` directory:\n```\npython -m pip install --upgrade pip\npython -m pip install -r ci-requirements.txt\npython -m build\n```\n\nBuild docker image:\n```\nexport PACHCTL_VERSION=aaa7434c714fab6130c3982ebdaa8f279bd850c2 # or whichever version you want\ndocker build --build-arg PACHCTL_VERSION=$PACHCTL_VERSION -t pachyderm/notebooks-user:dev .\ndocker run -ti -p 8888:8888 pachyderm/notebooks-user:dev\n```\n\nNavigate to the URL that's printed out by the docker container, then change `tree` to `lab` in your browser's address bar.\n\n## Install\n\nTo install the extension, execute:\n\n```bash\npip install jupyterlab_pachyderm\n```\n\n## Uninstall\n\nTo remove the extension, execute:\n\n```bash\npip uninstall jupyterlab_pachyderm\n```\n\n\n## Server extension\n\nFirst make sure the server extension is enabled:\n\n```\njupyter server extension list 2>&1 | grep -ie \"jupyterlab_pachyderm.*OK\"\n```\n\n# Install Jupyterhub locally (Sidecar mode testing)\n\n```\nhelm repo add jupyterhub https://jupyterhub.github.io/helm-chart/\nhelm repo update\nhelm install jhub jupyterhub/jupyterhub --version 2.0.0 --values scripts/jhub-dev-helm-values.yml\n```\n\nNote: Use any username and no password to login\n\n### API endpoints\n\n```\nGET /repos # returns a list of all repos/branches\nGET /mounts # returns a list of all active mounts and unmounted repos/branches\nPUT /_mount # mounts a single repo\nPUT /_unmount # unmounts a single repo\nPUT /_commit # commits any changes to the repo\nPUT /_unmount_all # unmounts all repos\nPUT /datums/_mount # mounts first datum of given input spec\nPUT /datums/_next # mounts next datum\nPUT /datums/_prev # mounts prev datum\nGET /datums # returns info on mounted datum\n```\n\nThe servers-side extension extends jupyter server, so it automatically starts as part of `jupyter lab`.\n\nAPI endpoints can be accessed via `localhost:8888/pachyderm/v2`\n\nThe frontend can access the endpoints via `/v2`, for example:\n\n```js\nrequestAPI<any>('/v2/repos')\n .then(data => {\n console.log(data);\n })\n .catch(reason => {\n console.error(reason);\n });\n```\n\nYou can also access it via `localhost:8888/v2`\n\n## SVG Images\n\nWe are leveraging [svgr](https://react-svgr.com/) to simplify the use of non icon svgs in in the project. Svg images that are to be converted live in the `svg-images` folder and get output to the `src/utils/components/Svgs` folder. If you want to add a new image to the project simply add the svg to the `svg-images` folder and run `npm run build:svg`. We have spent some time trying to get svgr to work through the `@svgr/webpack` plugin but have not been successful as of yet.\n\n# Project Structure\nJupyter extensions are composed of several plugins. These plugins can be selectively enabled or disabled. Because of this we have decided separate the functionality in the extension using plugins. Plugins exported in this extension are as follows.\n\n### Hub\nThis plugin contains custom styling and other features only used by hub. By default this extension is disabled.\n\n### Mount\nThis plugin contains the mount feature currently under development.\n\n\n## Plugin settings\nYou can disable certain plugins by specifying the following config data in the `<jupyter_config_path>/labconfig/page_config.json`:\n\n```\n{\n \"disabledExtensions\": {\n \"jupyterlab-pachyderm:examples\": true\n }\n} \n```\nYou can check your config paths by running `jupyter --paths`. Setting this config file is not part of the built extension and needs to be done by the user.\n\nAdding the following to the package.json in the `jupyterlab` object will disable the examples plugin by default.\n```\n\"disabledExtensions\": [\"jupyterlab-pachyderm:examples\"]\n```\nSo we can build the extension with hub features turned off and override the setting for hub.\n\n\n## Troubleshoot\n\nIf you are seeing the frontend extension, but it is not working, check\nthat the server extension is enabled:\n\n```bash\njupyter server extension list\n```\n\nIf the server extension is installed and enabled, but you are not seeing\nthe frontend extension, check the frontend extension is installed:\n\n```bash\njupyter labextension list\n```\n",
"bugtrack_url": null,
"license": "BSD-3-Clause",
"summary": "A JupyterLab extension.",
"version": "2.9.5",
"project_urls": {
"Homepage": "https://github.com/pachyderm/pachyderm"
},
"split_keywords": [
"jupyter",
" jupyterlab",
" jupyterlab-extension"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4c1efca59871ea7f4710343077cd0ca399f872b1dee6574892e72b89ca326e49",
"md5": "b56314908b26b28cdf17f9330f9abab8",
"sha256": "447906db415cef1f7079de450068fd076d2987b9a01d7fcdc909e059ca990066"
},
"downloads": -1,
"filename": "jupyterlab_pachyderm-2.9.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "b56314908b26b28cdf17f9330f9abab8",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4,>=3.8",
"size": 558380,
"upload_time": "2024-04-24T19:01:44",
"upload_time_iso_8601": "2024-04-24T19:01:44.601976Z",
"url": "https://files.pythonhosted.org/packages/4c/1e/fca59871ea7f4710343077cd0ca399f872b1dee6574892e72b89ca326e49/jupyterlab_pachyderm-2.9.5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "0eb4c77ca5ba89b20ebf28e8526c1d16dfa6257bd80986af297097332ff476c8",
"md5": "5e6987d4aefb2902506589753645d093",
"sha256": "a1de6582e7287b2a8feaf4ccbf2bf23bf35912fa3b0eefb757fb58a4a975c3c8"
},
"downloads": -1,
"filename": "jupyterlab_pachyderm-2.9.5.tar.gz",
"has_sig": false,
"md5_digest": "5e6987d4aefb2902506589753645d093",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4,>=3.8",
"size": 361739,
"upload_time": "2024-04-24T19:01:47",
"upload_time_iso_8601": "2024-04-24T19:01:47.610201Z",
"url": "https://files.pythonhosted.org/packages/0e/b4/c77ca5ba89b20ebf28e8526c1d16dfa6257bd80986af297097332ff476c8/jupyterlab_pachyderm-2.9.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-24 19:01:47",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "pachyderm",
"github_project": "pachyderm",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"circle": true,
"lcname": "jupyterlab-pachyderm"
}
JSON
