DiffDRR
================
> Auto-differentiable DRR rendering and optimization in PyTorch
[](https://github.com/eigenvivek/DiffDRR/actions/workflows/test.yaml)
[](https://arxiv.org/abs/2208.12737)
[](LICENSE)
[](https://pepy.tech/project/diffdrr)
[](https://vivekg.dev/DiffDRR/)
[](https://github.com/psf/black)
`DiffDRR` is a PyTorch-based digitally reconstructed radiograph (DRR) generator that provides
1. Differentiable X-ray rendering
2. GPU-accelerated synthesis and optimization
3. A pure Python implementation
Most importantly, `DiffDRR` implements DRR rendering as a PyTorch module, making it interoperable in deep learning pipelines.
## Install
To install the latest stable release (**recommended**):
```zsh
pip install diffdrr
```
To install the development version:
```zsh
git clone https://github.com/eigenvivek/DiffDRR.git --depth 1
pip install -e 'DiffDRR/[dev]'
```
## Hello, World!
The following minimal example specifies the geometry of the projectional radiograph imaging system and traces rays through a CT volume:
``` python
import matplotlib.pyplot as plt
import torch
from diffdrr.drr import DRR
from diffdrr.data import load_example_ct
from diffdrr.visualization import plot_drr
# Read in the volume and get its origin and spacing in world coordinates
subject = load_example_ct()
# Initialize the DRR module for generating synthetic X-rays
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
drr = DRR(
subject, # An object storing the CT volume, origin, and voxel spacing
sdd=1020.0, # Source-to-detector distance (i.e., focal length)
height=200, # Image height (if width is not provided, the generated DRR is square)
delx=2.0, # Pixel spacing (in mm)
).to(device)
# Set the camera pose with rotations (yaw, pitch, roll) and translations (x, y, z)
rotations = torch.tensor([[0.0, 0.0, 0.0]], device=device)
translations = torch.tensor([[0.0, 850.0, 0.0]], device=device)
# 📸 Also note that DiffDRR can take many representations of SO(3) 📸
# For example, quaternions, rotation matrix, axis-angle, etc...
img = drr(rotations, translations, parameterization="euler_angles", convention="ZXY")
plot_drr(img, ticks=False)
plt.show()
```
On a single NVIDIA RTX 2080 Ti GPU, producing such an image takes
25.2 ms ± 10.5 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
The full example is available at
[`introduction.ipynb`](https://vivekg.dev/DiffDRR/tutorials/introduction.html).
## Usage
### Rendering
The physics-based pipeline in `DiffDRR` renders photorealistic X-rays. For example, compare
a real X-ray to a synthetic X-ray rendered from a CT of the same patient using `DiffDRR`
(X-rays and CTs from the [DeepFluoro dataset](https://github.com/rg2/DeepFluoroLabeling-IPCAI2020)):
### 2D/3D Registration
The impotus for developing `DiffDRR` was to solve 2D/3D registration
problems with gradient-based optimization. Here, we demonstrate `DiffDRR`'s
capabilities by generating two DRRs:
1. A fixed DRR from a set of ground truth parameters
2. A moving DRR from randomly initialized parameters
To align the two images, we use gradient descent to maximize
an image similarity metric between the two DRRs. This produces
optimization runs like this:
The full example is available at
[`optimizers.ipynb`](https://vivekg.dev/DiffDRR/tutorials/optimizers.html).
#### *🆕 Examples on Real-World Data 🆕*
For examples running `DiffDRR` on real surgical datasets, check out our latest work, [`DiffPose`](https://github.com/eigenvivek/DiffPose):
This work includes a lot of real-world usecases of `DiffDRR` including
- Using `DiffDRR` as a layer in a deep learning architecture
- Alignment of real X-rays and rendered DRRs
- Achieving sub-millimeter registration accuracy very quickly
### X-ray Segmentation
`DiffDRR` can project 3D labelmaps into 2D simply using perspective geometry, helping identify particular structures in simulated X-rays
(these labels come from the [TotalSegmentator v2 dataset](https://github.com/wasserth/TotalSegmentator)):
### Volume Reconstruction
`DiffDRR` is differentiable with respect to the 3D volume as well as camera poses.
Therefore, it can be used for volume reconstruction via differentiable
rendering (see [`reconstruction.ipynb`](https://vivekg.dev/DiffDRR/tutorials/reconstruction.html))!
#### *🆕 Examples on Real-World Data 🆕*
For an in-depth example using `DiffDRR` for cone-beam CT reconstruction, check out [`DiffVox`](https://github.com/hossein-momeni/DiffVox).
## Development
> **TLDR:** Source code is stored in `notebooks/api/`, not `diffdrr/`. Update the notebooks instead!
`DiffDRR` source code, docs, and CI are all built using
[`nbdev`](https://nbdev.fast.ai/). To get set up with `nbdev`, install
the following
``` zsh
mamba install jupyterlab nbdev -c fastai -c conda-forge
nbdev_install_quarto # To build docs
nbdev_install_hooks # Make notebooks git-friendly
```
Running `nbdev_help` will give you the full list of options. The most
important ones are
``` zsh
nbdev_preview # Render docs locally and inspect in browser
nbdev_clean # NECESSARY BEFORE PUSHING
nbdev_test # tests notebooks
nbdev_export # builds package and builds docs
```
For more details, follow this [in-depth
tutorial](https://nbdev.fast.ai/tutorials/tutorial.html).
## How does `DiffDRR` work?
`DiffDRR` reformulates Siddon’s method,[^1] an exact
algorithm for calculating the radiologic path of an X-ray
through a volume, as a series of vectorized tensor operations. This
version of the algorithm is easily implemented in tensor algebra
libraries like PyTorch to achieve a fast auto-differentiable DRR
generator.
[^1]: [Siddon RL. Fast calculation of
the exact radiological path for a three-dimensional CT array. Medical
Physics, 2(12):252–5, 1985.](https://doi.org/10.1118/1.595715)
## Citing `DiffDRR`
If you find `DiffDRR` useful in your work, please cite our
[paper](https://arxiv.org/abs/2208.12737):
@inproceedings{gopalakrishnan2022fast,
title={Fast auto-differentiable digitally reconstructed radiographs for solving inverse problems in intraoperative imaging},
author={Gopalakrishnan, Vivek and Golland, Polina},
booktitle={Workshop on Clinical Image-Based Procedures},
pages={1--11},
year={2022},
organization={Springer}
}
If the 2D/3D registration capabilities are helpful, please cite our followup, [`DiffPose`](https://arxiv.org/abs/2312.06358):
@article{gopalakrishnan2023intraoperative,
title={Intraoperative {2D/3D} image registration via differentiable {X}-ray rendering},
author={Gopalakrishnan, Vivek and Dey, Neel and Golland, Polina},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={11662--11672},
year={2024}
}
If you use the 3D CBCT reconstruction capabilities, please cite our followup, [`DiffVox`](https://arxiv.org/abs/2411.19224):
@article{momeni2024voxel,
title={Voxel-based Differentiable X-ray Rendering Improves Self-Supervised 3D CBCT Reconstruction},
author={Momeni, Mohammadhossein and Gopalakrishnan, Vivek and Dey, Neel and Golland, Polina and Frisken, Sarah},
booktitle={Machine Learning and the Physical Sciences, NeurIPS 2024},
year={2024}
}
Raw data
{
"_id": null,
"home_page": "https://github.com/eigenvivek/DiffDRR",
"name": "diffdrr",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "nbdev jupyter notebook python",
"author": "Vivek Gopalakrishnan",
"author_email": "vivekg@mit.edu",
"download_url": "https://files.pythonhosted.org/packages/f9/e7/6a12e0c68019fd9859d3e1e78164995d1853cc472b1856d9f12bd20307c1/diffdrr-0.4.5.tar.gz",
"platform": null,
"description": "DiffDRR\n================\n\n> Auto-differentiable DRR rendering and optimization in PyTorch\n\n[](https://github.com/eigenvivek/DiffDRR/actions/workflows/test.yaml)\n[](https://arxiv.org/abs/2208.12737)\n[](LICENSE)\n[](https://pepy.tech/project/diffdrr)\n[](https://vivekg.dev/DiffDRR/)\n[](https://github.com/psf/black)\n\n`DiffDRR` is a PyTorch-based digitally reconstructed radiograph (DRR) generator that provides\n\n1. Differentiable X-ray rendering\n2. GPU-accelerated synthesis and optimization\n3. A pure Python implementation\n\nMost importantly, `DiffDRR` implements DRR rendering as a PyTorch module, making it interoperable in deep learning pipelines.\n\n## Install\n\nTo install the latest stable release (**recommended**):\n\n```zsh\npip install diffdrr\n```\n\nTo install the development version:\n\n```zsh\ngit clone https://github.com/eigenvivek/DiffDRR.git --depth 1\npip install -e 'DiffDRR/[dev]'\n```\n\n## Hello, World!\n\nThe following minimal example specifies the geometry of the projectional radiograph imaging system and traces rays through a CT volume:\n\n``` python\nimport matplotlib.pyplot as plt\nimport torch\n\nfrom diffdrr.drr import DRR\nfrom diffdrr.data import load_example_ct\nfrom diffdrr.visualization import plot_drr\n\n# Read in the volume and get its origin and spacing in world coordinates\nsubject = load_example_ct()\n\n# Initialize the DRR module for generating synthetic X-rays\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\ndrr = DRR(\n subject, # An object storing the CT volume, origin, and voxel spacing\n sdd=1020.0, # Source-to-detector distance (i.e., focal length)\n height=200, # Image height (if width is not provided, the generated DRR is square)\n delx=2.0, # Pixel spacing (in mm)\n).to(device)\n\n# Set the camera pose with rotations (yaw, pitch, roll) and translations (x, y, z)\nrotations = torch.tensor([[0.0, 0.0, 0.0]], device=device)\ntranslations = torch.tensor([[0.0, 850.0, 0.0]], device=device)\n\n# \ud83d\udcf8 Also note that DiffDRR can take many representations of SO(3) \ud83d\udcf8\n# For example, quaternions, rotation matrix, axis-angle, etc...\nimg = drr(rotations, translations, parameterization=\"euler_angles\", convention=\"ZXY\")\nplot_drr(img, ticks=False)\nplt.show()\n```\n\n\n\nOn a single NVIDIA RTX 2080 Ti GPU, producing such an image takes\n\n 25.2 ms \u00b1 10.5 \u00b5s per loop (mean \u00b1 std. dev. of 7 runs, 10 loops each)\n\nThe full example is available at\n[`introduction.ipynb`](https://vivekg.dev/DiffDRR/tutorials/introduction.html).\n\n## Usage\n\n### Rendering\n\nThe physics-based pipeline in `DiffDRR` renders photorealistic X-rays. For example, compare \na real X-ray to a synthetic X-ray rendered from a CT of the same patient using `DiffDRR`\n(X-rays and CTs from the [DeepFluoro dataset](https://github.com/rg2/DeepFluoroLabeling-IPCAI2020)):\n\n\n\n### 2D/3D Registration\n\nThe impotus for developing `DiffDRR` was to solve 2D/3D registration\nproblems with gradient-based optimization. Here, we demonstrate `DiffDRR`'s\ncapabilities by generating two DRRs:\n\n1. A fixed DRR from a set of ground truth parameters\n2. A moving DRR from randomly initialized parameters\n\nTo align the two images, we use gradient descent to maximize\nan image similarity metric between the two DRRs. This produces\noptimization runs like this:\n\n\n\nThe full example is available at\n[`optimizers.ipynb`](https://vivekg.dev/DiffDRR/tutorials/optimizers.html).\n\n#### *\ud83c\udd95 Examples on Real-World Data \ud83c\udd95*\n\nFor examples running `DiffDRR` on real surgical datasets, check out our latest work, [`DiffPose`](https://github.com/eigenvivek/DiffPose):\n\n\n\nThis work includes a lot of real-world usecases of `DiffDRR` including\n- Using `DiffDRR` as a layer in a deep learning architecture\n- Alignment of real X-rays and rendered DRRs\n- Achieving sub-millimeter registration accuracy very quickly\n\n### X-ray Segmentation\n\n`DiffDRR` can project 3D labelmaps into 2D simply using perspective geometry, helping identify particular structures in simulated X-rays\n(these labels come from the [TotalSegmentator v2 dataset](https://github.com/wasserth/TotalSegmentator)):\n\n\n\n### Volume Reconstruction\n\n`DiffDRR` is differentiable with respect to the 3D volume as well as camera poses.\nTherefore, it can be used for volume reconstruction via differentiable\nrendering (see [`reconstruction.ipynb`](https://vivekg.dev/DiffDRR/tutorials/reconstruction.html))!\n\n#### *\ud83c\udd95 Examples on Real-World Data \ud83c\udd95*\n\nFor an in-depth example using `DiffDRR` for cone-beam CT reconstruction, check out [`DiffVox`](https://github.com/hossein-momeni/DiffVox).\n\n## Development\n\n> **TLDR:** Source code is stored in `notebooks/api/`, not `diffdrr/`. Update the notebooks instead!\n\n`DiffDRR` source code, docs, and CI are all built using\n[`nbdev`](https://nbdev.fast.ai/). To get set up with `nbdev`, install\nthe following\n\n``` zsh\nmamba install jupyterlab nbdev -c fastai -c conda-forge \nnbdev_install_quarto # To build docs\nnbdev_install_hooks # Make notebooks git-friendly\n```\n\nRunning `nbdev_help` will give you the full list of options. The most\nimportant ones are\n\n``` zsh\nnbdev_preview # Render docs locally and inspect in browser\nnbdev_clean # NECESSARY BEFORE PUSHING\nnbdev_test # tests notebooks\nnbdev_export # builds package and builds docs\n```\n\nFor more details, follow this [in-depth\ntutorial](https://nbdev.fast.ai/tutorials/tutorial.html).\n\n## How does `DiffDRR` work?\n\n`DiffDRR` reformulates Siddon\u2019s method,[^1] an exact\nalgorithm for calculating the radiologic path of an X-ray\nthrough a volume, as a series of vectorized tensor operations. This\nversion of the algorithm is easily implemented in tensor algebra\nlibraries like PyTorch to achieve a fast auto-differentiable DRR\ngenerator.\n\n[^1]: [Siddon RL. Fast calculation of\nthe exact radiological path for a three-dimensional CT array. Medical\nPhysics, 2(12):252\u20135, 1985.](https://doi.org/10.1118/1.595715)\n\n## Citing `DiffDRR`\n\nIf you find `DiffDRR` useful in your work, please cite our\n[paper](https://arxiv.org/abs/2208.12737):\n\n @inproceedings{gopalakrishnan2022fast,\n title={Fast auto-differentiable digitally reconstructed radiographs for solving inverse problems in intraoperative imaging},\n author={Gopalakrishnan, Vivek and Golland, Polina},\n booktitle={Workshop on Clinical Image-Based Procedures},\n pages={1--11},\n year={2022},\n organization={Springer}\n }\n\nIf the 2D/3D registration capabilities are helpful, please cite our followup, [`DiffPose`](https://arxiv.org/abs/2312.06358):\n\n @article{gopalakrishnan2023intraoperative,\n title={Intraoperative {2D/3D} image registration via differentiable {X}-ray rendering},\n author={Gopalakrishnan, Vivek and Dey, Neel and Golland, Polina},\n booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n pages={11662--11672},\n year={2024}\n }\n\nIf you use the 3D CBCT reconstruction capabilities, please cite our followup, [`DiffVox`](https://arxiv.org/abs/2411.19224):\n\n @article{momeni2024voxel,\n title={Voxel-based Differentiable X-ray Rendering Improves Self-Supervised 3D CBCT Reconstruction},\n author={Momeni, Mohammadhossein and Gopalakrishnan, Vivek and Dey, Neel and Golland, Polina and Frisken, Sarah},\n booktitle={Machine Learning and the Physical Sciences, NeurIPS 2024},\n year={2024}\n }\n",
"bugtrack_url": null,
"license": "MIT License",
"summary": "Auto-differentiable digitally reconstructed radiographs in PyTorch",
"version": "0.4.5",
"project_urls": {
"Homepage": "https://github.com/eigenvivek/DiffDRR"
},
"split_keywords": [
"nbdev",
"jupyter",
"notebook",
"python"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "02a2924859cfe1211dea0d42a84b9203dbc470232e69d7055693f8b976396379",
"md5": "e2cac24f63c1364f48767a3039a5240d",
"sha256": "d3d7c7683ae584805ad4964a5358195fd3e4811d526544f52c6193e2cc51b160"
},
"downloads": -1,
"filename": "diffdrr-0.4.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "e2cac24f63c1364f48767a3039a5240d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 34801868,
"upload_time": "2024-12-12T20:13:06",
"upload_time_iso_8601": "2024-12-12T20:13:06.301300Z",
"url": "https://files.pythonhosted.org/packages/02/a2/924859cfe1211dea0d42a84b9203dbc470232e69d7055693f8b976396379/diffdrr-0.4.5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f9e76a12e0c68019fd9859d3e1e78164995d1853cc472b1856d9f12bd20307c1",
"md5": "b6d6028c6ddefae9fb06a09fa86134b1",
"sha256": "bc5154a71bc8264a2bb22c6f3910e7a7b95f4aa04126e5a2536ef28db93eb6d0"
},
"downloads": -1,
"filename": "diffdrr-0.4.5.tar.gz",
"has_sig": false,
"md5_digest": "b6d6028c6ddefae9fb06a09fa86134b1",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 34804777,
"upload_time": "2024-12-12T20:13:12",
"upload_time_iso_8601": "2024-12-12T20:13:12.786282Z",
"url": "https://files.pythonhosted.org/packages/f9/e7/6a12e0c68019fd9859d3e1e78164995d1853cc472b1856d9f12bd20307c1/diffdrr-0.4.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-12 20:13:12",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "eigenvivek",
"github_project": "DiffDRR",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "diffdrr"
}
JSON
