| Name | ds-splat JSON |
| Version |
0.0.2
JSON |
| download |
| home_page | None |
| Summary | A CUDA-based gaussian splatting rasterizer extension for PyTorch. |
| upload_time | 2024-07-02 06:32:02 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.7 |
| license | None |
| keywords |
pytorch
cuda
rasterizer
deep learning
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# 🖌️ deepsense.ai 3D Gaussian Splatting
Fastest open-source implementation (Apache 2.0 License) of 3D Gaussian Splatting rasterizer function as forward/backward
CUDA kernels. Forward call is our original work and our backward code is based on
[nerfstudio's gsplat](https://github.com/nerfstudio-project/gsplat) implementation.
We are using the same api as Vanilla [graphdeco-inria 3D Gaussian Splatting implementation](https://github.com/graphdeco-inria/gaussian-splatting),
so it is very easy to replace original render calls simply by swapping the import.
## Table of Contents
- [⚡ Get fastest open-source forward/backward kernels](#-get-fastest-open-source-forwardbackward-kernels)
- [📦 Get From PyPI](#-get-from-pypi)
- [💡 Integrated into Gaussian Splatting Lighting](#-integrated-into-gaussian-splatting-lighting)
- [🔧 Install from repository](#-install-from-repository)
- [🐍 Using Python Extension](#-using-python-extension)
- [🛠 CPP Project Initialization](#-cpp-project-initialization)
- [🔄 How to switch to open-source KNN](#-how-to-switch-to-open-source-knn)
- [📊 Benchmarks](#-benchmarks)
## ⚡ Get fastest open-source forward/backward kernels
Fastest open-source and easy to use replacement for these who are using non-commercial friendly Vanilla
[graphdeco-inria 3D Gaussian Splatting implementation](https://github.com/graphdeco-inria/gaussian-splatting).
- Forward and backward CUDA calls
- Fastest open-source
- Easy to integrate
- Thrust and Torch I/O API
### 📦 Get From PyPI
Follow this step, if you are already using Vanilla's [graphdeco-inria 3D Gaussian Splatting implementation](https://github.com/graphdeco-inria/gaussian-splatting)
in your project and you want to replace forward/backward kernels
with deepsense.ai open-source kernels.
Make sure CUDA compiler is installed in your environment and simply install:
```bash
pip install ds-splat
```
You are good to go just by swapping imports:
```diff
- from diff_gaussian_rasterization import GaussianRasterizationSettings, GaussianRasterizer
+ from ds_splat import GaussianRasterizationSettings, GaussianRasterizer
```
After swapping to our code, you will keep 3D Gaussian Splatting functionality (backward and forward passes) and you
will use open-source code. If you also want to use open-source code for the KNN step in preprocessing, scroll down!
### 💡 Integrated into Gaussian Splatting Lighting
If you are rather starting project from scratch and are interested in end-to-end environment, we recommend to check
our integration into [gaussian-splatting-lighting](https://github.com/yzslab/gaussian-splatting-lightning) repository.
Gaussian splatting lighting repository is under MIT License, but submodules like Vanilla's forward/backward kernels or
KNN implementation has non-commercial friendly license. You can use deepsense ds-splat as a backend, and this way using
fastest open-source forward/backward kernel calls.
## 🔧 Install from repository
Instead of installing from PyPI, you can install ds-splat package directly from this repository.
### 🐍 Using Python Extension
you can use pip install in the project's root directory:
```bash
pip install .
```
Via setup.py, this will compile CUDA and CPP code and will install ds-splat package.
### 🛠 CPP Project Initialization
This is a bit more manual and you don't have to make it if you installed from PyPI or with the above pip install.
If you prefer to build project from scratch follow instructions here.
This project uses conan for additional dependencies i.e. Catch2. To generate CMake project follow these instructions:
```bash
cd cuda_rasterizer # make sure you are in the root directory
conan install . -of=conan/x86_reldebug --settings=build_type=RelWithDebInfo --profile=default
mkdir build_cpp; cd build_cpp
cmake -DCMAKE_PREFIX_PATH=`python -c 'import torch;print(torch.utils.cmake_prefix_path)'` -DCMAKE_TOOLCHAIN_FILE=../conan/x86_reldebug/build/RelWithDebInfo/generators/conan_toolchain.cmake -DBUILD_TESTING=ON -DCMAKE_BUILD_TYPE=RelWithDebInfo ..
make
```
If there are any problems regarding runtime exception (e.g. `std::bad_alloc`) or link errors make sure to edit your conan profile to use specific ABI.
Following conanfile was tested:
```
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=17
compiler.libcxx=libstdc++
compiler.version=11
os=Linux
```
## 🔄 How to switch to open-source KNN
If you are using for e.g. [gaussain splatting lighting](https://github.com/yzslab/gaussian-splatting-lightning) repository,
then forward/backward CUDA kernels and KNN are under [Gaussian-Splatting License](https://github.com/graphdeco-inria/gaussian-splatting?tab=License-1-ov-file#readme).
When you switch to our code following instructions above, you will use our open source forward and backward calls.
Here, we provide instructions on how to also use open source KNN implementation via Faiss. This instruction is for
replacing KNN implementation in [gaussain splatting lighting](https://github.com/yzslab/gaussian-splatting-lightning)
repository.
#### Install Faiss
https://github.com/facebookresearch/faiss/blob/main/INSTALL.md
For example, if you are using conda, in your environment install:
```bash
conda install -c pytorch -c nvidia -c rapidsai -c conda-forge faiss-gpu-raft=1.8.0
```
#### Modify GaussianModel class
1. localize gaussian_model.py file that contains class GaussianModel
2. import faiss
```python
import faiss
```
3. add method for averaged distances
```python
def _get_averaged_distances(self, pcd_points_np: np.ndarray, method: str = "CPU_approx", device_id: int = 0,
k: int = 4, dim: int = 3, nlist: int = 200) -> np.ndarray:
"""
This method takes numpy array of points and returns averaged distances for k-nearest neighbours
for each query point (excluding query point). Database/reference points and query points are same set.
Using Faiss as a backend.
Args:
pcd_points_np: pcd points as numpy array
method: how faiss create indices and what is target device for calc. {"CPU", "GPU", "CPU_approx",
"GPU_approx"}
device_id: GPU device id
k: k-nearest neighbours (including self)
dim: dimentionality of the dataset. 3 by default.
nlist: the number of clusters or cells in the inverted file (IVF) structure when using an IndexIVFFlat
index. Only relevant for approximated methods.
Returns:
numpy array as mean from k-nearest neighbour (except self) for each query point
"""
valid_index_types = {"CPU", "GPU", "CPU_approx", "GPU_approx"}
pcd_points_float_32 = pcd_points_np.astype(np.float32)
if method == "CPU":
index = faiss.IndexFlatL2(dim)
elif method == "GPU":
res = faiss.StandardGpuResources()
index = faiss.GpuIndexFlatL2(res, dim)
elif method == "CPU_approx":
quantizer = faiss.IndexFlatL2(3) # the other index
index = faiss.IndexIVFFlat(quantizer, dim, nlist)
elif method == "GPU_approx":
res = faiss.StandardGpuResources()
quantizer = faiss.IndexFlatL2(3) # the other index. Must be CPU as nested GPU indexes are not supported
index = faiss.index_cpu_to_gpu(res, device_id, faiss.IndexIVFFlat(quantizer, dim, nlist))
else:
raise ValueError(f"Invalid index_type. Expected one of {valid_index_types}, but got {method}.")
if method in {"CPU_approx", "GPU_approx"}:
index.train(pcd_points_float_32)
index.add(pcd_points_float_32)
D, _ = index.search(pcd_points_float_32, k)
D_mean = np.mean(D[:, 1:], axis=1)
return D_mean
```
4. localize create_from_pcd(...) method and modify it.
Replace lines:
```diff
- dist2 = torch.clamp_min(distCUDA2(torch.from_numpy(np.asarray(pcd.points)).float().cuda()), 0.0000001).to(deivce)
+ dist_means_np = self._get_averaged_distances(pcd_points_np=pcd_points_np, method="CPU_approx")
+ dist2 = torch.clamp_min(torch.tensor(dist_means_np), 0.0000001).to(deivce)
```
This way you have modified the KNN method. Now it is independent from a
[licensed](https://github.com/graphdeco-inria/gaussian-splatting?tab=License-1-ov-file#readme)
submodule (distCUDA2 method) and now it is open source!
## 📊 Benchmarks
We have conducted a series of benchmarks, comparing deepsense implementation inference runtime to vanilla implementation
[graphdeco-inria 3D gaussian splatting implementation](https://github.com/graphdeco-inria/gaussian-splatting)
and to [nerfstudio's gsplat](https://github.com/nerfstudio-project/gsplat) implementation.
Below plots present inference time in ms measured for 120 frames as fly through a scene with zooming out to capture all
Gaussians. 6.1M Gaussians rendered in 1920x1080 with an NVIDIA 4070 Laptop GPU and 5.8M Gaussians rendered in 3840x2160
with an NVIDIA 3090 GPU.
For trained scenes, we have also compared PSNR (Peak Signal-to-Noise Ratio) for deepsense and gsplat methods to
Vanilla as ground truth. Using Vanilla's inria implementation, we rendered images when flying through a scene,
treating them as ground truth. For deepsense and gsplat implementations, we rendered scenes from the same camera
positions and compared them to Vanilla. This test shows how close our/gsplat implementation is to Vanilla's. Some
details are implementation-specific and result in slightly different outcomes, but both methods have very good PSNR in
this regard. Higher PSNR is better.
📥 [Download](https://drive.google.com/file/d/1MADQzb6onTV6JBJQqTx8H9bN6It4Vcdj/view?usp=sharing) more benchmark plots from GDrive.
Raw data
{
"_id": null,
"home_page": null,
"name": "ds-splat",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": null,
"keywords": "pytorch, cuda, rasterizer, deep learning",
"author": null,
"author_email": "\"deepsense.ai\" <contact@deepsense.ai>",
"download_url": "https://files.pythonhosted.org/packages/40/3f/3804bdcb3dce95db5fa643c9ec383de8ebd9065db2b751900d4969ade9a3/ds_splat-0.0.2.tar.gz",
"platform": null,
"description": "# \ud83d\udd8c\ufe0f deepsense.ai 3D Gaussian Splatting\n\nFastest open-source implementation (Apache 2.0 License) of 3D Gaussian Splatting rasterizer function as forward/backward \nCUDA kernels. Forward call is our original work and our backward code is based on \n[nerfstudio's gsplat](https://github.com/nerfstudio-project/gsplat) implementation.\nWe are using the same api as Vanilla [graphdeco-inria 3D Gaussian Splatting implementation](https://github.com/graphdeco-inria/gaussian-splatting), \nso it is very easy to replace original render calls simply by swapping the import.\n\n\n\n\n## Table of Contents\n- [\u26a1 Get fastest open-source forward/backward kernels](#-get-fastest-open-source-forwardbackward-kernels)\n - [\ud83d\udce6 Get From PyPI](#-get-from-pypi)\n - [\ud83d\udca1 Integrated into Gaussian Splatting Lighting](#-integrated-into-gaussian-splatting-lighting)\n- [\ud83d\udd27 Install from repository](#-install-from-repository)\n - [\ud83d\udc0d Using Python Extension](#-using-python-extension)\n - [\ud83d\udee0 CPP Project Initialization](#-cpp-project-initialization)\n- [\ud83d\udd04 How to switch to open-source KNN](#-how-to-switch-to-open-source-knn)\n- [\ud83d\udcca Benchmarks](#-benchmarks)\n\n## \u26a1 Get fastest open-source forward/backward kernels\n\nFastest open-source and easy to use replacement for these who are using non-commercial friendly Vanilla\n[graphdeco-inria 3D Gaussian Splatting implementation](https://github.com/graphdeco-inria/gaussian-splatting).\n\n- Forward and backward CUDA calls\n- Fastest open-source\n- Easy to integrate\n- Thrust and Torch I/O API\n\n### \ud83d\udce6 Get From PyPI\n\nFollow this step, if you are already using Vanilla's [graphdeco-inria 3D Gaussian Splatting implementation](https://github.com/graphdeco-inria/gaussian-splatting)\nin your project and you want to replace forward/backward kernels\nwith deepsense.ai open-source kernels.\n\nMake sure CUDA compiler is installed in your environment and simply install:\n```bash\npip install ds-splat\n```\n\nYou are good to go just by swapping imports:\n```diff\n- from diff_gaussian_rasterization import GaussianRasterizationSettings, GaussianRasterizer\n+ from ds_splat import GaussianRasterizationSettings, GaussianRasterizer\n```\n\nAfter swapping to our code, you will keep 3D Gaussian Splatting functionality (backward and forward passes) and you \nwill use open-source code. If you also want to use open-source code for the KNN step in preprocessing, scroll down!\n\n### \ud83d\udca1 Integrated into Gaussian Splatting Lighting\n\nIf you are rather starting project from scratch and are interested in end-to-end environment, we recommend to check\nour integration into [gaussian-splatting-lighting](https://github.com/yzslab/gaussian-splatting-lightning) repository.\nGaussian splatting lighting repository is under MIT License, but submodules like Vanilla's forward/backward kernels or \nKNN implementation has non-commercial friendly license. You can use deepsense ds-splat as a backend, and this way using \nfastest open-source forward/backward kernel calls.\n\n\n## \ud83d\udd27 Install from repository\n\nInstead of installing from PyPI, you can install ds-splat package directly from this repository.\n\n### \ud83d\udc0d Using Python Extension\n\nyou can use pip install in the project's root directory:\n```bash\npip install .\n```\n\nVia setup.py, this will compile CUDA and CPP code and will install ds-splat package.\n\n### \ud83d\udee0 CPP Project Initialization\n\nThis is a bit more manual and you don't have to make it if you installed from PyPI or with the above pip install.\n\nIf you prefer to build project from scratch follow instructions here.\n\nThis project uses conan for additional dependencies i.e. Catch2. To generate CMake project follow these instructions:\n\n```bash\ncd cuda_rasterizer # make sure you are in the root directory\nconan install . -of=conan/x86_reldebug --settings=build_type=RelWithDebInfo --profile=default\nmkdir build_cpp; cd build_cpp\ncmake -DCMAKE_PREFIX_PATH=`python -c 'import torch;print(torch.utils.cmake_prefix_path)'` -DCMAKE_TOOLCHAIN_FILE=../conan/x86_reldebug/build/RelWithDebInfo/generators/conan_toolchain.cmake -DBUILD_TESTING=ON -DCMAKE_BUILD_TYPE=RelWithDebInfo ..\nmake\n```\n\nIf there are any problems regarding runtime exception (e.g. `std::bad_alloc`) or link errors make sure to edit your conan profile to use specific ABI.\nFollowing conanfile was tested:\n\n\n``` \n[settings]\narch=x86_64\nbuild_type=Release\ncompiler=gcc\ncompiler.cppstd=17\ncompiler.libcxx=libstdc++\ncompiler.version=11\nos=Linux\n```\n\n## \ud83d\udd04 How to switch to open-source KNN\n\nIf you are using for e.g. [gaussain splatting lighting](https://github.com/yzslab/gaussian-splatting-lightning) repository,\nthen forward/backward CUDA kernels and KNN are under [Gaussian-Splatting License](https://github.com/graphdeco-inria/gaussian-splatting?tab=License-1-ov-file#readme).\nWhen you switch to our code following instructions above, you will use our open source forward and backward calls. \nHere, we provide instructions on how to also use open source KNN implementation via Faiss. This instruction is for \nreplacing KNN implementation in [gaussain splatting lighting](https://github.com/yzslab/gaussian-splatting-lightning) \nrepository.\n\n#### Install Faiss\nhttps://github.com/facebookresearch/faiss/blob/main/INSTALL.md\nFor example, if you are using conda, in your environment install:\n```bash\nconda install -c pytorch -c nvidia -c rapidsai -c conda-forge faiss-gpu-raft=1.8.0\n```\n\n#### Modify GaussianModel class\n\n1. localize gaussian_model.py file that contains class GaussianModel\n2. import faiss\n ```python\n import faiss\n ```\n3. add method for averaged distances\n ```python\n def _get_averaged_distances(self, pcd_points_np: np.ndarray, method: str = \"CPU_approx\", device_id: int = 0,\n k: int = 4, dim: int = 3, nlist: int = 200) -> np.ndarray:\n \"\"\"\n This method takes numpy array of points and returns averaged distances for k-nearest neighbours\n for each query point (excluding query point). Database/reference points and query points are same set.\n\n Using Faiss as a backend.\n\n\n Args:\n pcd_points_np: pcd points as numpy array\n method: how faiss create indices and what is target device for calc. {\"CPU\", \"GPU\", \"CPU_approx\",\n \"GPU_approx\"}\n device_id: GPU device id\n k: k-nearest neighbours (including self)\n dim: dimentionality of the dataset. 3 by default.\n nlist: the number of clusters or cells in the inverted file (IVF) structure when using an IndexIVFFlat\n index. Only relevant for approximated methods.\n\n Returns:\n numpy array as mean from k-nearest neighbour (except self) for each query point\n \"\"\"\n valid_index_types = {\"CPU\", \"GPU\", \"CPU_approx\", \"GPU_approx\"}\n pcd_points_float_32 = pcd_points_np.astype(np.float32)\n\n if method == \"CPU\":\n index = faiss.IndexFlatL2(dim)\n elif method == \"GPU\":\n res = faiss.StandardGpuResources()\n index = faiss.GpuIndexFlatL2(res, dim)\n elif method == \"CPU_approx\":\n quantizer = faiss.IndexFlatL2(3) # the other index\n index = faiss.IndexIVFFlat(quantizer, dim, nlist)\n elif method == \"GPU_approx\":\n res = faiss.StandardGpuResources()\n quantizer = faiss.IndexFlatL2(3) # the other index. Must be CPU as nested GPU indexes are not supported\n index = faiss.index_cpu_to_gpu(res, device_id, faiss.IndexIVFFlat(quantizer, dim, nlist))\n else:\n raise ValueError(f\"Invalid index_type. Expected one of {valid_index_types}, but got {method}.\")\n\n if method in {\"CPU_approx\", \"GPU_approx\"}:\n index.train(pcd_points_float_32)\n index.add(pcd_points_float_32)\n\n D, _ = index.search(pcd_points_float_32, k)\n D_mean = np.mean(D[:, 1:], axis=1)\n\n return D_mean\n ```\n4. localize create_from_pcd(...) method and modify it.\n Replace lines:\n ```diff\n - dist2 = torch.clamp_min(distCUDA2(torch.from_numpy(np.asarray(pcd.points)).float().cuda()), 0.0000001).to(deivce)\n + dist_means_np = self._get_averaged_distances(pcd_points_np=pcd_points_np, method=\"CPU_approx\")\n + dist2 = torch.clamp_min(torch.tensor(dist_means_np), 0.0000001).to(deivce)\n ```\n\nThis way you have modified the KNN method. Now it is independent from a \n[licensed](https://github.com/graphdeco-inria/gaussian-splatting?tab=License-1-ov-file#readme) \nsubmodule (distCUDA2 method) and now it is open source!\n\n## \ud83d\udcca Benchmarks\n\nWe have conducted a series of benchmarks, comparing deepsense implementation inference runtime to vanilla implementation\n[graphdeco-inria 3D gaussian splatting implementation](https://github.com/graphdeco-inria/gaussian-splatting)\nand to [nerfstudio's gsplat](https://github.com/nerfstudio-project/gsplat) implementation.\n\n\nBelow plots present inference time in ms measured for 120 frames as fly through a scene with zooming out to capture all\nGaussians. 6.1M Gaussians rendered in 1920x1080 with an NVIDIA 4070 Laptop GPU and 5.8M Gaussians rendered in 3840x2160 \nwith an NVIDIA 3090 GPU.\n\n\n\n\n\nFor trained scenes, we have also compared PSNR (Peak Signal-to-Noise Ratio) for deepsense and gsplat methods to \nVanilla as ground truth. Using Vanilla's inria implementation, we rendered images when flying through a scene, \ntreating them as ground truth. For deepsense and gsplat implementations, we rendered scenes from the same camera \npositions and compared them to Vanilla. This test shows how close our/gsplat implementation is to Vanilla's. Some \ndetails are implementation-specific and result in slightly different outcomes, but both methods have very good PSNR in \nthis regard. Higher PSNR is better.\n\n\ud83d\udce5 [Download](https://drive.google.com/file/d/1MADQzb6onTV6JBJQqTx8H9bN6It4Vcdj/view?usp=sharing) more benchmark plots from GDrive.\n\n\n",
"bugtrack_url": null,
"license": null,
"summary": "A CUDA-based gaussian splatting rasterizer extension for PyTorch.",
"version": "0.0.2",
"project_urls": null,
"split_keywords": [
"pytorch",
" cuda",
" rasterizer",
" deep learning"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "403f3804bdcb3dce95db5fa643c9ec383de8ebd9065db2b751900d4969ade9a3",
"md5": "8f7bc13432fd7647ce69c0a3a447be46",
"sha256": "6c47b94d2c9d73ed21e4c74f7a9b77768322866538d9bb96d8d302a15a45029a"
},
"downloads": -1,
"filename": "ds_splat-0.0.2.tar.gz",
"has_sig": false,
"md5_digest": "8f7bc13432fd7647ce69c0a3a447be46",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 4619801,
"upload_time": "2024-07-02T06:32:02",
"upload_time_iso_8601": "2024-07-02T06:32:02.389095Z",
"url": "https://files.pythonhosted.org/packages/40/3f/3804bdcb3dce95db5fa643c9ec383de8ebd9065db2b751900d4969ade9a3/ds_splat-0.0.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-07-02 06:32:02",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "ds-splat"
}
