# Multiview 3D Keypoint Detection (Muke) [](https://pypi.org/project/muke/)
A simple approach to 3D keypoint detection using 2D estimation methods and multiview rendering, based on the blender project for [automatic keypoint retopology](https://github.com/cansik/auto-keypoint-retopology).
Basically, the 3D model is rendered from different angles (views) and a 2D keypoint detection is performed. For each detected keypoint, a ray-cast is performed to determine the intersection point with the mesh surface. In the end, all intersection points of the different views are combined to calculate the current 3D position of the keypoint within the mesh. It is possible to define view-dependent keypoint indices to extract only the points that are visible in the current rendering. Muke returns a list of 3D keypoints containing both the position and the nearest vertex index.
*Muke Process*
Direct 3D keypoint recognition using mesh data would be more accurate, but it is still difficult to train 3D models or find already trained weights for them. By using 2D recognition alone, it is possible to use the entire zoo of keypoint image recognition models. Muke comes with a built-in [MediaPipe face](https://google.github.io/mediapipe/solutions/face_mesh.html) and [pose](https://google.github.io/mediapipe/solutions/pose.html) detector, but can be extended with any other 2D keypoint detection framework.
*3D Facial Landmark Estimation (Human Head by [VistaPrime](https://sketchfab.com/3d-models/human-head-f46d952886ae4a8c8851341b810bba43) [CC Attribution](https://creativecommons.org/licenses/by/4.0/))*
The project was originally implemented to have a simple and fast solution for 3D keypoints detection for retopology purposes. However, it can also be used for any other application where 3D keypoints are needed, such as rigging, animation, etc.
### Installation
To install the package use the following pip command:
```bash
pip install muke
```
### Usage
Muke can be used as a command line tool to extract the keypoints in a specific format (e.g. [Wrap3](https://www.russian3dscanner.com/)). For that a configuration has to be created which defines the detection parameters as well as the rendering views.
#### Configuration
Example configuration:
```json
{
"description": "MP Face",
"detector": "media-pipe-face",
"resolution": 1024,
"generator": "wrap3",
"views": [
{
"name": "frontal",
"rotation": 0,
"keypoints": [
4,
76,
306
]
}
]
}
```
To select a range of keypoint indices, it is possible to define a `start` and `end` (included) index. It is also possible to skip certain indices in that range. Here an example on how to create a range (`skip` is optional):
```json
{
"start": 10,
"end": 15,
"skip": [13, 14]
}
```
#### Infinite Ray
Per view it is possible to set the `infinite-ray` value to `True` to shoot a ray through the mesh to infinity. Every intersection point with the mesh is used as a point to calculate the average center of the keypoint inside the mesh.
#### Demo
Quickly try out Muke by using the following commands.
```bash
python -m muke assets/person.ply --display --resolution 1024
```
```bash
python -m muke assets/human_head.obj --display --resolution 1024 --detector media-pipe-face
```
```bash
python -m muke assets/human_head.obj --config config/media-pipe-face.json --display
```
#### Help
```bash
usage: muke [-h] [--detector {media-pipe-pose,media-pipe-face}]
[--resolution RESOLUTION] [--infinite-ray] [--generator {wrap3}]
[--config CONFIG] [--load-raw] [--display] [--debug]
input
Detects keypoint locations in a 3d model.
positional arguments:
input Input mesh to process.
optional arguments:
-h, --help show this help message and exit
--detector {media-pipe-pose,media-pipe-face}
Detection method for 2d keypoint detection (default:
media-pipe-pose).
--resolution RESOLUTION
Render resolution for each view pass (default: 512).
--infinite-ray Send ray through mesh to infinity and use average of
intersections (default: False)
--generator {wrap3} Generator methods for output generation (default:
wrap3).
--config CONFIG Path to the configuration JSON file.
--load-raw Load mesh raw without post-processing (default: False)
--display Shows result rendering with keypoints (default: False)
--debug Shows debug frames and information (default: False)
```
### Library
It is also possible to use Muke as a library to detect keypoints on an existing 3d mesh.
```python
import open3d as o3d
from muke.Muke import Muke
from muke.detector.MediaPipePoseDetector import MediaPipePoseDetector
from muke.model.DetectionView import DetectionView
# load mesh from filesystem
mesh = o3d.io.read_triangle_mesh("assets/person.ply")
# define rendered views
keypoint_indexes = {28, 27, 26, 25, 24, 23, 12, 11, 14, 13, 16, 15, 5, 2, 0}
views = [
DetectionView("front", 0, keypoint_indexes),
DetectionView("back", 180, keypoint_indexes),
]
# detect keypoints
with Muke(MediaPipePoseDetector()) as m:
result = m.detect(mesh, views)
# present results
for kp in result:
print(f"KP {kp.index}: {kp.x:.2f} {kp.y:.2f} {kp.z:.2f}")
```
### Detectors
It is possible to implement custom keypoint detectors. The custom detector has to implement the `BaseDetector` interface as shown in the following example.
```python
import numpy as np
from muke.detector.BaseDetector import BaseDetector
from muke.detector.KeyPoint2 import KeyPoint2
class CustomDetector(BaseDetector):
def setup(self):
# todo: initialize the custom detector
pass
def detect(self, image: np.ndarray) -> [KeyPoint2]:
# todo: implement the custom 2d keypoint detection
pass
def release(self):
# todo: clean up allocated resources
pass
```
### Renderer
The current version uses [pygfx](https://github.com/pygfx/pygfx) as lightweight and offscreen renderer, [trimesh](https://github.com/mikedh/trimesh) for model loading into pygfx and [Open3D](https://github.com/isl-org/Open3D) for raycasting. Initially, [trimesh](https://github.com/mikedh/trimesh) was used for everything, which is archived in the [trimesh-renderer branch](https://github.com/cansik/multiview-3d-keypoint-detection/tree/trimesh-renderer). Open3D was also once used for everything, but has been archived in version `0.2.x` and the [open3d-renderer branch](https://github.com/cansik/multiview-3d-keypoint-detection/tree/open3d-renderer).
### About
MIT License - Copyright (c) 2024 Florian Bruggisser
Raw data
{
"_id": null,
"home_page": "https://github.com/cansik/multiview-3d-keypoint-detection",
"name": "muke",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "",
"author": "Florian Bruggisser",
"author_email": "github@broox.ch",
"download_url": "",
"platform": null,
"description": "# Multiview 3D Keypoint Detection (Muke) [](https://pypi.org/project/muke/)\nA simple approach to 3D keypoint detection using 2D estimation methods and multiview rendering, based on the blender project for [automatic keypoint retopology](https://github.com/cansik/auto-keypoint-retopology).\n\nBasically, the 3D model is rendered from different angles (views) and a 2D keypoint detection is performed. For each detected keypoint, a ray-cast is performed to determine the intersection point with the mesh surface. In the end, all intersection points of the different views are combined to calculate the current 3D position of the keypoint within the mesh. It is possible to define view-dependent keypoint indices to extract only the points that are visible in the current rendering. Muke returns a list of 3D keypoints containing both the position and the nearest vertex index.\n\n\n\n*Muke Process*\n\nDirect 3D keypoint recognition using mesh data would be more accurate, but it is still difficult to train 3D models or find already trained weights for them. By using 2D recognition alone, it is possible to use the entire zoo of keypoint image recognition models. Muke comes with a built-in [MediaPipe face](https://google.github.io/mediapipe/solutions/face_mesh.html) and [pose](https://google.github.io/mediapipe/solutions/pose.html) detector, but can be extended with any other 2D keypoint detection framework.\n\n\n\n*3D Facial Landmark Estimation (Human Head by [VistaPrime](https://sketchfab.com/3d-models/human-head-f46d952886ae4a8c8851341b810bba43) [CC Attribution](https://creativecommons.org/licenses/by/4.0/))*\n\nThe project was originally implemented to have a simple and fast solution for 3D keypoints detection for retopology purposes. However, it can also be used for any other application where 3D keypoints are needed, such as rigging, animation, etc.\n\n### Installation\n\nTo install the package use the following pip command:\n\n```bash\npip install muke\n```\n\n### Usage\nMuke can be used as a command line tool to extract the keypoints in a specific format (e.g. [Wrap3](https://www.russian3dscanner.com/)). For that a configuration has to be created which defines the detection parameters as well as the rendering views.\n\n#### Configuration\n\nExample configuration:\n\n```json\n{\n \"description\": \"MP Face\",\n \"detector\": \"media-pipe-face\",\n \"resolution\": 1024,\n \"generator\": \"wrap3\",\n \"views\": [\n {\n \"name\": \"frontal\",\n \"rotation\": 0,\n \"keypoints\": [\n 4,\n 76,\n 306\n ]\n }\n ]\n}\n```\n\nTo select a range of keypoint indices, it is possible to define a `start` and `end` (included) index. It is also possible to skip certain indices in that range. Here an example on how to create a range (`skip` is optional):\n\n```json\n{\n \"start\": 10,\n \"end\": 15,\n \"skip\": [13, 14]\n}\n```\n\n#### Infinite Ray\nPer view it is possible to set the `infinite-ray` value to `True` to shoot a ray through the mesh to infinity. Every intersection point with the mesh is used as a point to calculate the average center of the keypoint inside the mesh.\n\n#### Demo\nQuickly try out Muke by using the following commands.\n\n```bash\npython -m muke assets/person.ply --display --resolution 1024\n```\n\n```bash\npython -m muke assets/human_head.obj --display --resolution 1024 --detector media-pipe-face\n```\n\n```bash\npython -m muke assets/human_head.obj --config config/media-pipe-face.json --display\n```\n\n#### Help\n\n```bash\nusage: muke [-h] [--detector {media-pipe-pose,media-pipe-face}]\n [--resolution RESOLUTION] [--infinite-ray] [--generator {wrap3}]\n [--config CONFIG] [--load-raw] [--display] [--debug]\n input\n\nDetects keypoint locations in a 3d model.\n\npositional arguments:\n input Input mesh to process.\n\noptional arguments:\n -h, --help show this help message and exit\n --detector {media-pipe-pose,media-pipe-face}\n Detection method for 2d keypoint detection (default:\n media-pipe-pose).\n --resolution RESOLUTION\n Render resolution for each view pass (default: 512).\n --infinite-ray Send ray through mesh to infinity and use average of\n intersections (default: False)\n --generator {wrap3} Generator methods for output generation (default:\n wrap3).\n --config CONFIG Path to the configuration JSON file.\n --load-raw Load mesh raw without post-processing (default: False)\n --display Shows result rendering with keypoints (default: False)\n --debug Shows debug frames and information (default: False)\n```\n\n### Library\nIt is also possible to use Muke as a library to detect keypoints on an existing 3d mesh.\n\n```python\nimport open3d as o3d\n\nfrom muke.Muke import Muke\nfrom muke.detector.MediaPipePoseDetector import MediaPipePoseDetector\nfrom muke.model.DetectionView import DetectionView\n\n# load mesh from filesystem\nmesh = o3d.io.read_triangle_mesh(\"assets/person.ply\")\n\n# define rendered views\nkeypoint_indexes = {28, 27, 26, 25, 24, 23, 12, 11, 14, 13, 16, 15, 5, 2, 0}\nviews = [\n DetectionView(\"front\", 0, keypoint_indexes),\n DetectionView(\"back\", 180, keypoint_indexes),\n]\n\n# detect keypoints\nwith Muke(MediaPipePoseDetector()) as m:\n result = m.detect(mesh, views)\n\n# present results\nfor kp in result:\n print(f\"KP {kp.index}: {kp.x:.2f} {kp.y:.2f} {kp.z:.2f}\")\n```\n\n### Detectors\nIt is possible to implement custom keypoint detectors. The custom detector has to implement the `BaseDetector` interface as shown in the following example.\n\n```python\nimport numpy as np\n\nfrom muke.detector.BaseDetector import BaseDetector\nfrom muke.detector.KeyPoint2 import KeyPoint2\n\n\nclass CustomDetector(BaseDetector):\n def setup(self):\n # todo: initialize the custom detector\n pass\n\n def detect(self, image: np.ndarray) -> [KeyPoint2]:\n # todo: implement the custom 2d keypoint detection \n pass\n\n def release(self):\n # todo: clean up allocated resources\n pass\n```\n\n### Renderer\nThe current version uses [pygfx](https://github.com/pygfx/pygfx) as lightweight and offscreen renderer, [trimesh](https://github.com/mikedh/trimesh) for model loading into pygfx and [Open3D](https://github.com/isl-org/Open3D) for raycasting. Initially, [trimesh](https://github.com/mikedh/trimesh) was used for everything, which is archived in the [trimesh-renderer branch](https://github.com/cansik/multiview-3d-keypoint-detection/tree/trimesh-renderer). Open3D was also once used for everything, but has been archived in version `0.2.x` and the [open3d-renderer branch](https://github.com/cansik/multiview-3d-keypoint-detection/tree/open3d-renderer).\n\n### About\nMIT License - Copyright (c) 2024 Florian Bruggisser\n\n\n",
"bugtrack_url": null,
"license": "MIT License",
"summary": "A simple approach to 3D keypoint detection using 2D estimation methods and multiview rendering.",
"version": "0.3.5.3",
"project_urls": {
"Homepage": "https://github.com/cansik/multiview-3d-keypoint-detection"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2ad7ab73fefcf9491c8c8abb273cf727da35398a2e84df4a2838388728f1b5d0",
"md5": "cb57075782f354af93f7620fad4d5174",
"sha256": "5433332dc9f9654fdd7c4e23d2623165625e4f3a44b9cbcdd261d3d9b5af7a2c"
},
"downloads": -1,
"filename": "muke-0.3.5.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "cb57075782f354af93f7620fad4d5174",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 19137,
"upload_time": "2024-03-07T11:41:20",
"upload_time_iso_8601": "2024-03-07T11:41:20.956681Z",
"url": "https://files.pythonhosted.org/packages/2a/d7/ab73fefcf9491c8c8abb273cf727da35398a2e84df4a2838388728f1b5d0/muke-0.3.5.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-07 11:41:20",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "cansik",
"github_project": "multiview-3d-keypoint-detection",
"github_not_found": true,
"lcname": "muke"
}
JSON
