# MuJoCo AR Viewer
A Python package for visualizing MuJoCo physics simulations in Augmented Reality using Apple Vision Pro and other AR devices.
## Installation
### Python API
```bash
pip install mujoco-ar-viewer
```
To use automatic MuJoCo XML-to-USD conversion feature (supported only on Linux and Windows via [mujoco-usd-converter](https://github.com/newton-physics/mujoco-usd-converter) from project [Newton](https://github.com/newton-physics)), use:
```bash
pip install "mujoco-ar-viewer[usd]"
```
### VisionOS App
Open App Store on VisionOS, and search for [mujocoARViewer].
## Quick Start
```python
from mujoco_arviewer import MJARViewer
import mujoco
# path to mujoco XML
xml_path = "path/to/your/model.xml"
# Set up your MuJoCo simulation
model = mujoco.MjModel.from_xml_path(xml_path)
data = mujoco.MjData(model)
# Initialize the AR viewer with your device's IP
# Device's IP will be presented when you launch the app
viewer = MJARViewer(avp_ip="192.168.1.100", \
enable_hand_tracking = True)
# Send a MuJoCo model to the AR device
# (Linux Only) it will automatically convert to USD
viewer.load_scene(xml_path)
# Register the model and data with the viewer
viewer.register(model, data)
# Simulation loop
while True:
# (Optional) access hand tracking results
hand_tracking = viewer.get_hand_tracking()
# (Optional) map hand tracking to mujoco ctrl
data.ctrl = hand2ctrl(hand_tracking)
# Step the simulation
mujoco.mj_step(model, data)
# Sync with AR device
viewer.sync()
```
## Recommended Read
### Where to attach your mujoco `world` frame
Since this is a viewer in augmented reality (which by defintion, blends your simulated environment with your real world environment), deciding where to attach your simulation scene's `world` frame in your actual physical space in real world is important. You can determine this by passing in `attach_to` as an argument either by
1. a 7-dim vector of `xyz` translation and scalar-first quaternion representation (i.e., `[x,y,z,qw,qx,qy,qz]`)
2. a 4-dim vector of `xyz` translation and rotation around `z-axis`, specified as a degree. (i.e., `[x,y,z,zrot]`)
```python
# attach the `world` frame 0.3m above the visionOS origin, rotating 90 degrees around z-axis.
viewer.load_scene(scene_path, attach_to=[0, 0, 0.3, 90])
```
1. **Default Setting**: When `viewer.load_scene` is called without `attach_to` specified, it attahces the simualtion scene to the origin frame registered inside VisionOS. VisionOS automatically detects the physical ground of your surrounding using its sensors and defines the origin on the ground. For instance, if you're standing, visionOS will attach origin frame right below your feet. If you're sitting down, it's gonna be right below your chair. For most *humanoid/Quadruped Locomotion* scenes or *mobile manipulation* scenes, for instance, the world frame is often defined on a surface that can be considered as a "ground". Then you don't need no offset, at least for the `z-axis`. Based on your use cases, you might still want to some offset for `x` and `y` translation, or rotation around `z-axis`.
2. **Custom Setting**: For many other cases, you might want to define a custom position to attach the `world` frame of a simulation scene. For most **Table-top Manipulation Scenes**, for instance, if your XML file is designed for table-top manipulation using fixed-base manipulators with your world frame defined on the surface of the table, you might want to attach the `world` frame with a slight `z-axis` offset in your AR environment.
| Examples from [MuJoCo Menagerie](https://github.com/google-deepmind/mujoco_menagerie) | [Unitree G1 XML](https://github.com/google-deepmind/mujoco_menagerie/tree/main/unitree_g1/scene.xml) | [Google Robot XML](https://github.com/google-deepmind/mujoco_menagerie/tree/main/google_robot/scene.xml) | [ALOHA 2 XML](https://github.com/google-deepmind/mujoco_menagerie/blob/main/aloha/scene.xml) |
|-------|---------|----------|----------|
| Visualization of `world` frame |  |  |  |
| | `world` frame is attached on a "ground". | `world` frame is attached on a "ground". | `world` frame is attached on a "table". |
| Recommended `attach_to` | Default Setting | Default Setting | Offset in `z-axis`, that can bring up the table surface to reasonable height in your real world. |
## FAQ
1. Why did you develop this package?
Collecting robot demonstrations in simulation is often useful and necessary for conducting research on robot learning pipelines. However, if you try to do that by watching a 2D viewer of a simulation scene on a 2D monitor, you quickly run into limitations since you really have no accurate perception of depth. Presenting a simulation scene as an AR environment offers a nice solution. Consider this as a **3D-lifted version of your existing 2D `mujoco.viewer`.**
2. Why is USD conversion only supported on Linux and Windows, and how should I use this for macOS then?
Limited macOS compatibility for automatic XML-to-USD conversion comes from [mujoco-usd-converter](https://github.com/newton-physics/mujoco-usd-converter), which internally relies on [OpenUSD Exchange SDK](https://github.com/NVIDIA-Omniverse/usd-exchange) which only supports Linux and Windows at this moment. For now, if you want to use this software on macOS, you can separately convert your XML into USD using [mujoco-usd-converter](https://github.com/newton-physics/mujoco-usd-converter) and bring the USD file over to macOS. Then, you instead of specifying a path to XML file, you can specify a path to USD file when calling `load_scene`:
```python
# instead of passing in a path to XML, pass in a path to converted USDZ
# convert XML to USDZ with any Linux system you have access to
viewer.load_scene("/path/to/your/converted/scene.usdz")
```
3. What axis convention does hand-tracking data stream use?
It uses the convention defined in [VisionProTeleop](https://github.com/Improbable-AI/VisionProTeleop).
## License
MIT License
Raw data
{
"_id": null,
"home_page": null,
"name": "mujoco-ar-viewer",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "mujoco, ar, vr, visualization, physics, simulation",
"author": "Improbable AI",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/b4/d1/05ef65d2d84f7db53f722c066df4c4dbf8eb7dbc06abfebf90a2ecf98cdb/mujoco_ar_viewer-0.3.6.tar.gz",
"platform": null,
"description": "# MuJoCo AR Viewer\n\nA Python package for visualizing MuJoCo physics simulations in Augmented Reality using Apple Vision Pro and other AR devices.\n\n\n\n\n## Installation\n\n### Python API \n\n```bash\npip install mujoco-ar-viewer\n```\n\nTo use automatic MuJoCo XML-to-USD conversion feature (supported only on Linux and Windows via [mujoco-usd-converter](https://github.com/newton-physics/mujoco-usd-converter) from project [Newton](https://github.com/newton-physics)), use: \n\n```bash \npip install \"mujoco-ar-viewer[usd]\"\n```\n\n\n### VisionOS App \n\nOpen App Store on VisionOS, and search for [mujocoARViewer]. \n\n## Quick Start\n\n```python\nfrom mujoco_arviewer import MJARViewer\nimport mujoco\n\n# path to mujoco XML \nxml_path = \"path/to/your/model.xml\"\n\n# Set up your MuJoCo simulation\nmodel = mujoco.MjModel.from_xml_path(xml_path)\ndata = mujoco.MjData(model)\n\n# Initialize the AR viewer with your device's IP\n# Device's IP will be presented when you launch the app \nviewer = MJARViewer(avp_ip=\"192.168.1.100\", \\\n enable_hand_tracking = True)\n# Send a MuJoCo model to the AR device\n# (Linux Only) it will automatically convert to USD\nviewer.load_scene(xml_path) \n# Register the model and data with the viewer\nviewer.register(model, data)\n\n# Simulation loop\nwhile True:\n # (Optional) access hand tracking results \n hand_tracking = viewer.get_hand_tracking() \n # (Optional) map hand tracking to mujoco ctrl\n data.ctrl = hand2ctrl(hand_tracking)\n\n # Step the simulation\n mujoco.mj_step(model, data)\n # Sync with AR device\n viewer.sync()\n```\n\n## Recommended Read \n\n### Where to attach your mujoco `world` frame \n\n\nSince this is a viewer in augmented reality (which by defintion, blends your simulated environment with your real world environment), deciding where to attach your simulation scene's `world` frame in your actual physical space in real world is important. You can determine this by passing in `attach_to` as an argument either by \n1. a 7-dim vector of `xyz` translation and scalar-first quaternion representation (i.e., `[x,y,z,qw,qx,qy,qz]`)\n2. a 4-dim vector of `xyz` translation and rotation around `z-axis`, specified as a degree. (i.e., `[x,y,z,zrot]`)\n\n```python \n# attach the `world` frame 0.3m above the visionOS origin, rotating 90 degrees around z-axis. \nviewer.load_scene(scene_path, attach_to=[0, 0, 0.3, 90]) \n```\n\n1. **Default Setting**: When `viewer.load_scene` is called without `attach_to` specified, it attahces the simualtion scene to the origin frame registered inside VisionOS. VisionOS automatically detects the physical ground of your surrounding using its sensors and defines the origin on the ground. For instance, if you're standing, visionOS will attach origin frame right below your feet. If you're sitting down, it's gonna be right below your chair. For most *humanoid/Quadruped Locomotion* scenes or *mobile manipulation* scenes, for instance, the world frame is often defined on a surface that can be considered as a \"ground\". Then you don't need no offset, at least for the `z-axis`. Based on your use cases, you might still want to some offset for `x` and `y` translation, or rotation around `z-axis`. \n\n2. **Custom Setting**: For many other cases, you might want to define a custom position to attach the `world` frame of a simulation scene. For most **Table-top Manipulation Scenes**, for instance, if your XML file is designed for table-top manipulation using fixed-base manipulators with your world frame defined on the surface of the table, you might want to attach the `world` frame with a slight `z-axis` offset in your AR environment. \n\n\n\n| Examples from [MuJoCo Menagerie](https://github.com/google-deepmind/mujoco_menagerie) | [Unitree G1 XML](https://github.com/google-deepmind/mujoco_menagerie/tree/main/unitree_g1/scene.xml) | [Google Robot XML](https://github.com/google-deepmind/mujoco_menagerie/tree/main/google_robot/scene.xml) | [ALOHA 2 XML](https://github.com/google-deepmind/mujoco_menagerie/blob/main/aloha/scene.xml) |\n|-------|---------|----------|----------|\n| Visualization of `world` frame |  |  |  |\n| | `world` frame is attached on a \"ground\". | `world` frame is attached on a \"ground\". | `world` frame is attached on a \"table\". |\n| Recommended `attach_to` | Default Setting | Default Setting | Offset in `z-axis`, that can bring up the table surface to reasonable height in your real world. |\n\n\n## FAQ \n\n1. Why did you develop this package? \n\n Collecting robot demonstrations in simulation is often useful and necessary for conducting research on robot learning pipelines. However, if you try to do that by watching a 2D viewer of a simulation scene on a 2D monitor, you quickly run into limitations since you really have no accurate perception of depth. Presenting a simulation scene as an AR environment offers a nice solution. Consider this as a **3D-lifted version of your existing 2D `mujoco.viewer`.** \n\n2. Why is USD conversion only supported on Linux and Windows, and how should I use this for macOS then? \n\n Limited macOS compatibility for automatic XML-to-USD conversion comes from [mujoco-usd-converter](https://github.com/newton-physics/mujoco-usd-converter), which internally relies on [OpenUSD Exchange SDK](https://github.com/NVIDIA-Omniverse/usd-exchange) which only supports Linux and Windows at this moment. For now, if you want to use this software on macOS, you can separately convert your XML into USD using [mujoco-usd-converter](https://github.com/newton-physics/mujoco-usd-converter) and bring the USD file over to macOS. Then, you instead of specifying a path to XML file, you can specify a path to USD file when calling `load_scene`:\n\n ```python\n # instead of passing in a path to XML, pass in a path to converted USDZ \n # convert XML to USDZ with any Linux system you have access to \n viewer.load_scene(\"/path/to/your/converted/scene.usdz\")\n ```\n\n3. What axis convention does hand-tracking data stream use? \n\n It uses the convention defined in [VisionProTeleop](https://github.com/Improbable-AI/VisionProTeleop). \n\n \n  \n\n\n\n## License\n\nMIT License\n\n",
"bugtrack_url": null,
"license": null,
"summary": "AR visualization for MuJoCo physics simulations",
"version": "0.3.6",
"project_urls": {
"Homepage": "https://github.com/Improbable-AI/mujocoARViewer",
"Issues": "https://github.com/Improbable-AI/mujocoARViewer/issues",
"Repository": "https://github.com/Improbable-AI/mujocoARViewer"
},
"split_keywords": [
"mujoco",
" ar",
" vr",
" visualization",
" physics",
" simulation"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "bfb0caefdf810a8518d1b0bdf7a45df64fb55b9525729e8efcab861daef981ea",
"md5": "38ea8f67f1f40d677c1f0abe7169a33a",
"sha256": "56c71527f81f20269cc192322143f388db1070fab8b80d9761bd78d87406635b"
},
"downloads": -1,
"filename": "mujoco_ar_viewer-0.3.6-py3-none-any.whl",
"has_sig": false,
"md5_digest": "38ea8f67f1f40d677c1f0abe7169a33a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 15515,
"upload_time": "2025-10-21T00:41:32",
"upload_time_iso_8601": "2025-10-21T00:41:32.797516Z",
"url": "https://files.pythonhosted.org/packages/bf/b0/caefdf810a8518d1b0bdf7a45df64fb55b9525729e8efcab861daef981ea/mujoco_ar_viewer-0.3.6-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "b4d105ef65d2d84f7db53f722c066df4c4dbf8eb7dbc06abfebf90a2ecf98cdb",
"md5": "44fd41eae3a0be7c2af8e2a5346be201",
"sha256": "9a776019568e9630d0ad9cef7062150b5d76eeb7e2ab094c405b21950f9644dc"
},
"downloads": -1,
"filename": "mujoco_ar_viewer-0.3.6.tar.gz",
"has_sig": false,
"md5_digest": "44fd41eae3a0be7c2af8e2a5346be201",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 14965,
"upload_time": "2025-10-21T00:41:34",
"upload_time_iso_8601": "2025-10-21T00:41:34.616470Z",
"url": "https://files.pythonhosted.org/packages/b4/d1/05ef65d2d84f7db53f722c066df4c4dbf8eb7dbc06abfebf90a2ecf98cdb/mujoco_ar_viewer-0.3.6.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-10-21 00:41:34",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Improbable-AI",
"github_project": "mujocoARViewer",
"github_not_found": true,
"lcname": "mujoco-ar-viewer"
}
JSON
