| Name | mujoco-ar JSON |
| Version |
1.3.0
JSON |
| download |
| home_page | https://github.com/omarrayyann/mujocoar |
| Summary | An AR controller designed to seamlessly control objects in MuJoCo and for other general uses by receiving position and rotation data from a connected iOS application. |
| upload_time | 2025-01-18 02:29:27 |
| maintainer | None |
| docs_url | None |
| author | Omar Rayyan |
| requires_python | >=3.6 |
| license | MIT |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# <img src="https://github.com/user-attachments/assets/9b731c7f-7ad1-4607-90aa-f6ff1830a936" width="50" align="center" alt="Logo"> MuJoCo AR
[](https://pypi.org/project/mujoco_ar/) [](https://pepy.tech/project/mujoco_ar) [](https://opensource.org/licenses/MIT)
MuJoCo AR is a plugin for [MuJoCo](https://github.com/google-deepmind/mujoco) that enables the integration of ARKit data from a connected iOS device to control MuJoCo frames in real-time. It can also be used in non-MuJoCo applications (see the flexible setup section below). This package accompanies the MuJoCo AR iOS app that can be downloaded from [here](https://apps.apple.com/ae/app/mujoco-ar/id6612039501). Check the [flexible setup](#flexible-setup-works-without-mujoco) if you wanna use it outside of MuJoCo. Finally, check the [FAQ](#faq) below.
### Recent Updates
- You can now trigger a vibration with configurable properties. This can be used to add haptic feedback, whether on contact with objects or to notify the user of specific events. More details in [release log](https://github.com/omarrayyann/MujocoAR/releases/tag/v1.2.0).
## Real Demo
## MuJoCo Demos
<table>
<tr>
<th colspan="4">
Position Control
</th>
</tr>
<tr>
<th colspan="2">
<a href="https://github.com/omarrayyann/mujoco_fruit_picking" target="_blank">MuJoCo Fruits Pick and Place</a>
</th>
<th colspan="2">
<a href="https://github.com/omarrayyann/mujoco_pusht" target="_blank">MuJoCo PushT</a>
</th>
</tr>
<tr>
<td><img src="https://github.com/user-attachments/assets/3d496ce1-0b5d-4a1f-a6d2-dc2e19d1e3d8" /></td>
<td><img src="https://github.com/user-attachments/assets/8fd2b0ae-f90a-4df5-b114-3feac7c87e37" /></td>
<td><img src="https://github.com/user-attachments/assets/c1e927c5-a4af-4c95-a6d0-fe7f8a026c34" /></td>
<td><img src="https://github.com/user-attachments/assets/a58ed764-4e05-40a5-b26a-5bd896584f34"/></td>
</tr>
<tr>
<th colspan="4">
Position and Rotation Control
</th>
</tr>
<tr>
<th colspan="2">
<a href="https://github.com/omarrayyann/mujoco_study_desk" target="_blank">MuJoCo Study Desk</a>
</th>
<th colspan="2">
<a href="https://github.com/omarrayyann/mujoco_blocks_stacking" target="_blank">MuJoCo Blocks Stacking</a>
</th>
</tr>
<tr>
<td><img src="https://github.com/user-attachments/assets/e70569ce-5ade-4161-95ab-007b1d612e0a" /></td>
<td><img src="https://github.com/user-attachments/assets/88635d5e-63f3-41b3-af83-3af03588c84f" /></td>
<td><img src="https://github.com/user-attachments/assets/dbb1dbb7-5dff-4c24-88fb-9f4b8afd7d8b" /></td>
<td><img src="https://github.com/user-attachments/assets/df43bb40-6e58-4e94-8d1c-a4fa90359d65" /></td>
</tr>
</table>
Examples of MuJoCo AR linked to the end-effectors of multiple manipulators can be found in this fork of [Mink](https://github.com/omarrayyann/mink-mujocoAR). For a simpler environemnt that the ones shwon in the demo section above, check the example by using ```mjpython demo/simple.py```
## Installation
1. iOS App: Download the iOS app from the App Store [here](https://apps.apple.com/ae/app/mujoco-ar/id6612039501)
2. Python Package: Install the Python package using pip:
```bash
pip install mujoco_ar
```
## Usage
### Quick MuJoCo Setup
This setup allows you to directly control a MuJoCo frame (body, geom, or site), with the frame's position and orientation matching the ARKit data received from the connected iOS device.
```python
from mujoco_ar import MujocoARConnector
# Initialize the connector with your desired parameters
mj_ar = MujocoARConnector(mujoco_model=my_model,mujoco_data=my_data)
# Link a MuJoCo frame (link_body(), link_geom() or link_site())
mj_ar.link_body(name="eef_target")
# Start the connector
mj_ar.start()
```
### Full MuJoCo Setup
In addition to what the quick setup allows you to do, this setup allows you to automate the applying of a translation, rotation or scaling of the recieved pose. Additionally, you can pass functions to button_fn and toggle_fn to be triggered when the button or toggle are activated
```python
from mujoco_ar import MujocoARConnector
# Initialize the connector with your desired parameters
mj_ar = MujocoARConnector(
mujoco_model=my_model,
mujoco_data=my_data,
port=8888, # Optional, defaults to 8888 if not provided
debug=False # Optional, defaults to False if not provided
)
# Link a MuJoCo frame (link_body(), link_geom() or link_site())
mj_ar.link_body(
name="eef_target",
scale=1.0, # Optional, defaults to 1.0 if not provided
position_origin=np.array([0.0, 0.0, 0.0]), # Optional, defaults to [0, 0, 0] if not provided
rotation_origin=np.identity(3), # Optional, defaults to I(3) if not provided
toggle_fn=my_toggle_function, # Optional, calls nothing if not provided
button_fn=my_button_function, # Optional, calls nothing if not provided
disable_pos=False, # Optional, defaults to False if not provided
disable_rot=False # Optional, defaults to False if not provided
)
# Start the connector
mj_ar.start()
```
### Flexible Setup (works without MuJoCo):
You can retrieve the ARKit data including the position, rotation, button, and toggle states directly from a connected iOS device, making it flexible for usage in various applications beyond physics simulations. Try running ```mjpython demos/flexible_setup.py```.
```python
from mujoco_ar import MujocoARConnector
# Initialize the connector with your desired parameters
connector = MujocoARConnector()
# Start the connector
connector.start()
# Retrieve the latest AR data (after connecting the iOS device, see the guide below)
data = connector.get_latest_data() # Returns {"position": (3, 1), "rotation": (3, 3), "button": bool, "toggle": bool}
```
## Additional Functions
```python
connector.vibrate(sharpness=0.8, intensity=0.4, duration=0.01) # Trigger a vibration on the connected device
connector.pause_updates() # Temporarily stops receiving updates from the connected device.
connector.resume_updates() # Resumes receiving updates from the connected device.
connector.reset_position() # Resets the current position as the origin (0,0,0).
```
## FAQ
#### How can I reduce latency?
- If you're experiencing latency, try connecting your PC to your device's hotspot. This should significantly reduce latency if you're far from a router since the communication happens locally via WebSockets.
#### It is not tracking my trajectory accurately. What can I do?
- Make sure you're not covering the camera with your fingers. If the tracking is too sensitive, you can adjust the `scale` attribute when linking the frame to make it less reactive to minor movements. In my experience, the position retrived is more accurate than the rotation.
#### Where can I start?
- I highly recommend running `demo/simple.py` to test things out if you're just getting started. This demo is configured to change the sphere's color with a button press and enlarge it with a toggle, providing a basic but effective introduction to a simple configuration.
#### Can I use it for a non-MuJoCo application?
- Yes, check the [Flexible Setup](#flexible-setup-works-without-mujoco) out where you can retrive the pure ARKit position and rotation and use it as you wish. You wouldn't need to pass in the MuJoCo model and data in such a case.
## How to contribute?
Report any bugs you encounter to the [issue tracker](https://github.com/omarrayyann/MujocoAR/issues). Also, feel free to suggest any features. Those could be regarding the app ([iOS swift code](https://github.com/omarrayyann/MujocoAR-iOS-App)) or the python package.
## Acknowledgement
Thanks to [@kevinzakka](https://www.github.com/kevinzakka) for the [mink](https://github.com/kevinzakka/mink) and [mjctrl](https://github.com/kevinzakka/mjctrl) libraries, which are used in the provided demos.
## Author
Omar Rayyan (olr7742@nyu.edu)
Raw data
{
"_id": null,
"home_page": "https://github.com/omarrayyann/mujocoar",
"name": "mujoco-ar",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": null,
"keywords": null,
"author": "Omar Rayyan",
"author_email": "olr7742@nyu.edu",
"download_url": "https://files.pythonhosted.org/packages/3a/60/79da96adf24459b3b7a9f140c5421c5f8fa78e9348c49d556f3e77d140ed/mujoco_ar-1.3.0.tar.gz",
"platform": null,
"description": "# <img src=\"https://github.com/user-attachments/assets/9b731c7f-7ad1-4607-90aa-f6ff1830a936\" width=\"50\" align=\"center\" alt=\"Logo\"> MuJoCo AR\n\n[](https://pypi.org/project/mujoco_ar/) [](https://pepy.tech/project/mujoco_ar) [](https://opensource.org/licenses/MIT) \n\nMuJoCo AR is a plugin for [MuJoCo](https://github.com/google-deepmind/mujoco) that enables the integration of ARKit data from a connected iOS device to control MuJoCo frames in real-time. It can also be used in non-MuJoCo applications (see the flexible setup section below). This package accompanies the MuJoCo AR iOS app that can be downloaded from [here](https://apps.apple.com/ae/app/mujoco-ar/id6612039501). Check the [flexible setup](#flexible-setup-works-without-mujoco) if you wanna use it outside of MuJoCo. Finally, check the [FAQ](#faq) below.\n\n### Recent Updates\n\n- You can now trigger a vibration with configurable properties. This can be used to add haptic feedback, whether on contact with objects or to notify the user of specific events. More details in [release log](https://github.com/omarrayyann/MujocoAR/releases/tag/v1.2.0).\n\n\n## Real Demo\n\n\n\n\n## MuJoCo Demos\n\n<table>\n <tr>\n <th colspan=\"4\">\n Position Control\n </th>\n </tr>\n <tr>\n <th colspan=\"2\">\n <a href=\"https://github.com/omarrayyann/mujoco_fruit_picking\" target=\"_blank\">MuJoCo Fruits Pick and Place</a>\n </th>\n <th colspan=\"2\">\n <a href=\"https://github.com/omarrayyann/mujoco_pusht\" target=\"_blank\">MuJoCo PushT</a>\n </th>\n </tr>\n <tr>\n <td><img src=\"https://github.com/user-attachments/assets/3d496ce1-0b5d-4a1f-a6d2-dc2e19d1e3d8\" /></td>\n <td><img src=\"https://github.com/user-attachments/assets/8fd2b0ae-f90a-4df5-b114-3feac7c87e37\" /></td>\n <td><img src=\"https://github.com/user-attachments/assets/c1e927c5-a4af-4c95-a6d0-fe7f8a026c34\" /></td>\n <td><img src=\"https://github.com/user-attachments/assets/a58ed764-4e05-40a5-b26a-5bd896584f34\"/></td>\n </tr>\n <tr>\n <th colspan=\"4\">\n Position and Rotation Control\n </th>\n </tr>\n <tr>\n <th colspan=\"2\">\n <a href=\"https://github.com/omarrayyann/mujoco_study_desk\" target=\"_blank\">MuJoCo Study Desk</a>\n </th>\n <th colspan=\"2\">\n <a href=\"https://github.com/omarrayyann/mujoco_blocks_stacking\" target=\"_blank\">MuJoCo Blocks Stacking</a>\n </th>\n </tr>\n <tr>\n <td><img src=\"https://github.com/user-attachments/assets/e70569ce-5ade-4161-95ab-007b1d612e0a\" /></td>\n <td><img src=\"https://github.com/user-attachments/assets/88635d5e-63f3-41b3-af83-3af03588c84f\" /></td>\n <td><img src=\"https://github.com/user-attachments/assets/dbb1dbb7-5dff-4c24-88fb-9f4b8afd7d8b\" /></td>\n <td><img src=\"https://github.com/user-attachments/assets/df43bb40-6e58-4e94-8d1c-a4fa90359d65\" /></td>\n </tr>\n</table>\n\nExamples of MuJoCo AR linked to the end-effectors of multiple manipulators can be found in this fork of [Mink](https://github.com/omarrayyann/mink-mujocoAR). For a simpler environemnt that the ones shwon in the demo section above, check the example by using ```mjpython demo/simple.py```\n\n## Installation\n\n1. iOS App: Download the iOS app from the App Store [here](https://apps.apple.com/ae/app/mujoco-ar/id6612039501)\n\n2. Python Package: Install the Python package using pip:\n\n```bash\npip install mujoco_ar\n```\n\n## Usage\n\n### Quick MuJoCo Setup\n\nThis setup allows you to directly control a MuJoCo frame (body, geom, or site), with the frame's position and orientation matching the ARKit data received from the connected iOS device.\n\n```python\nfrom mujoco_ar import MujocoARConnector\n\n# Initialize the connector with your desired parameters\nmj_ar = MujocoARConnector(mujoco_model=my_model,mujoco_data=my_data)\n\n# Link a MuJoCo frame (link_body(), link_geom() or link_site())\nmj_ar.link_body(name=\"eef_target\")\n\n# Start the connector\nmj_ar.start()\n```\n### Full MuJoCo Setup\n\nIn addition to what the quick setup allows you to do, this setup allows you to automate the applying of a translation, rotation or scaling of the recieved pose. Additionally, you can pass functions to button_fn and toggle_fn to be triggered when the button or toggle are activated\n\n```python\nfrom mujoco_ar import MujocoARConnector\n\n# Initialize the connector with your desired parameters\nmj_ar = MujocoARConnector(\n mujoco_model=my_model, \n mujoco_data=my_data, \n port=8888, # Optional, defaults to 8888 if not provided\n debug=False # Optional, defaults to False if not provided\n)\n\n# Link a MuJoCo frame (link_body(), link_geom() or link_site())\nmj_ar.link_body(\n name=\"eef_target\",\n scale=1.0, # Optional, defaults to 1.0 if not provided\n position_origin=np.array([0.0, 0.0, 0.0]), # Optional, defaults to [0, 0, 0] if not provided\n rotation_origin=np.identity(3), # Optional, defaults to I(3) if not provided\n toggle_fn=my_toggle_function, # Optional, calls nothing if not provided\n button_fn=my_button_function, # Optional, calls nothing if not provided\n disable_pos=False, # Optional, defaults to False if not provided\n disable_rot=False # Optional, defaults to False if not provided\n)\n\n# Start the connector\nmj_ar.start()\n```\n\n### Flexible Setup (works without MuJoCo):\n\nYou can retrieve the ARKit data including the position, rotation, button, and toggle states directly from a connected iOS device, making it flexible for usage in various applications beyond physics simulations. Try running ```mjpython demos/flexible_setup.py```.\n\n```python\nfrom mujoco_ar import MujocoARConnector\n\n# Initialize the connector with your desired parameters\nconnector = MujocoARConnector()\n\n# Start the connector\nconnector.start()\n\n# Retrieve the latest AR data (after connecting the iOS device, see the guide below)\ndata = connector.get_latest_data() # Returns {\"position\": (3, 1), \"rotation\": (3, 3), \"button\": bool, \"toggle\": bool}\n```\n## Additional Functions\n\n```python\nconnector.vibrate(sharpness=0.8, intensity=0.4, duration=0.01) # Trigger a vibration on the connected device\nconnector.pause_updates() # Temporarily stops receiving updates from the connected device.\nconnector.resume_updates() # Resumes receiving updates from the connected device.\nconnector.reset_position() # Resets the current position as the origin (0,0,0).\n\n```\n\n## FAQ\n\n#### How can I reduce latency?\n\n- If you're experiencing latency, try connecting your PC to your device's hotspot. This should significantly reduce latency if you're far from a router since the communication happens locally via WebSockets.\n\n#### It is not tracking my trajectory accurately. What can I do?\n\n- Make sure you're not covering the camera with your fingers. If the tracking is too sensitive, you can adjust the `scale` attribute when linking the frame to make it less reactive to minor movements. In my experience, the position retrived is more accurate than the rotation.\n\n#### Where can I start?\n\n- I highly recommend running `demo/simple.py` to test things out if you're just getting started. This demo is configured to change the sphere's color with a button press and enlarge it with a toggle, providing a basic but effective introduction to a simple configuration.\n \n#### Can I use it for a non-MuJoCo application?\n\n- Yes, check the [Flexible Setup](#flexible-setup-works-without-mujoco) out where you can retrive the pure ARKit position and rotation and use it as you wish. You wouldn't need to pass in the MuJoCo model and data in such a case.\n\n## How to contribute?\n\nReport any bugs you encounter to the [issue tracker](https://github.com/omarrayyann/MujocoAR/issues). Also, feel free to suggest any features. Those could be regarding the app ([iOS swift code](https://github.com/omarrayyann/MujocoAR-iOS-App)) or the python package.\n\n## Acknowledgement\n\nThanks to [@kevinzakka](https://www.github.com/kevinzakka) for the [mink](https://github.com/kevinzakka/mink) and [mjctrl](https://github.com/kevinzakka/mjctrl) libraries, which are used in the provided demos.\n\n## Author\n\nOmar Rayyan (olr7742@nyu.edu)\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "An AR controller designed to seamlessly control objects in MuJoCo and for other general uses by receiving position and rotation data from a connected iOS application.",
"version": "1.3.0",
"project_urls": {
"Homepage": "https://github.com/omarrayyann/mujocoar"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "108cc555d9c1a74f552ab6c3d4267e4abe196cf5d5b19d0f77b762b92117fc0c",
"md5": "4e67dfaef8544774e1f3177fc05ee910",
"sha256": "130e19e55831a97bc7538cdf65f2a685c57795060b192f4cb6cb032669313831"
},
"downloads": -1,
"filename": "mujoco_ar-1.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "4e67dfaef8544774e1f3177fc05ee910",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 9921,
"upload_time": "2025-01-18T02:29:25",
"upload_time_iso_8601": "2025-01-18T02:29:25.672765Z",
"url": "https://files.pythonhosted.org/packages/10/8c/c555d9c1a74f552ab6c3d4267e4abe196cf5d5b19d0f77b762b92117fc0c/mujoco_ar-1.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3a6079da96adf24459b3b7a9f140c5421c5f8fa78e9348c49d556f3e77d140ed",
"md5": "7bfd804136e0632b17c54aef2d5581f5",
"sha256": "76be299f6238a395dbda7b2a8351a49f42827aa0efaef25ba7083dcea6a16dcd"
},
"downloads": -1,
"filename": "mujoco_ar-1.3.0.tar.gz",
"has_sig": false,
"md5_digest": "7bfd804136e0632b17c54aef2d5581f5",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 12728,
"upload_time": "2025-01-18T02:29:27",
"upload_time_iso_8601": "2025-01-18T02:29:27.426001Z",
"url": "https://files.pythonhosted.org/packages/3a/60/79da96adf24459b3b7a9f140c5421c5f8fa78e9348c49d556f3e77d140ed/mujoco_ar-1.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-18 02:29:27",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "omarrayyann",
"github_project": "mujocoar",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "mujoco-ar"
}
