| Name | caliscope JSON |
| Version |
0.5.1
JSON |
| download |
| home_page | None |
| Summary | GUI based multicamera calibration that integrates with 2D landmark tracking to triangulate 3D landmark positions |
| upload_time | 2024-09-25 21:54:52 |
| maintainer | None |
| docs_url | None |
| author | Mac Prible |
| requires_python | <3.13,>=3.10 |
| license | BSD-2-Clause |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<div align="center">
# Caliscope
*Multicamera Calibration + Pose Estimation --> Open Source Motion Capture*
[](https://pypi.org/project/caliscope/)
[](https://opensource.org/license/bsd-2-clause/)
[](https://pypi.org/project/caliscope/)
[](https://github.com/mprib/caliscope/commits)
[](https://github.com/mprib/caliscope/stargazers)
</div>
## About
`Caliscope` is a GUI-based multicamera calibration package. When the intrinsic (focal length, optical center, and distortion) as well as extrinsic (relative rotation and translation) properties of a set of cameras are known, synchronized frames from those cameras can be used to triangulate landmarks identified across their multiple points of view. With more cameras, this 3D tracking becomes more robust to occlusion and the inevitable errors in 2D landmark tracking and camera property estimates.
While OpenCV provides straightforward functions for the estimation of single camera intrinsic properties as well as estimates of the extrinsic properties of two cameras, there is no straightforward way to estimate extrinsic properties for more than two cameras. Performing this requires [bundle adjustment](https://scipy-cookbook.readthedocs.io/items/bundle_adjustment.html), which demands an extensive series of computational steps and intermediate data tracking.
Caliscope automates this more complex calibration function along with providing visual feedback regarding parameter estimates at each stage of processing. Additionally, there are sample implementations of a Tracker class using Google's Mediapipe that demonstrate the capacity to integrate the full calibration results with landmark tracking tools to achieve 3D pose estimation. While Mediapipe pose estimation has limitations regarding accuracy and precision, it demonstrates a data processing pipeline that can easily integrate more powerful tracking tools as they emerge.
This README provides a general overview and quick guide to install `Calicope` on your system. For a more detailed description of the calibration process and workflow, please see our [docs](https://mprib.github.io/caliscope/).
**For a quick sense of how the calibration process works, check out the [video walk through](https://www.youtube.com/watch?v=voE3IKYtuIQ) of the [sample project](https://mprib.github.io/caliscope/sample_project/).**
---
### Demo Animation
https://github.com/mprib/caliscope/assets/31831778/803a4ce8-4012-4da1-87b9-66c5e6b31c59
*`Caliscope` was used to calibrate the cameras used in the above recordings (both intrinsic and extrinsic parameters), run 2D landmark estimation across all the videos using Holistic Mediapipe pose estimation, then triangulate the 3D landmark positions based on the camera system calibration.
The animation was created in Blender based on these 3D landmark estimates using a companion project called [Rigmarole](https://github.com/mprib/rigmarole).
The concurrent video was recorded with a 7 year old t440p laptop, 4 webcams (~$25 each) and another project called [MultiWebCam](https://github.com/mprib/multiwebcam).*
---
## Quick Start
Please note that given the size of some core dependencies (OpenCV, Mediapipe, and PySide6 are among them) installation and initial launch can take a while.
### Basic Installation and Launch
#### Windows
```bash
# Open Command Prompt and navigate to directory that will hold venv
# this does not need to be the same as where your project workspace is held
cd path\to\your\project
# Create a virtual environment named 'env' using Python 3.10
"C:\Path\To\Python3.10\python.exe" -m venv .venv
# Activate the virtual environment
.\.venv\Scripts\activate
# Your virtual environment is now active.
# You can install using pip
pip install caliscope
# Launch from the command line
caliscope
```
#### MacOS/Linux
```bash
# Open Command Prompt and navigate to directory that will hold venv
# this does not need to be the same as where your project workspace is held
cd path/to/your/project
# Create a virtual environment named 'venv' using Python 3.10
/path/to/python3.10 -m venv .venv
# Activate the virtual environment
source .venv/bin/activate
# Your virtual environment is now active.
# You can install using pip
pip3 install caliscope
# Launch from the command line
caliscope
```
### MacOS Note: Must adjust some environment variables
Thank you to @calcoloergosum for raising an issue regarding a bug in numpy on MacOS that was causing a crash (https://github.com/mprib/caliscope/issues/622).
It is advised to set the following environment variables on MacOS. If anyone has any additional details related to this or notices that the bug has been fixed, please let me know and I'll take this down.
```
export MKL_NUM_THREADS=1
export NUMEXPR_NUM_THREADS=1
export OMP_NUM_THREADS=1
```
### Basic Steps
1. Once the GUI launches, navigate to File->New/Open Project and create a folder to hold your project
- A basic [project structure](https://mprib.github.io/caliscope/project_setup/) will be created here
2. Define a Charuco calibration board via the Charuco tab and print it out, fixing it to something flat
3. Record footage for the calibration according to the guidelines for the [intrinsic](https://mprib.github.io/caliscope/intrinsic_calibration/) and [extrinsic](https://mprib.github.io/caliscope/extrinsic_calibration/) calibrations.
4. Record synchronized motion capture trials
- A companion project ([multiwebcam](https://github.com/mprib/multiwebcam)) has been set up to facilitate this though is still in early stages
5. Store video files within the project folder and reload the workspace
6. Run autocalibration on all cameras within the Cameras tab
7. Run "Calibration Capture Volume" from the Workspace Tab
8. Set origin within the capture volume (optional but helpful)
9. Run post-processing on individual recordings to generate 3D trajectory output
## Key Features
### Calibration board creation
- Easy creation of `png` files for ChArUco calibration boards
- board definition can be changed across intrinsic and extrinsic calibration allowing greater flexibiltiy
### Intrinsic Camera Calibration
- Automated calculation of camera intrinsic properties from input video
- Optical Center
- Focal Length
- Lens Distortion
- Visualization of distortion model to ensure reasonableness
### Extrinsic Camera Calibration
- Automated bundle adjustment to estimate 6 DoF relative position of cameras
- Visualizer to inspect the estimates from the bundle adjustment
- Setting of the World Origin within the visualizer to simplify data processing
### 3D Tracking
- Tracker API for integrating alternate tracking methods
- 3 sample implementations with Google Mediapipe (Hands/Pose/Holistic)
- Automated application of landmark tracking to synchronized videos
- Triangulation of 3D landmark position based on calibrated cameras
- Gap-filling and butterworth filtering to smooth trajectory estimates
### Trajectory Output
- output to `.trc` file format for use in biomechanical modelling
- output to tidy `.csv` format with well-labelled headers for straightforward integration with other workflows
- companion project [Rigmarole](https://github.com/mprib/rigmarole) in development to facilitate creation of animated rigs in Blender
## Limitations
### Requires Frame Sync
The workflow currently requires you to provide your own synchronized frames or to provide [a file](project_setup.md#frame_time_historycsv) that specifies the time at which each frame was read so that caliscope can perform the synchronization itself. There are plans to manage this synchronization automatically through audio files, though that has not yet been implemented.
### Currently only using Mediapipe
Google's Mediapipe provides a relatively easy and efficient method for human subject tracking, though for many uses it is limiting. Caliscope has a general Tracker base class that is implemented in a few versions (Pose/Hands/Holistic). This has provided a proof of concept implementation of markerless tracking, though for more robust use the roadmap calls for integration with more powerful tools such as [MMPose](https://github.com/open-mmlab/mmpose) and [DeepLabCut](https://github.com/DeepLabCut/DeepLabCut).
## Reporting Issues and Requesting Features
To report a bug or request a feature, please [open an issue](https://github.com/mprib/caliscope/issues). Please keep in mind that this is an open-source project supported by volunteer effort, so your patience is appreciated.
## General Questions and Conversation
Post any questions in the [Discussions](https://github.com/mprib/caliscope/discussions) section of the repo.
## Acknowledgments
This project was inspired by [FreeMoCap](https://github.com/freemocap/freemocap) (FMC), which is spearheaded by [Jon Matthis, PhD](https://jonmatthis.com/) of the HuMoN Research Lab. The FMC calibration and triangulation system is built upon [Anipose](https://github.com/lambdaloop/anipose), created by Lili Karushchek, PhD. Caliscope was originally envisioned as an alternative calibration tool to Anipose that would allow more granular estimation of intrinsics as well as visual feedback during the calibration process. Several lines of of the original Anipose triangulation code are used in this code base, though otherwise it was written from the ground up. I'm grateful to Dr. Matthis for his time developing FreeMoCap, discussing it with me, pointing out important code considerations, and providing a great deal of information regarding open-source project management.
## License
Caliscope is licensed under the permissive [BSD 2-Clause license](https://opensource.org/license/bsd-2-clause/). The triangulation function was adapted from the [Anipose](https://github.com/lambdaloop/anipose) code base which is also licensed under the BSD-2 Clause. A primary dependency of this project is PySide6 which provides the GUI front end. PySide6 is licensed under the [LGPLv3](https://www.gnu.org/licenses/lgpl-3.0.html). Caliscope does not modify the underlying source code of PySide6 which is available via [PyPI](https://pypi.org/project/PySide6/).
Raw data
{
"_id": null,
"home_page": null,
"name": "caliscope",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.13,>=3.10",
"maintainer_email": null,
"keywords": null,
"author": "Mac Prible",
"author_email": "prible@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/1d/c6/c90b0daa3c0442b2cebe0ab6c689a2ced3cb223d0eafec49399001c23ac1/caliscope-0.5.1.tar.gz",
"platform": null,
"description": "\n<div align=\"center\">\n\n# Caliscope\n\n*Multicamera Calibration + Pose Estimation --> Open Source Motion Capture*\n\n[](https://pypi.org/project/caliscope/)\n[](https://opensource.org/license/bsd-2-clause/)\n[](https://pypi.org/project/caliscope/)\n[](https://github.com/mprib/caliscope/commits)\n[](https://github.com/mprib/caliscope/stargazers)\n\n</div>\n\n\n## About\n`Caliscope` is a GUI-based multicamera calibration package. When the intrinsic (focal length, optical center, and distortion) as well as extrinsic (relative rotation and translation) properties of a set of cameras are known, synchronized frames from those cameras can be used to triangulate landmarks identified across their multiple points of view. With more cameras, this 3D tracking becomes more robust to occlusion and the inevitable errors in 2D landmark tracking and camera property estimates.\n\nWhile OpenCV provides straightforward functions for the estimation of single camera intrinsic properties as well as estimates of the extrinsic properties of two cameras, there is no straightforward way to estimate extrinsic properties for more than two cameras. Performing this requires [bundle adjustment](https://scipy-cookbook.readthedocs.io/items/bundle_adjustment.html), which demands an extensive series of computational steps and intermediate data tracking.\n\nCaliscope automates this more complex calibration function along with providing visual feedback regarding parameter estimates at each stage of processing. Additionally, there are sample implementations of a Tracker class using Google's Mediapipe that demonstrate the capacity to integrate the full calibration results with landmark tracking tools to achieve 3D pose estimation. While Mediapipe pose estimation has limitations regarding accuracy and precision, it demonstrates a data processing pipeline that can easily integrate more powerful tracking tools as they emerge.\n\nThis README provides a general overview and quick guide to install `Calicope` on your system. For a more detailed description of the calibration process and workflow, please see our [docs](https://mprib.github.io/caliscope/).\n\n**For a quick sense of how the calibration process works, check out the [video walk through](https://www.youtube.com/watch?v=voE3IKYtuIQ) of the [sample project](https://mprib.github.io/caliscope/sample_project/).**\n\n---\n\n### Demo Animation\n\nhttps://github.com/mprib/caliscope/assets/31831778/803a4ce8-4012-4da1-87b9-66c5e6b31c59\n\n*`Caliscope` was used to calibrate the cameras used in the above recordings (both intrinsic and extrinsic parameters), run 2D landmark estimation across all the videos using Holistic Mediapipe pose estimation, then triangulate the 3D landmark positions based on the camera system calibration.\nThe animation was created in Blender based on these 3D landmark estimates using a companion project called [Rigmarole](https://github.com/mprib/rigmarole).\nThe concurrent video was recorded with a 7 year old t440p laptop, 4 webcams (~$25 each) and another project called [MultiWebCam](https://github.com/mprib/multiwebcam).*\n\n---\n\n## Quick Start\n\nPlease note that given the size of some core dependencies (OpenCV, Mediapipe, and PySide6 are among them) installation and initial launch can take a while. \n\n### Basic Installation and Launch\n#### Windows\n\n```bash\n# Open Command Prompt and navigate to directory that will hold venv\n# this does not need to be the same as where your project workspace is held\ncd path\\to\\your\\project\n\n# Create a virtual environment named 'env' using Python 3.10\n\"C:\\Path\\To\\Python3.10\\python.exe\" -m venv .venv\n\n# Activate the virtual environment\n.\\.venv\\Scripts\\activate\n\n# Your virtual environment is now active.\n# You can install using pip\npip install caliscope\n\n# Launch from the command line\ncaliscope\n```\n\n#### MacOS/Linux\n```bash\n# Open Command Prompt and navigate to directory that will hold venv\n# this does not need to be the same as where your project workspace is held\ncd path/to/your/project\n\n# Create a virtual environment named 'venv' using Python 3.10\n/path/to/python3.10 -m venv .venv\n\n# Activate the virtual environment\nsource .venv/bin/activate\n\n# Your virtual environment is now active.\n# You can install using pip\npip3 install caliscope\n\n# Launch from the command line\ncaliscope\n```\n\n### MacOS Note: Must adjust some environment variables\n\nThank you to @calcoloergosum for raising an issue regarding a bug in numpy on MacOS that was causing a crash (https://github.com/mprib/caliscope/issues/622).\n\nIt is advised to set the following environment variables on MacOS. If anyone has any additional details related to this or notices that the bug has been fixed, please let me know and I'll take this down.\n\n```\nexport MKL_NUM_THREADS=1\nexport NUMEXPR_NUM_THREADS=1\nexport OMP_NUM_THREADS=1\n```\n\n### Basic Steps\n\n1. Once the GUI launches, navigate to File->New/Open Project and create a folder to hold your project\n - A basic [project structure](https://mprib.github.io/caliscope/project_setup/) will be created here\n2. Define a Charuco calibration board via the Charuco tab and print it out, fixing it to something flat\n3. Record footage for the calibration according to the guidelines for the [intrinsic](https://mprib.github.io/caliscope/intrinsic_calibration/) and [extrinsic](https://mprib.github.io/caliscope/extrinsic_calibration/) calibrations.\n4. Record synchronized motion capture trials\n - A companion project ([multiwebcam](https://github.com/mprib/multiwebcam)) has been set up to facilitate this though is still in early stages\n5. Store video files within the project folder and reload the workspace\n6. Run autocalibration on all cameras within the Cameras tab\n7. Run \"Calibration Capture Volume\" from the Workspace Tab\n8. Set origin within the capture volume (optional but helpful)\n9. Run post-processing on individual recordings to generate 3D trajectory output\n\n## Key Features\n\n### Calibration board creation\n- Easy creation of `png` files for ChArUco calibration boards \n- board definition can be changed across intrinsic and extrinsic calibration allowing greater flexibiltiy\n\n### Intrinsic Camera Calibration\n- Automated calculation of camera intrinsic properties from input video\n - Optical Center\n - Focal Length\n - Lens Distortion\n\n- Visualization of distortion model to ensure reasonableness\n\n### Extrinsic Camera Calibration\n- Automated bundle adjustment to estimate 6 DoF relative position of cameras\n- Visualizer to inspect the estimates from the bundle adjustment\n- Setting of the World Origin within the visualizer to simplify data processing\n\n\n### 3D Tracking\n- Tracker API for integrating alternate tracking methods\n - 3 sample implementations with Google Mediapipe (Hands/Pose/Holistic)\n- Automated application of landmark tracking to synchronized videos\n- Triangulation of 3D landmark position based on calibrated cameras\n- Gap-filling and butterworth filtering to smooth trajectory estimates\n\n### Trajectory Output\n\n- output to `.trc` file format for use in biomechanical modelling\n- output to tidy `.csv` format with well-labelled headers for straightforward integration with other workflows\n- companion project [Rigmarole](https://github.com/mprib/rigmarole) in development to facilitate creation of animated rigs in Blender\n\n## Limitations\n\n### Requires Frame Sync\nThe workflow currently requires you to provide your own synchronized frames or to provide [a file](project_setup.md#frame_time_historycsv) that specifies the time at which each frame was read so that caliscope can perform the synchronization itself. There are plans to manage this synchronization automatically through audio files, though that has not yet been implemented.\n\n### Currently only using Mediapipe\n\nGoogle's Mediapipe provides a relatively easy and efficient method for human subject tracking, though for many uses it is limiting. Caliscope has a general Tracker base class that is implemented in a few versions (Pose/Hands/Holistic). This has provided a proof of concept implementation of markerless tracking, though for more robust use the roadmap calls for integration with more powerful tools such as [MMPose](https://github.com/open-mmlab/mmpose) and [DeepLabCut](https://github.com/DeepLabCut/DeepLabCut).\n\n## Reporting Issues and Requesting Features\n\nTo report a bug or request a feature, please [open an issue](https://github.com/mprib/caliscope/issues). Please keep in mind that this is an open-source project supported by volunteer effort, so your patience is appreciated.\n\n## General Questions and Conversation\n\nPost any questions in the [Discussions](https://github.com/mprib/caliscope/discussions) section of the repo. \n\n\n## Acknowledgments\n\nThis project was inspired by [FreeMoCap](https://github.com/freemocap/freemocap) (FMC), which is spearheaded by [Jon Matthis, PhD](https://jonmatthis.com/) of the HuMoN Research Lab. The FMC calibration and triangulation system is built upon [Anipose](https://github.com/lambdaloop/anipose), created by Lili Karushchek, PhD. Caliscope was originally envisioned as an alternative calibration tool to Anipose that would allow more granular estimation of intrinsics as well as visual feedback during the calibration process. Several lines of of the original Anipose triangulation code are used in this code base, though otherwise it was written from the ground up. I'm grateful to Dr. Matthis for his time developing FreeMoCap, discussing it with me, pointing out important code considerations, and providing a great deal of information regarding open-source project management.\n\n## License\n\nCaliscope is licensed under the permissive [BSD 2-Clause license](https://opensource.org/license/bsd-2-clause/). The triangulation function was adapted from the [Anipose](https://github.com/lambdaloop/anipose) code base which is also licensed under the BSD-2 Clause. A primary dependency of this project is PySide6 which provides the GUI front end. PySide6 is licensed under the [LGPLv3](https://www.gnu.org/licenses/lgpl-3.0.html). Caliscope does not modify the underlying source code of PySide6 which is available via [PyPI](https://pypi.org/project/PySide6/).\n",
"bugtrack_url": null,
"license": "BSD-2-Clause",
"summary": "GUI based multicamera calibration that integrates with 2D landmark tracking to triangulate 3D landmark positions",
"version": "0.5.1",
"project_urls": {
"repository": "https://github.com/mprib/caliscope"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f4fe55bff6a4c2fbbbd986722b6e8193ce9035c040c25d9562107298fa7d8bd4",
"md5": "0dd9de881e4f71e95f4824baf686a692",
"sha256": "f626a5aadbf170e2cc6126fffba454dcfbe1cb0b2b6b1e3b096a3eefe2abb8cb"
},
"downloads": -1,
"filename": "caliscope-0.5.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0dd9de881e4f71e95f4824baf686a692",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.13,>=3.10",
"size": 146387,
"upload_time": "2024-09-25T21:54:50",
"upload_time_iso_8601": "2024-09-25T21:54:50.924215Z",
"url": "https://files.pythonhosted.org/packages/f4/fe/55bff6a4c2fbbbd986722b6e8193ce9035c040c25d9562107298fa7d8bd4/caliscope-0.5.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1dc6c90b0daa3c0442b2cebe0ab6c689a2ced3cb223d0eafec49399001c23ac1",
"md5": "aee0b4b4444de4a8dbb5abdada81065d",
"sha256": "3bd5ab9644043620f1ba976abf3d9dab4d277b1300695898841d12699a319464"
},
"downloads": -1,
"filename": "caliscope-0.5.1.tar.gz",
"has_sig": false,
"md5_digest": "aee0b4b4444de4a8dbb5abdada81065d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.13,>=3.10",
"size": 109962,
"upload_time": "2024-09-25T21:54:52",
"upload_time_iso_8601": "2024-09-25T21:54:52.465514Z",
"url": "https://files.pythonhosted.org/packages/1d/c6/c90b0daa3c0442b2cebe0ab6c689a2ced3cb223d0eafec49399001c23ac1/caliscope-0.5.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-25 21:54:52",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mprib",
"github_project": "caliscope",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "caliscope"
}
