[](https://github.com/perfanalytics/pose2sim/actions/workflows/continuous-integration.yml)
[](https://badge.fury.io/py/Pose2Sim) \
[](https://pepy.tech/project/pose2sim)
[](https://github.com/perfanalytics/pose2sim/stargazers)
[](https://GitHub.com/perfanalytics/pose2sim/forks)
[](https://github.com/perfanalytics/pose2sim/issues)
[](https://GitHub.com/perfanalytics/pose2sim/issues?q=is%3Aissue+is%3Aclosed)
\
[](https://joss.theoj.org/papers/a31cb207a180f7ac9838d049e3a0de26)
[](https://zenodo.org/doi/10.5281/zenodo.10658947)
[](https://opensource.org/licenses/BSD-3-Clause)
# Pose2Sim
##### N.B:. Please set undistort_points and handle_LR_swap to false for now since it currently leads to inaccuracies. I'll try to fix it soon.
> **_News_: Version 0.8:**\
> **Automatic camera synchronization is now supported!**\
> **Other recently added features**: Multi-person analysis, Blender visualization, Marker augmentation, Automatic batch processing.
<!-- Incidentally, right/left limb swapping is now handled, which is useful if few cameras are used;\
and lens distortions are better taken into account.\ -->
> To upgrade, type `pip install pose2sim --upgrade`.
<br>
`Pose2Sim` provides a workflow for 3D markerless kinematics, as an alternative to the more usual marker-based motion capture methods. It aims to provide a free tool to obtain research-grade results from consumer-grade equipment. Any combination of phone, webcam, GoPro, etc. can be used.
Pose2Sim stands for "OpenPose to OpenSim", as it uses OpenPose inputs (2D keypoints coordinates obtained from multiple videos) and leads to an OpenSim result (full-body 3D joint angles). Other 2D pose estimators such as BlazePose (MediaPipe), DeepLabCut, AlphaPose, can now be used as inputs.
If you can only use one single camera and don't mind losing some accuracy, please consider using [Sports2D](https://github.com/davidpagnon/Sports2D).
<img src="Content/Pose2Sim_workflow.jpg" width="760">
<img src='Content/Activities_verylow.gif' title='Other more or less challenging tasks and conditions.' width="760">
> *N.B.:* As always, I am more than happy to welcome contributors (see [How to contribute](#how-to-contribute)).
</br>
**Pose2Sim releases:**
- [x] **v0.1** *(08/2021)*: Published paper
- [x] **v0.2** *(01/2022)*: Published code
- [x] **v0.3** *(01/2023)*: Supported other pose estimation algorithms
- [x] **v0.4** *(07/2023)*: New calibration tool based on scene measurements
- [x] **v0.5** *(12/2023)*: Automatic batch processing
- [x] **v0.6** *(02/2024)*: Marker augmentation, Blender visualizer
- [x] **v0.7** *(03/2024)*: Multi-person analysis
- [x] **v0.8 *(04/2024)*: New synchronization tool**
- [ ] v0.9: Calibration based on keypoint detection, Handling left/right swaps, Correcting lens distortions
- [ ] v0.10: Graphical User Interface
- [ ] v1.0: First accomplished release
</br>
# Contents
1. [Installation and Demonstration](#installation-and-demonstration)
1. [Installation](#installation)
2. [Demonstration Part-1: Triangulate OpenPose outputs](#demonstration-part-1-build-3d-trc-file-on-python)
3. [Demonstration Part-2: Obtain 3D joint angles with OpenSim](#demonstration-part-2-obtain-3d-joint-angles-with-opensim)
4. [Demonstration Part-3 (optional): Visualize your results with Blender](#demonstration-part-3-optional-visualize-your-results-with-blender)
5. [Demonstration Part-4 (optional): Try multi-person analysis](#demonstration-part-4-optional-try-multi-person-analysis)
2. [Use on your own data](#use-on-your-own-data)
1. [Setting your project up](#setting-your-project-up)
1. [Retrieve the folder structure](#retrieve-the-folder-structure)
2. [Single Trial vs. Batch processing](#single-trial-vs-batch-processing)
2. [2D pose estimation](#2d-pose-estimation)
1. [With OpenPose](#with-openpose)
2. [With BlazePose (Mediapipe)](#with-blazepose-mediapipe)
3. [With DeepLabCut](#with-deeplabcut)
4. [With AlphaPose](#with-alphapose)
4. [Camera calibration](#camera-calibration)
1. [Convert from Qualisys, Optitrack, Vicon, OpenCap, EasyMocap, or bioCV](#convert-from-qualisys-optitrack-vicon-opencap-easymocap-or-biocv)
2. [Calculate from scratch](#calculate-from-scratch)
5. [Synchronization, Tracking, Triangulating, Filtering](#synchronization-tracking-triangulating-filtering)
1. [Synchronization](#synchronization)
2. [Associate persons across cameras](#associate-persons-across-cameras)
3. [Triangulating keypoints](#triangulating-keypoints)
4. [Filtering 3D coordinates](#filtering-3d-coordinates)
5. [Marker augmentation](#marker-augmentation)
6. [OpenSim kinematics](#opensim-kinematics)
1. [OpenSim Scaling](#opensim-scaling)
2. [OpenSim Inverse kinematics](#opensim-inverse-kinematics)
3. [Command Line](#command-line)
3. [Utilities](#utilities)
4. [How to cite and how to contribute](#how-to-cite-and-how-to-contribute)
1. [How to cite](#how-to-cite)
2. [How to contribute and to-do list](#how-to-contribute-and-to-do-list)
</br>
# Installation and Demonstration
## Installation
1. **Install OpenPose** (instructions [there](https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/installation/0_index.md)). \
*Windows portable demo is enough.*
2. **Install OpenSim 4.x** ([there](https://simtk.org/frs/index.php?group_id=91)). \
*Tested up to v4.4-beta on Windows. Has to be compiled from source on Linux (see [there](https://simtk-confluence.stanford.edu:8443/display/OpenSim/Linux+Support)).*
3. ***Optional.*** *Install Anaconda or [Miniconda](https://docs.conda.io/en/latest/miniconda.html). \
Open an Anaconda terminal and create a virtual environment with typing:*
<pre><i>conda create -n Pose2Sim python=3.8 -y
conda activate Pose2Sim</i></pre>
3. **Install Pose2Sim**:\
If you don't use Anaconda, type `python -V` in terminal to make sure python>=3.8 is installed.
- OPTION 1: **Quick install:** Open a terminal.
``` cmd
pip install pose2sim
```
- OPTION 2: **Build from source and test the last changes:**
Open a terminal in the directory of your choice and Clone the Pose2Sim repository.
``` cmd
git clone --depth 1 https://github.com/perfanalytics/pose2sim.git
cd pose2sim
pip install .
```
</br>
## Demonstration Part-1: Build 3D TRC file on Python
> _**This demonstration provides an example experiment of a person balancing on a beam, filmed with 4 calibrated cameras processed with OpenPose.**_
Open a terminal, enter `pip show pose2sim`, report package location. \
Copy this path and go to the Single participant Demo folder: `cd <path>\Pose2Sim\S01_Demo_SingleTrial`. \
Type `ipython`, and try the following code:
``` python
from Pose2Sim import Pose2Sim
Pose2Sim.calibration()
Pose2Sim.synchronization()
Pose2Sim.personAssociation()
Pose2Sim.triangulation()
Pose2Sim.filtering()
Pose2Sim.markerAugmentation()
```
3D results are stored as .trc files in each trial folder in the `pose-3d` directory.
*N.B.:* Default parameters have been provided in [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S01_Demo_SingleTrial/Config.toml) but can be edited.\
*N.B.:* *Try the calibration tool by changing `calibration_type` to `calculate` instead of `convert` (more info [there](#calculate-from-scratch)).*
<br/>
## Demonstration Part-2: Obtain 3D joint angles with OpenSim
> _**In the same vein as you would do with marker-based kinematics, start with scaling your model, and then perform inverse kinematics.**_
### Scaling
1. Open OpenSim.
2. Open the provided `Model_Pose2Sim_LSTM.osim` model from `Pose2Sim/OpenSim_Setup`. *(File -> Open Model)*
3. Load the provided `Scaling_Setup_Pose2Sim_LSTM.xml` scaling file from `Pose2Sim/OpenSim_Setup`. *(Tools -> Scale model -> Load)*
4. Run. You should see your skeletal model take the static pose.
5. Save your scaled model in `S01_Demo_SingleTrial/OpenSim/Model_Pose2Sim_S00_P00_LSTM_scaled.osim`. *(File -> Save Model As)*
### Inverse kinematics
1. Load the provided `IK_Setup_Pose2Sim_LSTM.xml` scaling file from `Pose2Sim/OpenSim_Setup`. *(Tools -> Inverse kinematics -> Load)*
2. Run. You should see your skeletal model move in the Vizualizer window.
5. Your IK motion file will be saved in `S00_P00_OpenSim`.
<br/>
<p style="text-align: center;"><img src="Content/OpenSim.JPG" width="380"></p>
</br>
## Demonstration Part-3 (optional): Visualize your results with Blender
> _**Visualize your results and look in detail for potential areas of improvement (and more).**_
### Install the add-on
Follow instructions on the [Pose2Sim_Blender](https://github.com/davidpagnon/Pose2Sim_Blender) add-on page.
### Visualize your results
Just play with the buttons!\
Visualize camera positions, videos, triangulated keypoints, OpenSim skeleton, and more.
**N.B.:** You need to proceed to the full install to import the inverse kinematic results from OpenSim. See instructions [there](https://github.com/davidpagnon/Pose2Sim_Blender?tab=readme-ov-file#full-install).
https://github.com/perfanalytics/pose2sim/assets/54667644/5d7c858f-7e46-40c1-928c-571a5679633a
<br/>
## Demonstration Part-4 (optional): Try multi-person analysis
> _**Another person, hidden all along, will appear when multi-person analysis is activated!**_
Go to the Multi-participant Demo folder: `cd <path>\Pose2Sim\S00_Demo_BatchSession\S00_P01_MultiParticipants\S00_P01_T02_Participants1-2`. \
Type `ipython`, and try the following code:
``` python
from Pose2Sim import Pose2Sim
Pose2Sim.personAssociation()
Pose2Sim.triangulation()
Pose2Sim.filtering()
Pose2Sim.markerAugmentation()
```
One .trc file per participant will be generated and stored in the `pose-3d` directory.\
You can then run OpenSim scaling and inverse kinematics for each resulting .trc file as in [Demonstration Part-2](#demonstration-part-2-obtain-3d-joint-angles-with-opensim).\
You can also visualize your results with Blender as in [Demonstration Part-3](#demonstration-part-3-optional-visualize-your-results-with-blender).
*N.B.:* Set *[project]* `multi_person = true` for each trial that contains multiple persons.\
Set *[triangulation]* `reorder_trc = true` if you need to run OpenSim and to match the generated .trc files with the static trials.\
Make sure that the order of *[markerAugmentation]* `participant_height` and `participant_mass` matches the order of the static trials.
*N.B.:* Note that in the case of our floating ghost participant, marker augmentation may worsen the results. See [Marker augmentation](#marker-augmentation) for instruction on when and when not to use it.
</br></br>
# Use on your own data
> _**Deeper explanations and instructions are given below.**_ \
> N.B.: If a step is not relevant for your use case (synchronization, person association, marker augmentation...), you can skip it.
</br>
## Setting your project up
> _**Get ready for automatic batch processing.**_
### Retrieve the folder structure
1. Open a terminal, enter `pip show pose2sim`, report package location. \
Copy this path and do `cd <path>\pose2sim`.
2. Copy the *single trial* or *batch session* folder wherever you like, and rename it as you wish.
3. The rest of the tutorial will explain to you how to populate the `Calibration` and `videos` folders, edit the [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S01_Demo_SingleTrial/Config.toml) files, and run each Pose2Sim step.
</br>
### Single Trial vs. Batch processing
> _**Copy and edit either the [S01_Demo_SingleTrial](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S01_Demo_SingleTrial) folder or the [S00_Demo_BatchSession](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S00_Demo_BatchSession) one.**_
> - Single trial is more straight-forward to set up for isolated experiments
> - Batch processing allows you to run numerous analysis with different parameters and minimal friction
#### Single trial
The single trial folder should contain a `Config.toml` file, a `calibration` folder, and a `pose` folder, the latter including one subfolder for each camera.
<pre>
SingleTrial \
├── calibration \
├── pose \
└── <i><b>Config.toml</i></b>
</pre>
#### Batch processing
For batch processing, each session directory should follow a `Session -> Participant -> Trial` structure, with a `Config.toml` file in each of the directory levels.
<pre>
Session_s1 \ <i><b>Config.toml</i></b>
├── Calibration\
└── Participant_p1 \ <i><b>Config.toml</i></b>
└── Trial_t1 \ <i><b>Config.toml</i></b>
└── pose \
</pre>
Run Pose2Sim from the `Session` folder if you want to batch process the whole session, from the `Participant` folder if you want to batch process all the trials of a participant, or from the `Trial` folder if you want to process a single trial. There should be one `Calibration` folder per session.
Global parameters are given in the `Config.toml` file of the `Session` folder, and can be altered for specific `Participants` or `Trials` by uncommenting keys and their values in their respective Config.toml files.\
Try uncommenting `[project]` and set `frame_range = [10,300]` for a Participant for example, or uncomment `[filtering.butterworth]` and set `cut_off_frequency = 10` for a Trial.
</br>
## 2D pose estimation
> _**Estimate 2D pose from images with Openpose or another pose estimation solution.**_ \
N.B.: First film a short static pose that will be used for scaling the OpenSim model (A-pose for example), and then film your motions of interest.\
N.B.: Note that the names of your camera folders must follow the same order as in the calibration file, and end with '_json'.
### With OpenPose:
The accuracy and robustness of Pose2Sim have been thoroughly assessed only with OpenPose, and especially with the BODY_25B model. Consequently, we recommend using this 2D pose estimation solution. See [OpenPose repository](https://github.com/CMU-Perceptual-Computing-Lab/openpose) for installation and running.
* Open a command prompt in your **OpenPose** directory. \
Launch OpenPose for each `videos` folder:
``` cmd
bin\OpenPoseDemo.exe --model_pose BODY_25B --video <PATH_TO_TRIAL_DIR>\videos\cam01.mp4 --write_json <PATH_TO_TRIAL_DIR>\pose\pose_cam01_json
```
* The [BODY_25B model](https://github.com/CMU-Perceptual-Computing-Lab/openpose_train/tree/master/experimental_models) has more accurate results than the standard BODY_25 one and has been extensively tested for Pose2Sim. \
You can also use the [BODY_135 model](https://github.com/CMU-Perceptual-Computing-Lab/openpose_train/tree/master/experimental_models), which allows for the evaluation of pronation/supination, wrist flexion, and wrist deviation.\
All other OpenPose models (BODY_25, COCO, MPII) are also supported.\
Make sure you modify the [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S01_Demo_SingleTrial/Config.toml) file accordingly.
* Use one of the `json_display_with_img.py` or `json_display_with_img.py` scripts (see [Utilities](#utilities)) if you want to display 2D pose detections.
**N.B.:** *OpenPose BODY_25B is the default 2D pose estimation model used in Pose2Sim. However, other skeleton models from other 2D pose estimation solutions can be used alternatively.*
<img src="Content/Pose2D.png" width="760">
### With BlazePose (MediaPipe):
[Mediapipe BlazePose](https://google.github.io/mediapipe/solutions/pose.html) is very fast, fully runs under Python, handles upside-down postures and wrist movements (but no subtalar ankle angles). \
However, it is less robust and accurate than OpenPose, and can only detect a single person.
* Use the script `Blazepose_runsave.py` (see [Utilities](#utilities)) to run BlazePose under Python, and store the detected coordinates in OpenPose (json) or DeepLabCut (h5 or csv) format:
``` cmd
python -m Blazepose_runsave -i input_file -dJs
```
Type in `python -m Blazepose_runsave -h` for explanation on parameters.
* Make sure you changed the `pose_model` and the `tracked_keypoint` in the [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S01_Demo_SingleTrial/Config.toml) file.
### With DeepLabCut:
If you need to detect specific points on a human being, an animal, or an object, you can also train your own model with [DeepLabCut](https://github.com/DeepLabCut/DeepLabCut). In this case, Pose2Sim is used as an alternative to [AniPose](https://github.com/lambdaloop/anipose), but it may yield better results since 3D reconstruction takes confidence into account (see [this article](https://doi.org/10.1080/21681163.2023.2292067)).
1. Train your DeepLabCut model and run it on your images or videos (more instruction on their repository)
2. Translate the h5 2D coordinates to json files (with `DLC_to_OpenPose.py` script, see [Utilities](#utilities)):
``` cmd
python -m DLC_to_OpenPose -i input_h5_file
```
3. Edit `pose.CUSTOM` in [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S01_Demo_SingleTrial/Config.toml), and edit the node IDs so that they correspond to the column numbers of the 2D pose file, starting from zero. Make sure you also changed the `pose_model` and the `tracked_keypoint`.\
You can visualize your skeleton's hierarchy by changing pose_model to CUSTOM and writing these lines:
``` python
config_path = r'path_to_Config.toml'
import toml, anytree
config = toml.load(config_path)
pose_model = config.get('pose').get('pose_model')
model = anytree.importer.DictImporter().import_(config.get('pose').get(pose_model))
for pre, _, node in anytree.RenderTree(model):
print(f'{pre}{node.name} id={node.id}')
```
4. Create an OpenSim model if you need inverse kinematics.
### With AlphaPose:
[AlphaPose](https://github.com/MVIG-SJTU/AlphaPose) is one of the main competitors of OpenPose, and its accuracy is comparable. As a top-down approach (unlike OpenPose which is bottom-up), it is faster on single-person detection, but slower on multi-person detection.\
All AlphaPose models are supported (HALPE_26, HALPE_68, HALPE_136, COCO_133, COCO, MPII). For COCO and MPII, AlphaPose must be run with the flag "--format cmu".
* Install and run AlphaPose on your videos (more instruction on their repository)
* Translate the AlphaPose single json file to OpenPose frame-by-frame files (with `AlphaPose_to_OpenPose.py` script, see [Utilities](#utilities)):
``` cmd
python -m AlphaPose_to_OpenPose -i input_alphapose_json_file
```
* Make sure you changed the `pose_model` and the `tracked_keypoint` in the [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S01_Demo_SingleTrial/Config.toml) file.
</br>
## Camera calibration
> _**Calculate camera intrinsic properties and extrinsic locations and positions.\
> Convert a preexisting calibration file, or calculate intrinsic and extrinsic parameters from scratch.**_
Open an Anaconda prompt or a terminal in a `Session`, `Participant`, or `Trial` folder.\
Type `ipython`.
``` python
from Pose2Sim import Pose2Sim
Pose2Sim.calibration()
```
Output:\
<img src="Content/Calib2D.png" width="760">
<img src="Content/CalibFile.png" width="760">
### Convert from Qualisys, Optitrack, Vicon, OpenCap, EasyMocap, or bioCV
If you already have a calibration file, set `calibration_type` type to `convert` in your [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Empty_project/User/Config.toml) file.
- **From [Qualisys](https://www.qualisys.com):**
- Export calibration to `.qca.txt` within QTM.
- Copy it in the `Calibration` Pose2Sim folder.
- set `convert_from` to 'qualisys' in your [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S01_Demo_SingleTrial/Config.toml) file. Change `binning_factor` to 2 if you film in 540p.
- **From [Optitrack](https://optitrack.com/):** Exporting calibration will be available in Motive 3.2. In the meantime:
- Calculate intrinsics with a board (see next section).
- Use their C++ API [to retrieve extrinsic properties](https://docs.optitrack.com/developer-tools/motive-api/motive-api-function-reference#tt_cameraxlocation). Translation can be copied as is in your `Calib.toml` file, but TT_CameraOrientationMatrix first needs to be [converted to a Rodrigues vector](https://docs.opencv.org/3.4/d9/d0c/group__calib3d.html#ga61585db663d9da06b68e70cfbf6a1eac) with OpenCV. See instructions [here](https://github.com/perfanalytics/pose2sim/issues/28).
- Use the `Calib.toml` file as is and do not run Pose2Sim.calibration()
- **From [Vicon](http://www.vicon.com/Software/Nexus):**
- Copy your `.xcp` Vicon calibration file to the Pose2Sim `Calibration` folder.
- set `convert_from` to 'vicon' in your [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S01_Demo_SingleTrial/Config.toml) file. No other setting is needed.
- **From [OpenCap](https://www.opencap.ai/):**
- Copy your `.pickle` OpenCap calibration files to the Pose2Sim `Calibration` folder.
- set `convert_from` to 'opencap' in your [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S01_Demo_SingleTrial/Config.toml) file. No other setting is needed.
- **From [EasyMocap](https://github.com/zju3dv/EasyMocap/):**
- Copy your `intri.yml` and `extri.yml` files to the Pose2Sim `Calibration` folder.
- set `convert_from` to 'easymocap' in your [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S01_Demo_SingleTrial/Config.toml) file. No other setting is needed.
- **From [bioCV](https://github.com/camera-mc-dev/.github/blob/main/profile/mocapPipe.md):**
- Copy your bioCV calibration files (no extension) to the Pose2Sim `Calibration` folder.
- set `convert_from` to 'biocv' in your [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S01_Demo_SingleTrial/Config.toml) file. No other setting is needed.
- **From [AniPose](https://github.com/lambdaloop/anipose) or [FreeMocap](https://github.com/freemocap/freemocap):**
- Copy your `.toml` calibration file to the Pose2Sim `Calibration` folder.
- Calibration can be skipped since Pose2Sim uses the same [Aniposelib](https://anipose.readthedocs.io/en/latest/aniposelibtutorial.html) format.
</br>
### Calculate from scratch
> _**Calculate calibration parameters with a checkerboard, with measurements on the scene, or automatically with detected keypoints.**_\
> Take heart, it is not that complicated once you get the hang of it!
> *N.B.:* Try the calibration tool on the Demo by changing `calibration_type` to `calculate` in [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S00_Demo_BatchSession/Config.toml).\
For the sake of practicality, there are voluntarily few board images for intrinsic calibration, and few points to click for extrinsic calibration. In spite of this, your reprojection error should be under 1-2 cm, which [does not hinder the quality of kinematic results in practice](https://www.mdpi.com/1424-8220/21/19/6530/htm#:~:text=Angle%20results%20were,Table%203).).
- **Calculate intrinsic parameters with a checkerboard:**
> *N.B.:* _Intrinsic parameters:_ camera properties (focal length, optical center, distortion), usually need to be calculated only once in their lifetime. In theory, cameras with same model and same settings will have identical intrinsic parameters.\
> *N.B.:* If you already calculated intrinsic parameters earlier, you can skip this step by setting `overwrite_intrinsics` to false.
- Create a folder for each camera in your `Calibration\intrinsics` folder.
- For each camera, film a checkerboard or a charucoboard. Either the board or the camera can be moved.
- Adjust parameters in the [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S00_Demo_BatchSession/Config.toml) file.
- Make sure that the board:
- is filmed from different angles, covers a large part of the video frame, and is in focus.
- is flat, without reflections, surrounded by a white border, and is not rotationally invariant (Nrows ≠ Ncols, and Nrows odd if Ncols even).
- A common error is to specify the external, instead of the internal number of corners. This may be one less than you would intuitively think.
<img src="Content/Calib_int.png" width="600">
***Intrinsic calibration error should be below 0.5 px.***
- **Calculate extrinsic parameters:**
> *N.B.:* _Extrinsic parameters:_ camera placement in space (position and orientation), need to be calculated every time a camera is moved. Can be calculated from a board, or from points in the scene with known coordinates.\
> *N.B.:* If there is no measurable item in the scene, you can temporarily bring something in (a table, for example), perform calibration, and then remove it before you start capturing motion.
- Create a folder for each camera in your `Calibration\extrinsics` folder.
- Once your cameras are in place, shortly film either a board laid on the floor, or the raw scene\
(only one frame is needed, but do not just take a photo unless you are sure it does not change the image format).
- Adjust parameters in the [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S00_Demo_BatchSession/Config.toml) file.
- Then,
- **With a checkerboard:**\
Make sure that it is seen by all cameras. \
It should preferably be larger than the one used for intrinsics, as results will not be very accurate out of the covered zone.
- **With scene measurements** (more flexible and potentially more accurate if points are spread out):\
Manually measure the 3D coordinates of 10 or more points in the scene (tiles, lines on wall, boxes, treadmill dimensions...). These points should be as spread out as possible. Replace `object_coords_3d` by these coordinates in Config.toml.\
Then you will click on the corresponding image points for each view.
- **With keypoints:**\
For a more automatic calibration, OpenPose keypoints could also be used for calibration.\
**COMING SOON!**
<img src="Content/Calib_ext.png" width="920">
***Extrinsic calibration error should be below 1 cm, but depending on your application, results will still be potentially acceptable up to 2.5 cm.***
</br>
## Synchronizing, Tracking, Triangulating, Filtering
### Synchronization
> _**Cameras need to be synchronized, so that 2D points correspond to the same position across cameras.**_\
***N.B.:** Skip this step if your cameras are natively synchronized.*
Open an Anaconda prompt or a terminal in a `Session`, `Participant`, or `Trial` folder.\
Type `ipython`.
``` python
from Pose2Sim import Pose2Sim
Pose2Sim.synchronization()
```
For each camera, this computes mean vertical speed for the chosen keypoints, and finds the time offset for which their correlation is highest.\
All keypoints can be taken into account, or a subset of them. The user can also specify a time for each camera when only one participant is in the scene, preferably performing a clear vertical motion.
If results are not satisfying, set `reset_sync` to true in `Config.toml` to revert to original state. Then switch to false again and edit the parameters.
*N.B.:* Alternatively, use a flashlight or a clap to synchronize them. GoPro cameras can also be synchronized with a timecode, by GPS (outdoors) or with a remote control (slightly less reliable).
</br>
### Associate persons across cameras
> _**If `multi_person` is set to `false`, the algorithm chooses the person for whom the reprojection error is smallest.\
If `multi_person` is set to `true`, it associates across views the people for whom the distances between epipolar lines are the smallest. People are then associated across frames according to their displacement speed.**_ \
***N.B.:** Skip this step if only one person is in the field of view.*
Open an Anaconda prompt or a terminal in a `Session`, `Participant`, or `Trial` folder.\
Type `ipython`.
``` python
from Pose2Sim import Pose2Sim
Pose2Sim.personAssociation()
```
Check printed output. If results are not satisfying, try and release the constraints in the [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S00_Demo_Session/Config.toml) file.
Output:\
<img src="Content/Track2D.png" width="760">
</br>
### Triangulating keypoints
> _**Triangulate your 2D coordinates in a robust way.**_ \
> The triangulation is weighted by the likelihood of each detected 2D keypoint, provided that they this likelihood is above a threshold.\
If the reprojection error is above another threshold, right and left sides are swapped; if it is still above, cameras are removed until the threshold is met. If more cameras are removed than a predefined number, triangulation is skipped for this point and this frame. In the end, missing values are interpolated.
Open an Anaconda prompt or a terminal in a `Session`, `Participant`, or `Trial` folder.\
Type `ipython`.
``` python
from Pose2Sim import Pose2Sim
Pose2Sim.triangulation()
```
Check printed output, and visualize your trc in OpenSim: `File -> Preview experimental data`.\
If your triangulation is not satisfying, try and release the constraints in the `Config.toml` file.
Output:\
<img src="Content/Triangulate3D.png" width="760">
</br>
### Filtering 3D coordinates
> _**Filter your 3D coordinates.**_\
> Numerous filter types are provided, and can be tuned accordingly.
Open an Anaconda prompt or a terminal in a `Session`, `Participant`, or `Trial` folder.\
Type `ipython`.
``` python
from Pose2Sim import Pose2Sim
Pose2Sim.filtering()
```
Check your filtration with the displayed figures, and visualize your .trc file in OpenSim. If your filtering is not satisfying, try and change the parameters in the `Config.toml` file.
Output:\
<img src="Content/FilterPlot.png" width="760">
<img src="Content/Filter3D.png" width="760">
</br>
### Marker Augmentation
> _**Use the Stanford LSTM model to estimate the position of 47 virtual markers.**_\
_**Note that inverse kinematic results are not necessarily better after marker augmentation.**_ Skip if results are not convincing.
*N.B.:* Marker augmentation tends to give a more stable, but less precise output. In practice, it is mostly beneficial when using less than 4 cameras.
**Make sure that `participant_height` is correct in your `Config.toml` file.** `participant_mass` is mostly optional for IK.\
Only works with models estimating at least the following keypoints (e.g., not COCO):
``` python
["Neck", "RShoulder", "LShoulder", "RHip", "LHip", "RKnee", "LKnee",
"RAnkle", "LAnkle", "RHeel", "LHeel", "RSmallToe", "LSmallToe",
"RBigToe", "LBigToe", "RElbow", "LElbow", "RWrist", "LWrist"]
```
Will not work properly if missing values are not interpolated (i.e., if there are Nan value in the .trc file).
Open an Anaconda prompt or a terminal in a `Session`, `Participant`, or `Trial` folder.\
Type `ipython`.
``` python
from Pose2Sim import Pose2Sim
Pose2Sim.markerAugmentation()
```
</br>
## OpenSim kinematics
> _**Obtain 3D joint angles.**_\
> Your OpenSim .osim scaled model and .mot inverse kinematic results will be found in the OpenSim folder of your `Participant` directory.
### OpenSim Scaling
1. Use the previous steps to capture a static pose, typically an A-pose or a T-pose.
2. Open OpenSim.
3. Open the provided `Model_Pose2Sim_LSTM.osim` model from `Pose2Sim/OpenSim_Setup`. *(File -> Open Model)*
4. Load the provided `Scaling_Setup_Pose2Sim_LSTM.xml` scaling file. *(Tools -> Scale model -> Load)*
5. Replace the example static .trc file with your own data.
6. Run
7. Save the new scaled OpenSim model.
### OpenSim Inverse kinematics
1. Use Pose2Sim to generate 3D trajectories.
2. Open OpenSim.
3. Load the provided `IK_Setup_Pose2Sim_LSTM.xml` scaling file from `Pose2Sim/OpenSim_Setup`. *(Tools -> Inverse kinematics -> Load)*
4. Replace the example .trc file with your own data, and specify the path to your angle kinematics output file.
5. Run.
<img src="Content/OpenSim.JPG" width="380">
</br>
### Command line
Alternatively, you can use command-line tools:
- Open an Anaconda terminal in your OpenSim/bin directory, typically `C:\OpenSim <Version>\bin`.\
You'll need to adjust the `time_range`, `output_motion_file`, and enter the full paths to the input and output `.osim`, `.trc`, and `.mot` files in your setup file.
``` cmd
opensim-cmd run-tool <PATH TO YOUR SCALING OR IK SETUP FILE>.xml
```
- You can also run OpenSim directly in Python:
``` python
import subprocess
subprocess.call(["opensim-cmd", "run-tool", r"<PATH TO YOUR SCALING OR IK SETUP FILE>.xml"])
```
- Or take advantage of the full the OpenSim Python API. See [there](https://simtk-confluence.stanford.edu:8443/display/OpenSim/Scripting+in+Python) for installation instructions (conda install may take a while).\
Make sure to replace `py38np120` with your Python version (3.8 in this case) and with your numpy version (1.20 here).
``` cmd
conda install -c opensim-org opensim-moco=4.4=py38np120 -y
```
If you run into a DLL error while importing opensim, open the file `<Pose2Sim-env>\Lib\opensim\__init__.py` and replace `conda`by `conda-meta` line 4. `<Pose2Sim-env>` location can be found with `conda env list`.\
Then run:
`ipython`
``` python
import opensim
opensim.ScaleTool("<PATH TO YOUR SCALING OR IK SETUP FILE>.xml").run()
opensim.InverseKinematicsTool("<PATH TO YOUR SCALING OR IK SETUP FILE>.xml").run()
```
You can also run other API commands. See [there](https://simtk-confluence.stanford.edu:8443/display/OpenSim/Common+Scripting+Commands#CommonScriptingCommands-UsingtheTools) for more instructions on how to use it.
</br>
# Utilities
A list of standalone tools (see [Utilities](https://github.com/perfanalytics/pose2sim/tree/main/Pose2Sim/Utilities)), which can be either run as scripts, or imported as functions. Check usage in the docstring of each Python file. The figure below shows how some of these tools can be used to further extend Pose2Sim usage.
<details>
<summary><b>Converting pose files</b> (CLICK TO SHOW)</summary>
<pre>
[Blazepose_runsave.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/Blazepose_runsave.py)
Runs BlazePose on a video, and saves coordinates in OpenPose (json) or DeepLabCut (h5 or csv) format.
[DLC_to_OpenPose.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/DLC_to_OpenPose.py)
Converts a DeepLabCut (h5) 2D pose estimation file into OpenPose (json) files.
[AlphaPose_to_OpenPose.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/AlphaPose_to_OpenPose.py)
Converts AlphaPose single json file to OpenPose frame-by-frame files.
</pre>
</details>
<details>
<summary><b>Converting calibration files</b> (CLICK TO SHOW)</summary>
<pre>
[calib_toml_to_easymocap.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/calib_toml_to_easymocap.py)
Converts an OpenCV .toml calibration file to EasyMocap intrinsic and extrinsic .yml calibration files.
[calib_easymocap_to_toml.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/calib_easymocap_to_toml.py)
Converts EasyMocap intrinsic and extrinsic .yml calibration files to an OpenCV .toml calibration file.
[calib_from_checkerboard.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/calib_from_checkerboard.py)
Calibrates cameras with images or a video of a checkerboard, saves calibration in a Pose2Sim .toml calibration file.
You should probably use Pose2Sim.calibration() instead, which is much easier and better.
[calib_qca_to_toml.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/calib_qca_to_toml.py)
Converts a Qualisys .qca.txt calibration file to the Pose2Sim .toml calibration file (similar to what is used in [AniPose](https://anipose.readthedocs.io/en/latest/)).
[calib_toml_to_qca.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/calib_toml_to_qca.py)
Converts a Pose2Sim .toml calibration file (e.g., from a checkerboard) to a Qualisys .qca.txt calibration file.
[calib_toml_to_opencap.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/calib_toml_to_opencap.py)
Converts an OpenCV .toml calibration file to OpenCap .pickle calibration files.
[calib_toml_to_opencap.py]( )
To convert OpenCap calibration tiles to a .toml file, please use Pose2Sim.calibration() and set convert_from = 'opencap' in Config.toml.
</pre>
</details>
<details>
<summary><b>Plotting tools</b> (CLICK TO SHOW)</summary>
<pre>
[json_display_with_img.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/json_display_with_img.py)
Overlays 2D detected json coordinates on original raw images. High confidence keypoints are green, low confidence ones are red.
[json_display_without_img.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/json_display_without_img.py)
Plots an animation of 2D detected json coordinates.
[trc_plot.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/trc_plot.py)
Displays X, Y, Z coordinates of each 3D keypoint of a TRC file in a different matplotlib tab.
</pre>
</details>
<details>
<summary><b>Other trc tools</b> (CLICK TO SHOW)</summary>
<pre>
[trc_from_easymocap.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/trc_from_easymocap.py)
Convert EasyMocap results keypoints3d json files to .trc.
[c3d_to_trc.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/c3d_to_trc.py)
Converts 3D point data of a .c3d file to a .trc file compatible with OpenSim. No analog data (force plates, emg) nor computed data (angles, powers, etc.) are retrieved.
[trc_desample.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/trc_desample.py)
Undersamples a trc file.
[trc_Zup_to_Yup.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/trc_Zup_to_Yup.py)
Changes Z-up system coordinates to Y-up system coordinates.
[trc_filter.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/trc_filter.py)
Filters trc files. Available filters: Butterworth, Kalman, Butterworth on speed, Gaussian, LOESS, Median.
[trc_gaitevents.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/trc_gaitevents.py)
Detects gait events from point coordinates according to [Zeni et al. (2008)](https://www.sciencedirect.com/science/article/abs/pii/S0966636207001804?via%3Dihub).
[trc_combine.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/trc_combine.py)
Combine two trc files, for example a triangulated DeepLabCut trc file and a triangulated OpenPose trc file.
[trc_from_mot_osim.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/trc_from_mot_osim.py)
Build a trc file from a .mot motion file and a .osim model file.
[bodykin_from_mot_osim.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/bodykin_from_mot_osim.py)
Converts a mot file to a .csv file with rotation and orientation of all segments.
[reproj_from_trc_calib.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/reproj_from_trc_calib.py)
Reprojects 3D coordinates of a trc file to the image planes defined by a calibration file. Output in OpenPose or DeepLabCut format.
</pre>
</details>
<img src="Content/Pose2Sim_workflow_utilities.jpg" width="760">
</br>
# How to cite and how to contribute
### How to cite
If you use this code or data, please cite [Pagnon et al., 2022b](https://doi.org/10.21105/joss.04362), [Pagnon et al., 2022a](https://www.mdpi.com/1424-8220/22/7/2712), or [Pagnon et al., 2021](https://www.mdpi.com/1424-8220/21/19/6530).
@Article{Pagnon_2022_JOSS,
AUTHOR = {Pagnon, David and Domalain, Mathieu and Reveret, Lionel},
TITLE = {Pose2Sim: An open-source Python package for multiview markerless kinematics},
JOURNAL = {Journal of Open Source Software},
YEAR = {2022},
DOI = {10.21105/joss.04362},
URL = {https://joss.theoj.org/papers/10.21105/joss.04362}
}
@Article{Pagnon_2022_Accuracy,
AUTHOR = {Pagnon, David and Domalain, Mathieu and Reveret, Lionel},
TITLE = {Pose2Sim: An End-to-End Workflow for 3D Markerless Sports Kinematics—Part 2: Accuracy},
JOURNAL = {Sensors},
YEAR = {2022},
DOI = {10.3390/s22072712},
URL = {https://www.mdpi.com/1424-8220/22/7/2712}
}
@Article{Pagnon_2021_Robustness,
AUTHOR = {Pagnon, David and Domalain, Mathieu and Reveret, Lionel},
TITLE = {Pose2Sim: An End-to-End Workflow for 3D Markerless Sports Kinematics—Part 1: Robustness},
JOURNAL = {Sensors},
YEAR = {2021},
DOI = {10.3390/s21196530},
URL = {https://www.mdpi.com/1424-8220/21/19/6530}
}
</br>
### How to contribute and to-do list
I would happily welcome any proposal for new features, code improvement, and more!\
If you want to contribute to Pose2Sim, please see [this issue](https://github.com/perfanalytics/pose2sim/issues/40).\
You will be proposed a to-do list, but please feel absolutely free to propose your own ideas and improvements.
</br>
**Main to-do list**
- Graphical User Interface
- Synchronization
- Self-calibration based on keypoint detection
</br>
<details>
<summary><b>Detailed GOT-DONE and TO-DO list</b> (CLICK TO SHOW)</summary>
<pre>
✔ **Pose:** Support OpenPose [body_25b](https://github.com/CMU-Perceptual-Computing-Lab/openpose_train/tree/master/experimental_models#body_25b-model---option-2-recommended) for more accuracy, [body_135](https://github.com/CMU-Perceptual-Computing-Lab/openpose_train/tree/master/experimental_models#single-network-whole-body-pose-estimation-model) for pronation/supination.
✔ **Pose:** Support [BlazePose](https://developers.google.com/mediapipe/solutions/vision/pose_landmarker) for faster inference (on mobile device).
✔ **Pose:** Support [DeepLabCut](http://www.mackenziemathislab.org/deeplabcut) for training on custom datasets.
✔ **Pose:** Support [AlphaPose](https://github.com/MVIG-SJTU/AlphaPose) as an alternative to OpenPose.
✔ **Pose:** Define custom model in config.toml rather than in skeletons.py.
▢ **Pose:** Support [MMPose](https://github.com/open-mmlab/mmpose), [SLEAP](https://sleap.ai/), etc.
▢ **Pose:** Directly reading from DeepLabCut .csv or .h5 files instead of converting to .json (triangulation, person association, calibration, synchronization...)
▢ **Pose:** GUI help for DeepLabCut model creation.
✔ **Calibration:** Convert [Qualisys](https://www.qualisys.com) .qca.txt calibration file.
✔ **Calibration:** Convert [Optitrack](https://optitrack.com/) extrinsic calibration file.
✔ **Calibration:** Convert [Vicon](http://www.vicon.com/Software/Nexus) .xcp calibration file.
✔ **Calibration:** Convert [OpenCap](https://www.opencap.ai/) .pickle calibration files.
✔ **Calibration:** Convert [EasyMocap](https://github.com/zju3dv/EasyMocap/) .yml calibration files.
✔ **Calibration:** Convert [bioCV](https://github.com/camera-mc-dev/.github/blob/main/profile/mocapPipe.md) calibration files.
✔ **Calibration:** Easier and clearer calibration procedure: separate intrinsic and extrinsic parameter calculation, edit corner detection if some are wrongly detected (or not visible).
✔ **Calibration:** Possibility to evaluate extrinsic parameters from cues on scene.
▢ **Calibration:** Once object points have been detected or clicked once, track them for live calibration of moving cameras. Propose to click again when they are lost.
▢ **Calibration:** Calibrate cameras by pairs and compute average extrinsic calibration with [aniposelib](https://github.com/lambdaloop/aniposelib/blob/d03b485c4e178d7cff076e9fe1ac36837db49158/aniposelib/utils.py#L167).
▢ **Calibration:** Fine-tune calibration with bundle adjustment.
▢ **Calibration:** Support ChArUco board detection (see [there](https://mecaruco2.readthedocs.io/en/latest/notebooks_rst/Aruco/sandbox/ludovic/aruco_calibration_rotation.html)).
▢ **Calibration:** Calculate calibration with points rather than board. (1) SBA calibration with wand (cf [Argus](https://argus.web.unc.edu), see converter [here](https://github.com/backyardbiomech/DLCconverterDLT/blob/master/DLTcameraPosition.py)). Set world reference frame in the end.
▢ **Calibration:** Alternatively, self-calibrate with [OpenPose keypoints](https://ietresearch.onlinelibrary.wiley.com/doi/full/10.1049/cvi2.12130). Set world reference frame in the end.
▢ **Calibration:** Convert [fSpy calibration](https://fspy.io/) based on vanishing point.
▢ **Synchronization:** Synchronize cameras on 2D keypoint speeds. Cf [this draft script](https://github.com/perfanalytics/pose2sim/blob/draft/Pose2Sim/Utilities/synchronize_cams.py).
✔ **Person Association:** Automatically choose the main person to triangulate.
✔ **Person Association:** Multiple persons association. 1. Triangulate all the persons whose reprojection error is below a certain threshold (instead of only the one with minimum error), and then track in time with speed cf [Slembrouck 2020](https://link.springer.com/chapter/10.1007/978-3-030-40605-9_15)? or 2. Based on affinity matrices [Dong 2021](https://arxiv.org/pdf/1901.04111.pdf)? or 3. Based on occupancy maps [Yildiz 2012](https://link.springer.com/chapter/10.1007/978-3-642-35749-7_10)? or 4. With a neural network [Huang 2023](https://arxiv.org/pdf/2304.09471.pdf)?
✔ **Triangulation:** Triangulation weighted with confidence.
✔ **Triangulation:** Set a likelihood threshold below which a camera should not be used, a reprojection error threshold, and a minimum number of remaining cameras below which triangulation is skipped for this frame.
✔ **Triangulation:** Interpolate missing frames (cubic, bezier, linear, slinear, quadratic)
✔ **Triangulation:** Show mean reprojection error in px and in mm for each keypoint.
✔ **Triangulation:** Show how many cameras on average had to be excluded for each keypoint.
✔ **Triangulation:** Evaluate which cameras were the least reliable.
✔ **Triangulation:** Show which frames had to be interpolated for each keypoint.
✔ **Triangulation:** Solve limb swapping (although not really an issue with Body_25b). Try triangulating with opposite side if reprojection error too large. Alternatively, ignore right and left sides, use RANSAC or SDS triangulation, and then choose right or left by majority voting. More confidence can be given to cameras whose plane is the most coplanar to the right/left line.
✔ **Triangulation:** [Undistort](https://docs.opencv.org/3.4/da/d54/group__imgproc__transform.html#ga887960ea1bde84784e7f1710a922b93c) 2D points before triangulating (and [distort](https://github.com/lambdaloop/aniposelib/blob/d03b485c4e178d7cff076e9fe1ac36837db49158/aniposelib/cameras.py#L301) them before computing reprojection error).
✔ **Triangulation:** Offer the possibility to augment the triangulated data with [the OpenCap LSTM](https://github.com/stanfordnmbl/opencap-core/blob/main/utilsAugmenter.py). Create "BODY_25_AUGMENTED" model, Scaling_setup, IK_Setup.
✔ **Triangulation:** Multiple person kinematics (output multiple .trc coordinates files). Triangulate all persons with reprojection error above threshold, and identify them by minimizing their displacement across frames.
▢ **Triangulation:** Pre-compile weighted_triangulation and reprojection with @jit(nopython=True, parallel=True) for faster execution.
▢ **Triangulation:** Offer the possibility of triangulating with Sparse Bundle Adjustment (SBA), Extended Kalman Filter (EKF), Full Trajectory Estimation (FTE) (see [AcinoSet](https://github.com/African-Robotics-Unit/AcinoSet)).
▢ **Triangulation:** Implement normalized DLT and RANSAC triangulation, Outlier rejection (sliding z-score?), as well as a [triangulation refinement step](https://doi.org/10.1109/TMM.2022.3171102).
▢ **Triangulation:** Track hands and face (won't be taken into account in OpenSim at this stage).
✔ **Filtering:** Available filtering methods: Butterworth, Butterworth on speed, Gaussian, Median, LOESS (polynomial smoothing).
✔ **Filtering:** Implement Kalman filter and Kalman smoother.
▢ **Filtering:** Implement [smoothNet](https://github.com/perfanalytics/pose2sim/issues/29)
✔ **OpenSim:** Integrate better spine from [lifting fullbody model](https://pubmed.ncbi.nlm.nih.gov/30714401) to the [gait full-body model](https://nmbl.stanford.edu/wp-content/uploads/07505900.pdf), more accurate for the knee.
✔ **OpenSim:** Optimize model marker positions as compared to ground-truth marker-based positions.
✔ **OpenSim:** Add scaling and inverse kinematics setup files.
✔ **OpenSim:** Add full model with contact spheres ([SmoothSphereHalfSpaceForce](https://simtk.org/api_docs/opensim/api_docs/classOpenSim_1_1SmoothSphereHalfSpaceForce.html#details)) and full-body muscles ([DeGrooteFregly2016Muscle](https://simtk.org/api_docs/opensim/api_docs/classOpenSim_1_1DeGrooteFregly2016Muscle.html#details)), for [Moco](https://opensim-org.github.io/opensim-moco-site/) for example.
✔ **OpenSim:** Add model with [ISB shoulder](https://github.com/stanfordnmbl/opencap-core/blob/main/opensimPipeline/Models/LaiUhlrich2022_shoulder.osim).
▢ **OpenSim:** Implement optimal fixed-interval Kalman smoothing for inverse kinematics ([this OpenSim fork](https://github.com/antoinefalisse/opensim-core/blob/kalman_smoother/OpenSim/Tools/InverseKinematicsKSTool.cpp)), or [Biorbd](https://github.com/pyomeca/biorbd/blob/f776fe02e1472aebe94a5c89f0309360b52e2cbc/src/RigidBody/KalmanReconsMarkers.cpp))
✔ **GUI:** Blender add-on (cf [MPP2SOS](https://blendermarket.com/products/mocap-mpp2soss)), or webapp (e.g., with [Napari](https://napari.org/stable). See my draft project [Maya-Mocap](https://github.com/davidpagnon/Maya-Mocap) and [BlendOsim](https://github.com/JonathanCamargo/BlendOsim).
▢ **GUI:** 3D plot of cameras and of triangulated keypoints.
▢ **GUI:** Demo on Google Colab (see [Sports2D](https://bit.ly/Sports2D_Colab) for OpenPose and Python package installation on Google Drive).
✔ **Demo:** Provide Demo data for users to test the code.
▢ **Demo:** Add videos for users to experiment with other pose detection frameworks
▢ **Demo:** Time shift videos and json to demonstrate synchronization
▢ **Demo:** Add another virtual person to demonstrate personAssociation
▢ **Tutorials:** Make video tutorials.
▢ **Doc:** Use [Sphinx](https://www.sphinx-doc.org/en/master), [MkDocs](https://www.mkdocs.org), or (maybe better), [github.io](https://docs.github.com/fr/pages/quickstart) for clearer documentation.
✔ **Pip package**
✔ **Batch processing**
✔ **Catch errors**
▢ **Conda package**
▢ **Docker image**
▢ Run pose estimation and OpenSim from within Pose2Sim
▢ Real-time: Run Pose estimation, Person association, Triangulation, Kalman filter, IK frame by frame (instead of running each step for all frames)
▢ Config parameter for non batch peocessing
▢ **Run from command line via click or typer**
▢ **Utilities**: Export other data from c3d files into .mot or .sto files (angles, powers, forces, moments, GRF, EMG...)
▢ **Utilities**: Create trc_to_c3d.py script
✔ **Bug:** calibration.py. FFMPEG error message when calibration files are images. See [there](https://github.com/perfanalytics/pose2sim/issues/33#:~:text=In%20order%20to%20check,filter%20this%20message%20yet.).
▢ **Bug:** common.py, class plotWindow(). Python crashes after a few runs of `Pose2Sim.filtering()` when `display_figures=true`. See [there](https://github.com/superjax/plotWindow/issues/7).
</pre>
</details>
</br>
**Acknowledgements:**
- Supervised my PhD: [@lreveret](https://github.com/lreveret) (INRIA, Université Grenoble Alpes), and [@mdomalai](https://github.com/mdomalai) (Université de Poitiers).
- Provided the Demo data: [@aaiaueil](https://github.com/aaiaueil) from Université Gustave Eiffel.
- Tested the code and provided feedback: [@simonozan](https://github.com/simonozan), [@daeyongyang](https://github.com/daeyongyang), [@ANaaim](https://github.com/ANaaim), [@rlagnsals](https://github.com/rlagnsals)
- Submitted various accepted pull requests: [@ANaaim](https://github.com/ANaaim), [@rlagnsals](https://github.com/rlagnsals)
- Provided a code snippet for Optitrack calibration: [@claraaudap](https://github.com/claraaudap) (Université Bretagne Sud).
- Issued MPP2SOS, a (non-free) Blender extension based on Pose2Sim: [@carlosedubarreto](https://github.com/carlosedubarreto)
BSD 3-Clause License
Copyright (c) 2022, perfanalytics
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Raw data
{
"_id": null,
"home_page": "https://github.com/perfanalytics/pose2sim",
"name": "pose2sim",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "markerless, kinematics, OpenPose, OpenSim, 3D human pose, biomechanics",
"author": "David Pagnon",
"author_email": "contact@david-pagnon.com",
"download_url": "https://files.pythonhosted.org/packages/b1/dc/1148609340b0e0c51a0325eb3b32287b1c92420aeaa0cc382e84898ee97f/pose2sim-0.8.1.tar.gz",
"platform": "any",
"description": "[](https://github.com/perfanalytics/pose2sim/actions/workflows/continuous-integration.yml)\n[](https://badge.fury.io/py/Pose2Sim) \\\n[](https://pepy.tech/project/pose2sim)\n[](https://github.com/perfanalytics/pose2sim/stargazers)\n[](https://GitHub.com/perfanalytics/pose2sim/forks)\n[](https://github.com/perfanalytics/pose2sim/issues)\n[](https://GitHub.com/perfanalytics/pose2sim/issues?q=is%3Aissue+is%3Aclosed)\n\\\n[](https://joss.theoj.org/papers/a31cb207a180f7ac9838d049e3a0de26)\n[](https://zenodo.org/doi/10.5281/zenodo.10658947)\n[](https://opensource.org/licenses/BSD-3-Clause)\n\n\n# Pose2Sim\n\n\n##### N.B:. Please set undistort_points and handle_LR_swap to false for now since it currently leads to inaccuracies. I'll try to fix it soon.\n\n> **_News_: Version 0.8:**\\\n> **Automatic camera synchronization is now supported!**\\\n> **Other recently added features**: Multi-person analysis, Blender visualization, Marker augmentation, Automatic batch processing.\n<!-- Incidentally, right/left limb swapping is now handled, which is useful if few cameras are used;\\\nand lens distortions are better taken into account.\\ -->\n> To upgrade, type `pip install pose2sim --upgrade`.\n\n<br>\n\n`Pose2Sim` provides a workflow for 3D markerless kinematics, as an alternative to the more usual marker-based motion capture methods. It aims to provide a free tool to obtain research-grade results from consumer-grade equipment. Any combination of phone, webcam, GoPro, etc. can be used.\n\nPose2Sim stands for \"OpenPose to OpenSim\", as it uses OpenPose inputs (2D keypoints coordinates obtained from multiple videos) and leads to an OpenSim result (full-body 3D joint angles). Other 2D pose estimators such as BlazePose (MediaPipe), DeepLabCut, AlphaPose, can now be used as inputs.\n\nIf you can only use one single camera and don't mind losing some accuracy, please consider using [Sports2D](https://github.com/davidpagnon/Sports2D).\n\n\n<img src=\"Content/Pose2Sim_workflow.jpg\" width=\"760\">\n\n<img src='Content/Activities_verylow.gif' title='Other more or less challenging tasks and conditions.' width=\"760\">\n\n> *N.B.:* As always, I am more than happy to welcome contributors (see [How to contribute](#how-to-contribute)).\n</br>\n\n**Pose2Sim releases:**\n- [x] **v0.1** *(08/2021)*: Published paper\n- [x] **v0.2** *(01/2022)*: Published code\n- [x] **v0.3** *(01/2023)*: Supported other pose estimation algorithms\n- [x] **v0.4** *(07/2023)*: New calibration tool based on scene measurements\n- [x] **v0.5** *(12/2023)*: Automatic batch processing\n- [x] **v0.6** *(02/2024)*: Marker augmentation, Blender visualizer\n- [x] **v0.7** *(03/2024)*: Multi-person analysis\n- [x] **v0.8 *(04/2024)*: New synchronization tool**\n- [ ] v0.9: Calibration based on keypoint detection, Handling left/right swaps, Correcting lens distortions\n- [ ] v0.10: Graphical User Interface\n- [ ] v1.0: First accomplished release\n\n</br>\n\n# Contents\n1. [Installation and Demonstration](#installation-and-demonstration)\n 1. [Installation](#installation)\n 2. [Demonstration Part-1: Triangulate OpenPose outputs](#demonstration-part-1-build-3d-trc-file-on-python)\n 3. [Demonstration Part-2: Obtain 3D joint angles with OpenSim](#demonstration-part-2-obtain-3d-joint-angles-with-opensim)\n 4. [Demonstration Part-3 (optional): Visualize your results with Blender](#demonstration-part-3-optional-visualize-your-results-with-blender)\n 5. [Demonstration Part-4 (optional): Try multi-person analysis](#demonstration-part-4-optional-try-multi-person-analysis)\n2. [Use on your own data](#use-on-your-own-data)\n 1. [Setting your project up](#setting-your-project-up)\n 1. [Retrieve the folder structure](#retrieve-the-folder-structure)\n 2. [Single Trial vs. Batch processing](#single-trial-vs-batch-processing)\n 2. [2D pose estimation](#2d-pose-estimation)\n 1. [With OpenPose](#with-openpose)\n 2. [With BlazePose (Mediapipe)](#with-blazepose-mediapipe)\n 3. [With DeepLabCut](#with-deeplabcut)\n 4. [With AlphaPose](#with-alphapose)\n 4. [Camera calibration](#camera-calibration)\n 1. [Convert from Qualisys, Optitrack, Vicon, OpenCap, EasyMocap, or bioCV](#convert-from-qualisys-optitrack-vicon-opencap-easymocap-or-biocv)\n 2. [Calculate from scratch](#calculate-from-scratch)\n 5. [Synchronization, Tracking, Triangulating, Filtering](#synchronization-tracking-triangulating-filtering)\n 1. [Synchronization](#synchronization)\n 2. [Associate persons across cameras](#associate-persons-across-cameras)\n 3. [Triangulating keypoints](#triangulating-keypoints)\n 4. [Filtering 3D coordinates](#filtering-3d-coordinates)\n 5. [Marker augmentation](#marker-augmentation)\n 6. [OpenSim kinematics](#opensim-kinematics)\n 1. [OpenSim Scaling](#opensim-scaling)\n 2. [OpenSim Inverse kinematics](#opensim-inverse-kinematics)\n 3. [Command Line](#command-line)\n3. [Utilities](#utilities)\n4. [How to cite and how to contribute](#how-to-cite-and-how-to-contribute)\n 1. [How to cite](#how-to-cite)\n 2. [How to contribute and to-do list](#how-to-contribute-and-to-do-list)\n\n</br>\n\n# Installation and Demonstration\n\n## Installation\n1. **Install OpenPose** (instructions [there](https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/installation/0_index.md)). \\\n*Windows portable demo is enough.*\n2. **Install OpenSim 4.x** ([there](https://simtk.org/frs/index.php?group_id=91)). \\\n*Tested up to v4.4-beta on Windows. Has to be compiled from source on Linux (see [there](https://simtk-confluence.stanford.edu:8443/display/OpenSim/Linux+Support)).*\n3. ***Optional.*** *Install Anaconda or [Miniconda](https://docs.conda.io/en/latest/miniconda.html). \\\n Open an Anaconda terminal and create a virtual environment with typing:*\n <pre><i>conda create -n Pose2Sim python=3.8 -y \n conda activate Pose2Sim</i></pre>\n \n3. **Install Pose2Sim**:\\\nIf you don't use Anaconda, type `python -V` in terminal to make sure python>=3.8 is installed.\n - OPTION 1: **Quick install:** Open a terminal. \n ``` cmd\n pip install pose2sim\n ```\n \n - OPTION 2: **Build from source and test the last changes:**\n Open a terminal in the directory of your choice and Clone the Pose2Sim repository.\n ``` cmd\n git clone --depth 1 https://github.com/perfanalytics/pose2sim.git\n cd pose2sim\n pip install .\n ```\n\n</br>\n\n## Demonstration Part-1: Build 3D TRC file on Python \n> _**This demonstration provides an example experiment of a person balancing on a beam, filmed with 4 calibrated cameras processed with OpenPose.**_ \n\nOpen a terminal, enter `pip show pose2sim`, report package location. \\\nCopy this path and go to the Single participant Demo folder: `cd <path>\\Pose2Sim\\S01_Demo_SingleTrial`. \\\nType `ipython`, and try the following code:\n``` python\nfrom Pose2Sim import Pose2Sim\nPose2Sim.calibration()\nPose2Sim.synchronization()\nPose2Sim.personAssociation()\nPose2Sim.triangulation()\nPose2Sim.filtering()\nPose2Sim.markerAugmentation()\n```\n3D results are stored as .trc files in each trial folder in the `pose-3d` directory.\n\n*N.B.:* Default parameters have been provided in [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S01_Demo_SingleTrial/Config.toml) but can be edited.\\\n*N.B.:* *Try the calibration tool by changing `calibration_type` to `calculate` instead of `convert` (more info [there](#calculate-from-scratch)).*\n\n<br/>\n\n## Demonstration Part-2: Obtain 3D joint angles with OpenSim \n> _**In the same vein as you would do with marker-based kinematics, start with scaling your model, and then perform inverse kinematics.**_ \n\n### Scaling\n1. Open OpenSim.\n2. Open the provided `Model_Pose2Sim_LSTM.osim` model from `Pose2Sim/OpenSim_Setup`. *(File -> Open Model)*\n3. Load the provided `Scaling_Setup_Pose2Sim_LSTM.xml` scaling file from `Pose2Sim/OpenSim_Setup`. *(Tools -> Scale model -> Load)*\n4. Run. You should see your skeletal model take the static pose.\n5. Save your scaled model in `S01_Demo_SingleTrial/OpenSim/Model_Pose2Sim_S00_P00_LSTM_scaled.osim`. *(File -> Save Model As)*\n\n### Inverse kinematics\n1. Load the provided `IK_Setup_Pose2Sim_LSTM.xml` scaling file from `Pose2Sim/OpenSim_Setup`. *(Tools -> Inverse kinematics -> Load)*\n2. Run. You should see your skeletal model move in the Vizualizer window.\n5. Your IK motion file will be saved in `S00_P00_OpenSim`.\n<br/>\n\n<p style=\"text-align: center;\"><img src=\"Content/OpenSim.JPG\" width=\"380\"></p>\n\n</br>\n\n## Demonstration Part-3 (optional): Visualize your results with Blender\n> _**Visualize your results and look in detail for potential areas of improvement (and more).**_ \n\n### Install the add-on\nFollow instructions on the [Pose2Sim_Blender](https://github.com/davidpagnon/Pose2Sim_Blender) add-on page.\n\n### Visualize your results\nJust play with the buttons!\\\nVisualize camera positions, videos, triangulated keypoints, OpenSim skeleton, and more.\n\n**N.B.:** You need to proceed to the full install to import the inverse kinematic results from OpenSim. See instructions [there](https://github.com/davidpagnon/Pose2Sim_Blender?tab=readme-ov-file#full-install).\n\nhttps://github.com/perfanalytics/pose2sim/assets/54667644/5d7c858f-7e46-40c1-928c-571a5679633a\n\n<br/>\n\n## Demonstration Part-4 (optional): Try multi-person analysis\n> _**Another person, hidden all along, will appear when multi-person analysis is activated!**_\n\nGo to the Multi-participant Demo folder: `cd <path>\\Pose2Sim\\S00_Demo_BatchSession\\S00_P01_MultiParticipants\\S00_P01_T02_Participants1-2`. \\\nType `ipython`, and try the following code:\n``` python\nfrom Pose2Sim import Pose2Sim\nPose2Sim.personAssociation()\nPose2Sim.triangulation()\nPose2Sim.filtering()\nPose2Sim.markerAugmentation()\n```\n\nOne .trc file per participant will be generated and stored in the `pose-3d` directory.\\\nYou can then run OpenSim scaling and inverse kinematics for each resulting .trc file as in [Demonstration Part-2](#demonstration-part-2-obtain-3d-joint-angles-with-opensim).\\\nYou can also visualize your results with Blender as in [Demonstration Part-3](#demonstration-part-3-optional-visualize-your-results-with-blender).\n\n*N.B.:* Set *[project]* `multi_person = true` for each trial that contains multiple persons.\\\nSet *[triangulation]* `reorder_trc = true` if you need to run OpenSim and to match the generated .trc files with the static trials.\\\nMake sure that the order of *[markerAugmentation]* `participant_height` and `participant_mass` matches the order of the static trials.\n\n*N.B.:* Note that in the case of our floating ghost participant, marker augmentation may worsen the results. See [Marker augmentation](#marker-augmentation) for instruction on when and when not to use it.\n\n\n</br></br>\n\n# Use on your own data\n\n> _**Deeper explanations and instructions are given below.**_ \\\n> N.B.: If a step is not relevant for your use case (synchronization, person association, marker augmentation...), you can skip it.\n\n</br>\n\n## Setting your project up\n > _**Get ready for automatic batch processing.**_\n \n### Retrieve the folder structure\n 1. Open a terminal, enter `pip show pose2sim`, report package location. \\\n Copy this path and do `cd <path>\\pose2sim`.\n 2. Copy the *single trial* or *batch session* folder wherever you like, and rename it as you wish. \n 3. The rest of the tutorial will explain to you how to populate the `Calibration` and `videos` folders, edit the [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S01_Demo_SingleTrial/Config.toml) files, and run each Pose2Sim step.\n\n</br>\n\n### Single Trial vs. Batch processing\n\n> _**Copy and edit either the [S01_Demo_SingleTrial](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S01_Demo_SingleTrial) folder or the [S00_Demo_BatchSession](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S00_Demo_BatchSession) one.**_ \n> - Single trial is more straight-forward to set up for isolated experiments\n> - Batch processing allows you to run numerous analysis with different parameters and minimal friction\n\n\n\n#### Single trial\n\nThe single trial folder should contain a `Config.toml` file, a `calibration` folder, and a `pose` folder, the latter including one subfolder for each camera.\n\n<pre>\nSingleTrial \\\n\u251c\u2500\u2500 calibration \\\n\u251c\u2500\u2500 pose \\\n\u2514\u2500\u2500 <i><b>Config.toml</i></b>\n</pre>\n\n#### Batch processing\n\nFor batch processing, each session directory should follow a `Session -> Participant -> Trial` structure, with a `Config.toml` file in each of the directory levels. \n\n<pre>\nSession_s1 \\ <i><b>Config.toml</i></b>\n\u251c\u2500\u2500 Calibration\\ \n\u2514\u2500\u2500 Participant_p1 \\ <i><b>Config.toml</i></b>\n \u2514\u2500\u2500 Trial_t1 \\ <i><b>Config.toml</i></b>\n \u2514\u2500\u2500 pose \\\n</pre>\n\nRun Pose2Sim from the `Session` folder if you want to batch process the whole session, from the `Participant` folder if you want to batch process all the trials of a participant, or from the `Trial` folder if you want to process a single trial. There should be one `Calibration` folder per session. \n\nGlobal parameters are given in the `Config.toml` file of the `Session` folder, and can be altered for specific `Participants` or `Trials` by uncommenting keys and their values in their respective Config.toml files.\\\nTry uncommenting `[project]` and set `frame_range = [10,300]` for a Participant for example, or uncomment `[filtering.butterworth]` and set `cut_off_frequency = 10` for a Trial.\n\n</br>\n\n## 2D pose estimation\n> _**Estimate 2D pose from images with Openpose or another pose estimation solution.**_ \\\nN.B.: First film a short static pose that will be used for scaling the OpenSim model (A-pose for example), and then film your motions of interest.\\\nN.B.: Note that the names of your camera folders must follow the same order as in the calibration file, and end with '_json'.\n\n### With OpenPose:\nThe accuracy and robustness of Pose2Sim have been thoroughly assessed only with OpenPose, and especially with the BODY_25B model. Consequently, we recommend using this 2D pose estimation solution. See [OpenPose repository](https://github.com/CMU-Perceptual-Computing-Lab/openpose) for installation and running.\n* Open a command prompt in your **OpenPose** directory. \\\n Launch OpenPose for each `videos` folder: \n ``` cmd\n bin\\OpenPoseDemo.exe --model_pose BODY_25B --video <PATH_TO_TRIAL_DIR>\\videos\\cam01.mp4 --write_json <PATH_TO_TRIAL_DIR>\\pose\\pose_cam01_json\n ```\n* The [BODY_25B model](https://github.com/CMU-Perceptual-Computing-Lab/openpose_train/tree/master/experimental_models) has more accurate results than the standard BODY_25 one and has been extensively tested for Pose2Sim. \\\nYou can also use the [BODY_135 model](https://github.com/CMU-Perceptual-Computing-Lab/openpose_train/tree/master/experimental_models), which allows for the evaluation of pronation/supination, wrist flexion, and wrist deviation.\\\nAll other OpenPose models (BODY_25, COCO, MPII) are also supported.\\\nMake sure you modify the [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S01_Demo_SingleTrial/Config.toml) file accordingly.\n* Use one of the `json_display_with_img.py` or `json_display_with_img.py` scripts (see [Utilities](#utilities)) if you want to display 2D pose detections.\n\n**N.B.:** *OpenPose BODY_25B is the default 2D pose estimation model used in Pose2Sim. However, other skeleton models from other 2D pose estimation solutions can be used alternatively.* \n\n<img src=\"Content/Pose2D.png\" width=\"760\">\n\n### With BlazePose (MediaPipe):\n[Mediapipe BlazePose](https://google.github.io/mediapipe/solutions/pose.html) is very fast, fully runs under Python, handles upside-down postures and wrist movements (but no subtalar ankle angles). \\\nHowever, it is less robust and accurate than OpenPose, and can only detect a single person.\n* Use the script `Blazepose_runsave.py` (see [Utilities](#utilities)) to run BlazePose under Python, and store the detected coordinates in OpenPose (json) or DeepLabCut (h5 or csv) format: \n ``` cmd\n python -m Blazepose_runsave -i input_file -dJs\n ```\n Type in `python -m Blazepose_runsave -h` for explanation on parameters.\n* Make sure you changed the `pose_model` and the `tracked_keypoint` in the [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S01_Demo_SingleTrial/Config.toml) file.\n\n### With DeepLabCut:\nIf you need to detect specific points on a human being, an animal, or an object, you can also train your own model with [DeepLabCut](https://github.com/DeepLabCut/DeepLabCut). In this case, Pose2Sim is used as an alternative to [AniPose](https://github.com/lambdaloop/anipose), but it may yield better results since 3D reconstruction takes confidence into account (see [this article](https://doi.org/10.1080/21681163.2023.2292067)).\n1. Train your DeepLabCut model and run it on your images or videos (more instruction on their repository)\n2. Translate the h5 2D coordinates to json files (with `DLC_to_OpenPose.py` script, see [Utilities](#utilities)): \n ``` cmd\n python -m DLC_to_OpenPose -i input_h5_file\n ```\n3. Edit `pose.CUSTOM` in [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S01_Demo_SingleTrial/Config.toml), and edit the node IDs so that they correspond to the column numbers of the 2D pose file, starting from zero. Make sure you also changed the `pose_model` and the `tracked_keypoint`.\\\n You can visualize your skeleton's hierarchy by changing pose_model to CUSTOM and writing these lines: \n ``` python\n config_path = r'path_to_Config.toml'\n import toml, anytree\n config = toml.load(config_path)\n pose_model = config.get('pose').get('pose_model')\n model = anytree.importer.DictImporter().import_(config.get('pose').get(pose_model))\n for pre, _, node in anytree.RenderTree(model): \n print(f'{pre}{node.name} id={node.id}')\n ```\n4. Create an OpenSim model if you need inverse kinematics.\n\n### With AlphaPose:\n[AlphaPose](https://github.com/MVIG-SJTU/AlphaPose) is one of the main competitors of OpenPose, and its accuracy is comparable. As a top-down approach (unlike OpenPose which is bottom-up), it is faster on single-person detection, but slower on multi-person detection.\\\nAll AlphaPose models are supported (HALPE_26, HALPE_68, HALPE_136, COCO_133, COCO, MPII). For COCO and MPII, AlphaPose must be run with the flag \"--format cmu\".\n* Install and run AlphaPose on your videos (more instruction on their repository)\n* Translate the AlphaPose single json file to OpenPose frame-by-frame files (with `AlphaPose_to_OpenPose.py` script, see [Utilities](#utilities)): \n ``` cmd\n python -m AlphaPose_to_OpenPose -i input_alphapose_json_file\n ```\n* Make sure you changed the `pose_model` and the `tracked_keypoint` in the [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S01_Demo_SingleTrial/Config.toml) file.\n\n</br>\n\n## Camera calibration\n> _**Calculate camera intrinsic properties and extrinsic locations and positions.\\\n> Convert a preexisting calibration file, or calculate intrinsic and extrinsic parameters from scratch.**_\n\nOpen an Anaconda prompt or a terminal in a `Session`, `Participant`, or `Trial` folder.\\\nType `ipython`.\n\n\n``` python\nfrom Pose2Sim import Pose2Sim\nPose2Sim.calibration()\n```\n\nOutput:\\\n<img src=\"Content/Calib2D.png\" width=\"760\">\n<img src=\"Content/CalibFile.png\" width=\"760\">\n\n\n### Convert from Qualisys, Optitrack, Vicon, OpenCap, EasyMocap, or bioCV\n\nIf you already have a calibration file, set `calibration_type` type to `convert` in your [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Empty_project/User/Config.toml) file.\n- **From [Qualisys](https://www.qualisys.com):**\n - Export calibration to `.qca.txt` within QTM.\n - Copy it in the `Calibration` Pose2Sim folder.\n - set `convert_from` to 'qualisys' in your [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S01_Demo_SingleTrial/Config.toml) file. Change `binning_factor` to 2 if you film in 540p.\n- **From [Optitrack](https://optitrack.com/):** Exporting calibration will be available in Motive 3.2. In the meantime:\n - Calculate intrinsics with a board (see next section).\n - Use their C++ API [to retrieve extrinsic properties](https://docs.optitrack.com/developer-tools/motive-api/motive-api-function-reference#tt_cameraxlocation). Translation can be copied as is in your `Calib.toml` file, but TT_CameraOrientationMatrix first needs to be [converted to a Rodrigues vector](https://docs.opencv.org/3.4/d9/d0c/group__calib3d.html#ga61585db663d9da06b68e70cfbf6a1eac) with OpenCV. See instructions [here](https://github.com/perfanalytics/pose2sim/issues/28).\n - Use the `Calib.toml` file as is and do not run Pose2Sim.calibration()\n- **From [Vicon](http://www.vicon.com/Software/Nexus):** \n - Copy your `.xcp` Vicon calibration file to the Pose2Sim `Calibration` folder.\n - set `convert_from` to 'vicon' in your [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S01_Demo_SingleTrial/Config.toml) file. No other setting is needed.\n- **From [OpenCap](https://www.opencap.ai/):** \n - Copy your `.pickle` OpenCap calibration files to the Pose2Sim `Calibration` folder.\n - set `convert_from` to 'opencap' in your [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S01_Demo_SingleTrial/Config.toml) file. No other setting is needed.\n- **From [EasyMocap](https://github.com/zju3dv/EasyMocap/):** \n - Copy your `intri.yml` and `extri.yml` files to the Pose2Sim `Calibration` folder.\n - set `convert_from` to 'easymocap' in your [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S01_Demo_SingleTrial/Config.toml) file. No other setting is needed.\n- **From [bioCV](https://github.com/camera-mc-dev/.github/blob/main/profile/mocapPipe.md):** \n - Copy your bioCV calibration files (no extension) to the Pose2Sim `Calibration` folder.\n - set `convert_from` to 'biocv' in your [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S01_Demo_SingleTrial/Config.toml) file. No other setting is needed.\n- **From [AniPose](https://github.com/lambdaloop/anipose) or [FreeMocap](https://github.com/freemocap/freemocap):** \n - Copy your `.toml` calibration file to the Pose2Sim `Calibration` folder.\n - Calibration can be skipped since Pose2Sim uses the same [Aniposelib](https://anipose.readthedocs.io/en/latest/aniposelibtutorial.html) format.\n\n</br>\n\n### Calculate from scratch\n\n> _**Calculate calibration parameters with a checkerboard, with measurements on the scene, or automatically with detected keypoints.**_\\\n> Take heart, it is not that complicated once you get the hang of it!\n\n > *N.B.:* Try the calibration tool on the Demo by changing `calibration_type` to `calculate` in [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S00_Demo_BatchSession/Config.toml).\\\n For the sake of practicality, there are voluntarily few board images for intrinsic calibration, and few points to click for extrinsic calibration. In spite of this, your reprojection error should be under 1-2 cm, which [does not hinder the quality of kinematic results in practice](https://www.mdpi.com/1424-8220/21/19/6530/htm#:~:text=Angle%20results%20were,Table%203).).\n \n - **Calculate intrinsic parameters with a checkerboard:**\n\n > *N.B.:* _Intrinsic parameters:_ camera properties (focal length, optical center, distortion), usually need to be calculated only once in their lifetime. In theory, cameras with same model and same settings will have identical intrinsic parameters.\\\n > *N.B.:* If you already calculated intrinsic parameters earlier, you can skip this step by setting `overwrite_intrinsics` to false.\n\n - Create a folder for each camera in your `Calibration\\intrinsics` folder.\n - For each camera, film a checkerboard or a charucoboard. Either the board or the camera can be moved.\n - Adjust parameters in the [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S00_Demo_BatchSession/Config.toml) file.\n - Make sure that the board:\n - is filmed from different angles, covers a large part of the video frame, and is in focus.\n - is flat, without reflections, surrounded by a white border, and is not rotationally invariant (Nrows \u2260 Ncols, and Nrows odd if Ncols even).\n - A common error is to specify the external, instead of the internal number of corners. This may be one less than you would intuitively think. \n \n <img src=\"Content/Calib_int.png\" width=\"600\">\n\n ***Intrinsic calibration error should be below 0.5 px.***\n \n- **Calculate extrinsic parameters:** \n\n > *N.B.:* _Extrinsic parameters:_ camera placement in space (position and orientation), need to be calculated every time a camera is moved. Can be calculated from a board, or from points in the scene with known coordinates.\\\n > *N.B.:* If there is no measurable item in the scene, you can temporarily bring something in (a table, for example), perform calibration, and then remove it before you start capturing motion.\n\n - Create a folder for each camera in your `Calibration\\extrinsics` folder.\n - Once your cameras are in place, shortly film either a board laid on the floor, or the raw scene\\\n (only one frame is needed, but do not just take a photo unless you are sure it does not change the image format).\n - Adjust parameters in the [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S00_Demo_BatchSession/Config.toml) file.\n - Then,\n - **With a checkerboard:**\\\n Make sure that it is seen by all cameras. \\\n It should preferably be larger than the one used for intrinsics, as results will not be very accurate out of the covered zone.\n - **With scene measurements** (more flexible and potentially more accurate if points are spread out):\\\n Manually measure the 3D coordinates of 10 or more points in the scene (tiles, lines on wall, boxes, treadmill dimensions...). These points should be as spread out as possible. Replace `object_coords_3d` by these coordinates in Config.toml.\\\n Then you will click on the corresponding image points for each view.\n - **With keypoints:**\\\n For a more automatic calibration, OpenPose keypoints could also be used for calibration.\\\n **COMING SOON!**\n\n <img src=\"Content/Calib_ext.png\" width=\"920\">\n \n ***Extrinsic calibration error should be below 1 cm, but depending on your application, results will still be potentially acceptable up to 2.5 cm.***\n\n</br>\n\n\n## Synchronizing, Tracking, Triangulating, Filtering\n\n### Synchronization\n\n> _**Cameras need to be synchronized, so that 2D points correspond to the same position across cameras.**_\\\n***N.B.:** Skip this step if your cameras are natively synchronized.*\n\nOpen an Anaconda prompt or a terminal in a `Session`, `Participant`, or `Trial` folder.\\\nType `ipython`.\n\n``` python\nfrom Pose2Sim import Pose2Sim\nPose2Sim.synchronization()\n```\n\nFor each camera, this computes mean vertical speed for the chosen keypoints, and finds the time offset for which their correlation is highest.\\\nAll keypoints can be taken into account, or a subset of them. The user can also specify a time for each camera when only one participant is in the scene, preferably performing a clear vertical motion.\n\nIf results are not satisfying, set `reset_sync` to true in `Config.toml` to revert to original state. Then switch to false again and edit the parameters.\n\n*N.B.:* Alternatively, use a flashlight or a clap to synchronize them. GoPro cameras can also be synchronized with a timecode, by GPS (outdoors) or with a remote control (slightly less reliable).\n\n\n\n</br>\n\n### Associate persons across cameras\n\n> _**If `multi_person` is set to `false`, the algorithm chooses the person for whom the reprojection error is smallest.\\\n If `multi_person` is set to `true`, it associates across views the people for whom the distances between epipolar lines are the smallest. People are then associated across frames according to their displacement speed.**_ \\\n***N.B.:** Skip this step if only one person is in the field of view.*\n\nOpen an Anaconda prompt or a terminal in a `Session`, `Participant`, or `Trial` folder.\\\nType `ipython`.\n``` python\nfrom Pose2Sim import Pose2Sim\nPose2Sim.personAssociation()\n```\n\nCheck printed output. If results are not satisfying, try and release the constraints in the [Config.toml](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/S00_Demo_Session/Config.toml) file.\n\nOutput:\\\n<img src=\"Content/Track2D.png\" width=\"760\">\n \n</br>\n\n### Triangulating keypoints\n> _**Triangulate your 2D coordinates in a robust way.**_ \\\n> The triangulation is weighted by the likelihood of each detected 2D keypoint, provided that they this likelihood is above a threshold.\\\n If the reprojection error is above another threshold, right and left sides are swapped; if it is still above, cameras are removed until the threshold is met. If more cameras are removed than a predefined number, triangulation is skipped for this point and this frame. In the end, missing values are interpolated.\n\nOpen an Anaconda prompt or a terminal in a `Session`, `Participant`, or `Trial` folder.\\\nType `ipython`.\n\n``` python\nfrom Pose2Sim import Pose2Sim\nPose2Sim.triangulation()\n```\n\nCheck printed output, and visualize your trc in OpenSim: `File -> Preview experimental data`.\\\nIf your triangulation is not satisfying, try and release the constraints in the `Config.toml` file.\n\nOutput:\\\n<img src=\"Content/Triangulate3D.png\" width=\"760\">\n\n</br>\n\n### Filtering 3D coordinates\n> _**Filter your 3D coordinates.**_\\\n> Numerous filter types are provided, and can be tuned accordingly.\n\nOpen an Anaconda prompt or a terminal in a `Session`, `Participant`, or `Trial` folder.\\\nType `ipython`.\n\n``` python\nfrom Pose2Sim import Pose2Sim\nPose2Sim.filtering()\n```\n\nCheck your filtration with the displayed figures, and visualize your .trc file in OpenSim. If your filtering is not satisfying, try and change the parameters in the `Config.toml` file.\n\nOutput:\\\n<img src=\"Content/FilterPlot.png\" width=\"760\">\n\n<img src=\"Content/Filter3D.png\" width=\"760\">\n\n</br>\n\n### Marker Augmentation\n> _**Use the Stanford LSTM model to estimate the position of 47 virtual markers.**_\\\n_**Note that inverse kinematic results are not necessarily better after marker augmentation.**_ Skip if results are not convincing.\n\n*N.B.:* Marker augmentation tends to give a more stable, but less precise output. In practice, it is mostly beneficial when using less than 4 cameras. \n\n**Make sure that `participant_height` is correct in your `Config.toml` file.** `participant_mass` is mostly optional for IK.\\\nOnly works with models estimating at least the following keypoints (e.g., not COCO):\n``` python\n [\"Neck\", \"RShoulder\", \"LShoulder\", \"RHip\", \"LHip\", \"RKnee\", \"LKnee\",\n \"RAnkle\", \"LAnkle\", \"RHeel\", \"LHeel\", \"RSmallToe\", \"LSmallToe\",\n \"RBigToe\", \"LBigToe\", \"RElbow\", \"LElbow\", \"RWrist\", \"LWrist\"]\n```\nWill not work properly if missing values are not interpolated (i.e., if there are Nan value in the .trc file).\n\n\nOpen an Anaconda prompt or a terminal in a `Session`, `Participant`, or `Trial` folder.\\\nType `ipython`.\n\n``` python\nfrom Pose2Sim import Pose2Sim\nPose2Sim.markerAugmentation()\n```\n\n</br>\n\n## OpenSim kinematics\n> _**Obtain 3D joint angles.**_\\\n> Your OpenSim .osim scaled model and .mot inverse kinematic results will be found in the OpenSim folder of your `Participant` directory.\n\n### OpenSim Scaling\n1. Use the previous steps to capture a static pose, typically an A-pose or a T-pose.\n2. Open OpenSim.\n3. Open the provided `Model_Pose2Sim_LSTM.osim` model from `Pose2Sim/OpenSim_Setup`. *(File -> Open Model)*\n4. Load the provided `Scaling_Setup_Pose2Sim_LSTM.xml` scaling file. *(Tools -> Scale model -> Load)*\n5. Replace the example static .trc file with your own data.\n6. Run\n7. Save the new scaled OpenSim model.\n\n### OpenSim Inverse kinematics\n1. Use Pose2Sim to generate 3D trajectories.\n2. Open OpenSim.\n3. Load the provided `IK_Setup_Pose2Sim_LSTM.xml` scaling file from `Pose2Sim/OpenSim_Setup`. *(Tools -> Inverse kinematics -> Load)*\n4. Replace the example .trc file with your own data, and specify the path to your angle kinematics output file.\n5. Run.\n\n<img src=\"Content/OpenSim.JPG\" width=\"380\">\n\n</br>\n\n### Command line\nAlternatively, you can use command-line tools:\n\n- Open an Anaconda terminal in your OpenSim/bin directory, typically `C:\\OpenSim <Version>\\bin`.\\\n You'll need to adjust the `time_range`, `output_motion_file`, and enter the full paths to the input and output `.osim`, `.trc`, and `.mot` files in your setup file.\n ``` cmd\n opensim-cmd run-tool <PATH TO YOUR SCALING OR IK SETUP FILE>.xml\n ```\n\n- You can also run OpenSim directly in Python:\n ``` python\n import subprocess\n subprocess.call([\"opensim-cmd\", \"run-tool\", r\"<PATH TO YOUR SCALING OR IK SETUP FILE>.xml\"])\n ```\n\n- Or take advantage of the full the OpenSim Python API. See [there](https://simtk-confluence.stanford.edu:8443/display/OpenSim/Scripting+in+Python) for installation instructions (conda install may take a while).\\\nMake sure to replace `py38np120` with your Python version (3.8 in this case) and with your numpy version (1.20 here).\n ``` cmd\n conda install -c opensim-org opensim-moco=4.4=py38np120 -y\n ```\n If you run into a DLL error while importing opensim, open the file `<Pose2Sim-env>\\Lib\\opensim\\__init__.py` and replace `conda`by `conda-meta` line 4. `<Pose2Sim-env>` location can be found with `conda env list`.\\\n Then run: \n `ipython`\n ``` python\n import opensim\n opensim.ScaleTool(\"<PATH TO YOUR SCALING OR IK SETUP FILE>.xml\").run()\n opensim.InverseKinematicsTool(\"<PATH TO YOUR SCALING OR IK SETUP FILE>.xml\").run()\n ```\n You can also run other API commands. See [there](https://simtk-confluence.stanford.edu:8443/display/OpenSim/Common+Scripting+Commands#CommonScriptingCommands-UsingtheTools) for more instructions on how to use it.\n\n</br>\n\n# Utilities\nA list of standalone tools (see [Utilities](https://github.com/perfanalytics/pose2sim/tree/main/Pose2Sim/Utilities)), which can be either run as scripts, or imported as functions. Check usage in the docstring of each Python file. The figure below shows how some of these tools can be used to further extend Pose2Sim usage.\n\n\n<details>\n <summary><b>Converting pose files</b> (CLICK TO SHOW)</summary>\n <pre>\n\n[Blazepose_runsave.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/Blazepose_runsave.py)\nRuns BlazePose on a video, and saves coordinates in OpenPose (json) or DeepLabCut (h5 or csv) format.\n\n[DLC_to_OpenPose.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/DLC_to_OpenPose.py)\nConverts a DeepLabCut (h5) 2D pose estimation file into OpenPose (json) files.\n\n[AlphaPose_to_OpenPose.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/AlphaPose_to_OpenPose.py)\nConverts AlphaPose single json file to OpenPose frame-by-frame files.\n </pre>\n</details>\n\n<details>\n <summary><b>Converting calibration files</b> (CLICK TO SHOW)</summary>\n <pre>\n\n[calib_toml_to_easymocap.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/calib_toml_to_easymocap.py)\nConverts an OpenCV .toml calibration file to EasyMocap intrinsic and extrinsic .yml calibration files.\n\n[calib_easymocap_to_toml.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/calib_easymocap_to_toml.py)\nConverts EasyMocap intrinsic and extrinsic .yml calibration files to an OpenCV .toml calibration file.\n\n[calib_from_checkerboard.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/calib_from_checkerboard.py)\nCalibrates cameras with images or a video of a checkerboard, saves calibration in a Pose2Sim .toml calibration file.\nYou should probably use Pose2Sim.calibration() instead, which is much easier and better.\n\n[calib_qca_to_toml.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/calib_qca_to_toml.py)\nConverts a Qualisys .qca.txt calibration file to the Pose2Sim .toml calibration file (similar to what is used in [AniPose](https://anipose.readthedocs.io/en/latest/)).\n\n[calib_toml_to_qca.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/calib_toml_to_qca.py)\nConverts a Pose2Sim .toml calibration file (e.g., from a checkerboard) to a Qualisys .qca.txt calibration file.\n\n[calib_toml_to_opencap.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/calib_toml_to_opencap.py)\nConverts an OpenCV .toml calibration file to OpenCap .pickle calibration files.\n\n[calib_toml_to_opencap.py]( )\nTo convert OpenCap calibration tiles to a .toml file, please use Pose2Sim.calibration() and set convert_from = 'opencap' in Config.toml.\n </pre>\n</details>\n\n<details>\n <summary><b>Plotting tools</b> (CLICK TO SHOW)</summary>\n <pre>\n\n[json_display_with_img.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/json_display_with_img.py)\nOverlays 2D detected json coordinates on original raw images. High confidence keypoints are green, low confidence ones are red.\n\n[json_display_without_img.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/json_display_without_img.py)\nPlots an animation of 2D detected json coordinates. \n\n[trc_plot.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/trc_plot.py)\nDisplays X, Y, Z coordinates of each 3D keypoint of a TRC file in a different matplotlib tab.\n </pre>\n</details>\n\n<details>\n <summary><b>Other trc tools</b> (CLICK TO SHOW)</summary>\n <pre>\n\n[trc_from_easymocap.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/trc_from_easymocap.py) \nConvert EasyMocap results keypoints3d json files to .trc.\n\n[c3d_to_trc.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/c3d_to_trc.py)\nConverts 3D point data of a .c3d file to a .trc file compatible with OpenSim. No analog data (force plates, emg) nor computed data (angles, powers, etc.) are retrieved.\n\n[trc_desample.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/trc_desample.py)\nUndersamples a trc file.\n\n[trc_Zup_to_Yup.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/trc_Zup_to_Yup.py)\nChanges Z-up system coordinates to Y-up system coordinates.\n\n[trc_filter.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/trc_filter.py)\nFilters trc files. Available filters: Butterworth, Kalman, Butterworth on speed, Gaussian, LOESS, Median.\n\n[trc_gaitevents.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/trc_gaitevents.py)\nDetects gait events from point coordinates according to [Zeni et al. (2008)](https://www.sciencedirect.com/science/article/abs/pii/S0966636207001804?via%3Dihub).\n\n[trc_combine.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/trc_combine.py)\nCombine two trc files, for example a triangulated DeepLabCut trc file and a triangulated OpenPose trc file.\n\n[trc_from_mot_osim.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/trc_from_mot_osim.py)\nBuild a trc file from a .mot motion file and a .osim model file.\n\n[bodykin_from_mot_osim.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/bodykin_from_mot_osim.py)\nConverts a mot file to a .csv file with rotation and orientation of all segments.\n\n[reproj_from_trc_calib.py](https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Utilities/reproj_from_trc_calib.py)\nReprojects 3D coordinates of a trc file to the image planes defined by a calibration file. Output in OpenPose or DeepLabCut format.\n\n </pre>\n</details>\n\n<img src=\"Content/Pose2Sim_workflow_utilities.jpg\" width=\"760\">\n\n</br>\n\n# How to cite and how to contribute\n### How to cite\nIf you use this code or data, please cite [Pagnon et al., 2022b](https://doi.org/10.21105/joss.04362), [Pagnon et al., 2022a](https://www.mdpi.com/1424-8220/22/7/2712), or [Pagnon et al., 2021](https://www.mdpi.com/1424-8220/21/19/6530).\n \n @Article{Pagnon_2022_JOSS, \n AUTHOR = {Pagnon, David and Domalain, Mathieu and Reveret, Lionel}, \n TITLE = {Pose2Sim: An open-source Python package for multiview markerless kinematics}, \n JOURNAL = {Journal of Open Source Software}, \n YEAR = {2022},\n DOI = {10.21105/joss.04362}, \n URL = {https://joss.theoj.org/papers/10.21105/joss.04362}\n }\n\n @Article{Pagnon_2022_Accuracy,\n AUTHOR = {Pagnon, David and Domalain, Mathieu and Reveret, Lionel},\n TITLE = {Pose2Sim: An End-to-End Workflow for 3D Markerless Sports Kinematics\u2014Part 2: Accuracy},\n JOURNAL = {Sensors},\n YEAR = {2022},\n DOI = {10.3390/s22072712},\n URL = {https://www.mdpi.com/1424-8220/22/7/2712}\n }\n\n @Article{Pagnon_2021_Robustness,\n AUTHOR = {Pagnon, David and Domalain, Mathieu and Reveret, Lionel},\n TITLE = {Pose2Sim: An End-to-End Workflow for 3D Markerless Sports Kinematics\u2014Part 1: Robustness},\n JOURNAL = {Sensors},\n YEAR = {2021},\n DOI = {10.3390/s21196530},\n URL = {https://www.mdpi.com/1424-8220/21/19/6530}\n }\n\n</br>\n\n### How to contribute and to-do list\n\nI would happily welcome any proposal for new features, code improvement, and more!\\\nIf you want to contribute to Pose2Sim, please see [this issue](https://github.com/perfanalytics/pose2sim/issues/40).\\\nYou will be proposed a to-do list, but please feel absolutely free to propose your own ideas and improvements.\n\n</br>\n\n**Main to-do list**\n- Graphical User Interface\n- Synchronization\n- Self-calibration based on keypoint detection\n\n</br>\n\n<details>\n <summary><b>Detailed GOT-DONE and TO-DO list</b> (CLICK TO SHOW)</summary>\n <pre>\n \n✔ **Pose:** Support OpenPose [body_25b](https://github.com/CMU-Perceptual-Computing-Lab/openpose_train/tree/master/experimental_models#body_25b-model---option-2-recommended) for more accuracy, [body_135](https://github.com/CMU-Perceptual-Computing-Lab/openpose_train/tree/master/experimental_models#single-network-whole-body-pose-estimation-model) for pronation/supination.\n✔ **Pose:** Support [BlazePose](https://developers.google.com/mediapipe/solutions/vision/pose_landmarker) for faster inference (on mobile device).\n✔ **Pose:** Support [DeepLabCut](http://www.mackenziemathislab.org/deeplabcut) for training on custom datasets.\n✔ **Pose:** Support [AlphaPose](https://github.com/MVIG-SJTU/AlphaPose) as an alternative to OpenPose.\n✔ **Pose:** Define custom model in config.toml rather than in skeletons.py.\n▢ **Pose:** Support [MMPose](https://github.com/open-mmlab/mmpose), [SLEAP](https://sleap.ai/), etc.\n▢ **Pose:** Directly reading from DeepLabCut .csv or .h5 files instead of converting to .json (triangulation, person association, calibration, synchronization...) \n▢ **Pose:** GUI help for DeepLabCut model creation.\n\n✔ **Calibration:** Convert [Qualisys](https://www.qualisys.com) .qca.txt calibration file.\n✔ **Calibration:** Convert [Optitrack](https://optitrack.com/) extrinsic calibration file.\n✔ **Calibration:** Convert [Vicon](http://www.vicon.com/Software/Nexus) .xcp calibration file.\n✔ **Calibration:** Convert [OpenCap](https://www.opencap.ai/) .pickle calibration files.\n✔ **Calibration:** Convert [EasyMocap](https://github.com/zju3dv/EasyMocap/) .yml calibration files.\n✔ **Calibration:** Convert [bioCV](https://github.com/camera-mc-dev/.github/blob/main/profile/mocapPipe.md) calibration files.\n✔ **Calibration:** Easier and clearer calibration procedure: separate intrinsic and extrinsic parameter calculation, edit corner detection if some are wrongly detected (or not visible). \n✔ **Calibration:** Possibility to evaluate extrinsic parameters from cues on scene.\n▢ **Calibration:** Once object points have been detected or clicked once, track them for live calibration of moving cameras. Propose to click again when they are lost.\n▢ **Calibration:** Calibrate cameras by pairs and compute average extrinsic calibration with [aniposelib](https://github.com/lambdaloop/aniposelib/blob/d03b485c4e178d7cff076e9fe1ac36837db49158/aniposelib/utils.py#L167). \n▢ **Calibration:** Fine-tune calibration with bundle adjustment.\n▢ **Calibration:** Support ChArUco board detection (see [there](https://mecaruco2.readthedocs.io/en/latest/notebooks_rst/Aruco/sandbox/ludovic/aruco_calibration_rotation.html)).\n▢ **Calibration:** Calculate calibration with points rather than board. (1) SBA calibration with wand (cf [Argus](https://argus.web.unc.edu), see converter [here](https://github.com/backyardbiomech/DLCconverterDLT/blob/master/DLTcameraPosition.py)). Set world reference frame in the end.\n▢ **Calibration:** Alternatively, self-calibrate with [OpenPose keypoints](https://ietresearch.onlinelibrary.wiley.com/doi/full/10.1049/cvi2.12130). Set world reference frame in the end.\n▢ **Calibration:** Convert [fSpy calibration](https://fspy.io/) based on vanishing point.\n\n▢ **Synchronization:** Synchronize cameras on 2D keypoint speeds. Cf [this draft script](https://github.com/perfanalytics/pose2sim/blob/draft/Pose2Sim/Utilities/synchronize_cams.py).\n\n✔ **Person Association:** Automatically choose the main person to triangulate.\n✔ **Person Association:** Multiple persons association. 1. Triangulate all the persons whose reprojection error is below a certain threshold (instead of only the one with minimum error), and then track in time with speed cf [Slembrouck 2020](https://link.springer.com/chapter/10.1007/978-3-030-40605-9_15)? or 2. Based on affinity matrices [Dong 2021](https://arxiv.org/pdf/1901.04111.pdf)? or 3. Based on occupancy maps [Yildiz 2012](https://link.springer.com/chapter/10.1007/978-3-642-35749-7_10)? or 4. With a neural network [Huang 2023](https://arxiv.org/pdf/2304.09471.pdf)?\n\n✔ **Triangulation:** Triangulation weighted with confidence.\n✔ **Triangulation:** Set a likelihood threshold below which a camera should not be used, a reprojection error threshold, and a minimum number of remaining cameras below which triangulation is skipped for this frame. \n✔ **Triangulation:** Interpolate missing frames (cubic, bezier, linear, slinear, quadratic)\n✔ **Triangulation:** Show mean reprojection error in px and in mm for each keypoint.\n✔ **Triangulation:** Show how many cameras on average had to be excluded for each keypoint.\n✔ **Triangulation:** Evaluate which cameras were the least reliable.\n✔ **Triangulation:** Show which frames had to be interpolated for each keypoint.\n✔ **Triangulation:** Solve limb swapping (although not really an issue with Body_25b). Try triangulating with opposite side if reprojection error too large. Alternatively, ignore right and left sides, use RANSAC or SDS triangulation, and then choose right or left by majority voting. More confidence can be given to cameras whose plane is the most coplanar to the right/left line.\n✔ **Triangulation:** [Undistort](https://docs.opencv.org/3.4/da/d54/group__imgproc__transform.html#ga887960ea1bde84784e7f1710a922b93c) 2D points before triangulating (and [distort](https://github.com/lambdaloop/aniposelib/blob/d03b485c4e178d7cff076e9fe1ac36837db49158/aniposelib/cameras.py#L301) them before computing reprojection error).\n✔ **Triangulation:** Offer the possibility to augment the triangulated data with [the OpenCap LSTM](https://github.com/stanfordnmbl/opencap-core/blob/main/utilsAugmenter.py). Create \"BODY_25_AUGMENTED\" model, Scaling_setup, IK_Setup. \n✔ **Triangulation:** Multiple person kinematics (output multiple .trc coordinates files). Triangulate all persons with reprojection error above threshold, and identify them by minimizing their displacement across frames.\n▢ **Triangulation:** Pre-compile weighted_triangulation and reprojection with @jit(nopython=True, parallel=True) for faster execution.\n▢ **Triangulation:** Offer the possibility of triangulating with Sparse Bundle Adjustment (SBA), Extended Kalman Filter (EKF), Full Trajectory Estimation (FTE) (see [AcinoSet](https://github.com/African-Robotics-Unit/AcinoSet)).\n▢ **Triangulation:** Implement normalized DLT and RANSAC triangulation, Outlier rejection (sliding z-score?), as well as a [triangulation refinement step](https://doi.org/10.1109/TMM.2022.3171102).\n▢ **Triangulation:** Track hands and face (won't be taken into account in OpenSim at this stage).\n\n✔ **Filtering:** Available filtering methods: Butterworth, Butterworth on speed, Gaussian, Median, LOESS (polynomial smoothing).\n✔ **Filtering:** Implement Kalman filter and Kalman smoother.\n▢ **Filtering:** Implement [smoothNet](https://github.com/perfanalytics/pose2sim/issues/29)\n\n✔ **OpenSim:** Integrate better spine from [lifting fullbody model](https://pubmed.ncbi.nlm.nih.gov/30714401) to the [gait full-body model](https://nmbl.stanford.edu/wp-content/uploads/07505900.pdf), more accurate for the knee.\n✔ **OpenSim:** Optimize model marker positions as compared to ground-truth marker-based positions.\n✔ **OpenSim:** Add scaling and inverse kinematics setup files.\n✔ **OpenSim:** Add full model with contact spheres ([SmoothSphereHalfSpaceForce](https://simtk.org/api_docs/opensim/api_docs/classOpenSim_1_1SmoothSphereHalfSpaceForce.html#details)) and full-body muscles ([DeGrooteFregly2016Muscle](https://simtk.org/api_docs/opensim/api_docs/classOpenSim_1_1DeGrooteFregly2016Muscle.html#details)), for [Moco](https://opensim-org.github.io/opensim-moco-site/) for example.\n✔ **OpenSim:** Add model with [ISB shoulder](https://github.com/stanfordnmbl/opencap-core/blob/main/opensimPipeline/Models/LaiUhlrich2022_shoulder.osim).\n▢ **OpenSim:** Implement optimal fixed-interval Kalman smoothing for inverse kinematics ([this OpenSim fork](https://github.com/antoinefalisse/opensim-core/blob/kalman_smoother/OpenSim/Tools/InverseKinematicsKSTool.cpp)), or [Biorbd](https://github.com/pyomeca/biorbd/blob/f776fe02e1472aebe94a5c89f0309360b52e2cbc/src/RigidBody/KalmanReconsMarkers.cpp))\n\n✔ **GUI:** Blender add-on (cf [MPP2SOS](https://blendermarket.com/products/mocap-mpp2soss)), or webapp (e.g., with [Napari](https://napari.org/stable). See my draft project [Maya-Mocap](https://github.com/davidpagnon/Maya-Mocap) and [BlendOsim](https://github.com/JonathanCamargo/BlendOsim).\n▢ **GUI:** 3D plot of cameras and of triangulated keypoints.\n▢ **GUI:** Demo on Google Colab (see [Sports2D](https://bit.ly/Sports2D_Colab) for OpenPose and Python package installation on Google Drive).\n\n✔ **Demo:** Provide Demo data for users to test the code.\n▢ **Demo:** Add videos for users to experiment with other pose detection frameworks\n▢ **Demo:** Time shift videos and json to demonstrate synchronization\n▢ **Demo:** Add another virtual person to demonstrate personAssociation\n▢ **Tutorials:** Make video tutorials.\n▢ **Doc:** Use [Sphinx](https://www.sphinx-doc.org/en/master), [MkDocs](https://www.mkdocs.org), or (maybe better), [github.io](https://docs.github.com/fr/pages/quickstart) for clearer documentation.\n\n✔ **Pip package**\n✔ **Batch processing**\n✔ **Catch errors**\n▢ **Conda package** \n▢ **Docker image**\n▢ Run pose estimation and OpenSim from within Pose2Sim\n▢ Real-time: Run Pose estimation, Person association, Triangulation, Kalman filter, IK frame by frame (instead of running each step for all frames)\n▢ Config parameter for non batch peocessing\n\n▢ **Run from command line via click or typer**\n▢ **Utilities**: Export other data from c3d files into .mot or .sto files (angles, powers, forces, moments, GRF, EMG...)\n▢ **Utilities**: Create trc_to_c3d.py script\n\n✔ **Bug:** calibration.py. FFMPEG error message when calibration files are images. See [there](https://github.com/perfanalytics/pose2sim/issues/33#:~:text=In%20order%20to%20check,filter%20this%20message%20yet.).\n▢ **Bug:** common.py, class plotWindow(). Python crashes after a few runs of `Pose2Sim.filtering()` when `display_figures=true`. See [there](https://github.com/superjax/plotWindow/issues/7).\n</pre>\n</details>\n\n</br>\n\n**Acknowledgements:**\n- Supervised my PhD: [@lreveret](https://github.com/lreveret) (INRIA, Universit\u00e9 Grenoble Alpes), and [@mdomalai](https://github.com/mdomalai) (Universit\u00e9 de Poitiers).\n- Provided the Demo data: [@aaiaueil](https://github.com/aaiaueil) from Universit\u00e9 Gustave Eiffel.\n- Tested the code and provided feedback: [@simonozan](https://github.com/simonozan), [@daeyongyang](https://github.com/daeyongyang), [@ANaaim](https://github.com/ANaaim), [@rlagnsals](https://github.com/rlagnsals)\n- Submitted various accepted pull requests: [@ANaaim](https://github.com/ANaaim), [@rlagnsals](https://github.com/rlagnsals)\n- Provided a code snippet for Optitrack calibration: [@claraaudap](https://github.com/claraaudap) (Universit\u00e9 Bretagne Sud).\n- Issued MPP2SOS, a (non-free) Blender extension based on Pose2Sim: [@carlosedubarreto](https://github.com/carlosedubarreto)\n\n\n\nBSD 3-Clause License\n\nCopyright (c) 2022, perfanalytics\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or without\nmodification, are permitted provided that the following conditions are met:\n\n1. Redistributions of source code must retain the above copyright notice, this\n list of conditions and the following disclaimer.\n\n2. Redistributions in binary form must reproduce the above copyright notice,\n this list of conditions and the following disclaimer in the documentation\n and/or other materials provided with the distribution.\n\n3. Neither the name of the copyright holder nor the names of its\n contributors may be used to endorse or promote products derived from\n this software without specific prior written permission.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\nAND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\nIMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\nDISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE\nFOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL\nDAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR\nSERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\nCAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,\nOR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\nOF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n",
"bugtrack_url": null,
"license": "BSD 3-Clause License",
"summary": "Perform a markerless kinematic analysis from multiple calibrated views as a unified workflow from an OpenPose input to an OpenSim result.",
"version": "0.8.1",
"project_urls": {
"Bug Tracker": "https://github.com/perfanalytics/pose2sim/issues",
"Homepage": "https://github.com/perfanalytics/pose2sim"
},
"split_keywords": [
"markerless",
" kinematics",
" openpose",
" opensim",
" 3d human pose",
" biomechanics"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "203e91cac3a77c803a5fb233b54cf2259c5adb897fe36cf71083fcc41973fbc5",
"md5": "61c03ff50fe55672cc08b71e052bab00",
"sha256": "ff88020cca9a96c92f8d88851e1fc324dbb475174949356d109d1d65cfec1166"
},
"downloads": -1,
"filename": "pose2sim-0.8.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "61c03ff50fe55672cc08b71e052bab00",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 27901305,
"upload_time": "2024-04-16T15:46:27",
"upload_time_iso_8601": "2024-04-16T15:46:27.741240Z",
"url": "https://files.pythonhosted.org/packages/20/3e/91cac3a77c803a5fb233b54cf2259c5adb897fe36cf71083fcc41973fbc5/pose2sim-0.8.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b1dc1148609340b0e0c51a0325eb3b32287b1c92420aeaa0cc382e84898ee97f",
"md5": "87714590c3efc427e0167a788bb5acc4",
"sha256": "4d60e9626702e2efabe30cebcf3f29f150205d97bdcb0c63b32ca322627234f2"
},
"downloads": -1,
"filename": "pose2sim-0.8.1.tar.gz",
"has_sig": false,
"md5_digest": "87714590c3efc427e0167a788bb5acc4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 27114888,
"upload_time": "2024-04-16T15:46:31",
"upload_time_iso_8601": "2024-04-16T15:46:31.646496Z",
"url": "https://files.pythonhosted.org/packages/b1/dc/1148609340b0e0c51a0325eb3b32287b1c92420aeaa0cc382e84898ee97f/pose2sim-0.8.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-16 15:46:31",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "perfanalytics",
"github_project": "pose2sim",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "pose2sim"
}
JSON
