# BoxMOT: pluggable SOTA tracking modules for segmentation, object detection and pose estimation models
<div align="center">
<p>
<img src="assets/images/track_all_seg_1280_025conf.gif" width="400"/>
</p>
<br>
<div>
<a href="https://github.com/mikel-brostrom/yolov8_tracking/actions/workflows/ci.yml"><img src="https://github.com/mikel-brostrom/yolov8_tracking/actions/workflows/ci.yml/badge.svg" alt="CI CPU testing"></a>
<a href="https://pepy.tech/project/boxmot"><img src="https://static.pepy.tech/badge/boxmot"></a>
<br>
<a href="https://colab.research.google.com/drive/18nIqkBr68TkK8dHdarxTco6svHUJGggY?usp=sharing"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a>
<a href="https://doi.org/10.5281/zenodo.8132989"><img src="https://zenodo.org/badge/DOI/10.5281/zenodo.8132989.svg" alt="DOI"></a>
</div>
</div>
## Introduction
This repo contains a collections of pluggable state-of-the-art multi-object trackers for segmentation, object detection and pose estimation models. For the methods using appearance description, both heavy ([CLIPReID](https://arxiv.org/pdf/2211.13977.pdf)) and lightweight state-of-the-art ReID models ([LightMBN](https://arxiv.org/pdf/2101.10774.pdf), [OSNet](https://arxiv.org/pdf/1905.00953.pdf) and more) are available for automatic download. We provide examples on how to use this package together with popular object detection models such as: [Yolov8](https://github.com/ultralytics), [Yolo-NAS](https://github.com/Deci-AI/super-gradients) and [YOLOX](https://github.com/Megvii-BaseDetection/YOLOX).
<div align="center">
| Tracker | HOTA↑ | MOTA↑ | IDF1↑ |
| -------- | ----- | ----- | ----- |
| [BoTSORT](https://arxiv.org/pdf/2206.14651.pdf) | 77.8 | 78.9 | 88.9 |
| [DeepOCSORT](https://arxiv.org/pdf/2302.11813.pdf) | 77.4 | 78.4 | 89.0 |
| [OCSORT](https://arxiv.org/pdf/2203.14360.pdf) | 77.4 | 78.4 | 89.0 |
| [HybridSORT](https://arxiv.org/pdf/2308.00783.pdf) | 77.3 | 77.9 | 88.8 |
| [ByteTrack](https://arxiv.org/pdf/2110.06864.pdf) | 75.6 | 74.6 | 86.0 |
| [StrongSORT](https://arxiv.org/pdf/2202.13514.pdf) | | | |
| <img width=200/> | <img width=100/> | <img width=100/> | <img width=100/> |
<sub> NOTES: performed on the 10 first frames of each MOT17 sequence. The detector used is ByteTrack's YoloXm, trained on: CrowdHuman, MOT17, Cityperson and ETHZ. Each tracker is configured with its original parameters found in their respective official repository.</sub>
</div>
</details>
<details>
<summary>Tutorials</summary>
* [Yolov8 training (link to external repository)](https://docs.ultralytics.com/modes/train/)
* [Deep appearance descriptor training (link to external repository)](https://kaiyangzhou.github.io/deep-person-reid/user_guide.html)
* [ReID model export to ONNX, OpenVINO, TensorRT and TorchScript](https://github.com/mikel-brostrom/yolo_tracking/wiki/ReID-multi-framework-model-export)
* [Evaluation on custom tracking dataset](https://github.com/mikel-brostrom/yolo_tracking/wiki/How-to-evaluate-on-custom-tracking-dataset)
* [ReID inference acceleration with Nebullvm](https://colab.research.google.com/drive/1APUZ1ijCiQFBR9xD0gUvFUOC8yOJIvHm?usp=sharing)
</details>
<details>
<summary>Experiments</summary>
In inverse chronological order:
* [Evaluation of the params evolved for first half of MOT17 on the complete MOT17](https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/wiki/Evaluation-of-the-params-evolved-for-first-half-of-MOT17-on-the-complete-MOT17)
* [Segmentation model vs object detetion model on MOT metrics](https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/wiki/Segmentation-model-vs-object-detetion-model-on-MOT-metrics)
* [Effect of masking objects before feature extraction](https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/wiki/Masked-detection-crops-vs-regular-detection-crops-for-ReID-feature-extraction)
* [conf-thres vs HOTA, MOTA and IDF1](https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/wiki/conf-thres-vs-MOT-metrics)
* [Effect of KF updates ahead for tracks with no associations on MOT17](https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/wiki/Effect-of-KF-updates-ahead-for-tracks-with-no-associations,-on-MOT17)
* [Effect of full images vs 1280 input to StrongSORT on MOT17](https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/wiki/Effect-of-passing-full-image-input-vs-1280-re-scaled-to-StrongSORT-on-MOT17)
* [Effect of different OSNet architectures on MOT16](https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/wiki/OSNet-architecture-performances-on-MOT16)
* [Yolov5 StrongSORT vs BoTSORT vs OCSORT](https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/wiki/StrongSORT-vs-BoTSORT-vs-OCSORT)
* Yolov5 [BoTSORT](https://arxiv.org/abs/2206.14651) branch: https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/tree/botsort
* [Yolov5 StrongSORT OSNet vs other trackers MOT17](https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/wiki/MOT-17-evaluation-(private-detector))
* [StrongSORT MOT16 ablation study](https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/wiki/Yolov5DeepSORTwithOSNet-vs-Yolov5StrongSORTwithOSNet-ablation-study-on-MOT16)
* [Yolov5 StrongSORT OSNet vs other trackers MOT16 (deprecated)](https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/wiki/MOT-16-evaluation)
</details>
#### News
* Enabled tracking per class for all trackers besides StrongSORT by `--per-class` (March 2024)
* Enabled trajectory plotting for all trackers besides StrongSORT by `--show-trajectories` (March 2024)
* All trackers inherit from BaseTracker (Mars 2024)
* Switched from setuptools to poetry for unified: dependency resolution, packaging and publishing management (March 2024)
* ~x3 pipeline speedup by: using pregenerated detections + embeddings and jobs parallelization (March 2024)
* Ultra fast exerimentation enabled by allowing local detections and embeddings saving. This data can then be loaded into any tracking algorithm, avoiding the overhead of repeatedly generating it (February 2024)
* Centroid-based cost function added to OCSORT and DeepOCSORT (suitable for: small and/or high speed objects and low FPS videos) (January 2024)
* Custom Ultralytics package updated from 8.0.124 to 8.0.224 (December 2023)
* HybridSORT available (August 2023)
* SOTA CLIP-ReID people and vehicle models available (August 2023)
## Why BOXMOT?
Today's multi-object tracking options are heavily dependant on the computation capabilities of the underlaying hardware. BoxMOT provides a great variety of tracking methods that meet different hardware limitations, all the way from CPU only to larger GPUs. Morover, we provide scripts for ultra fast experimentation by saving detections and embeddings, which then be loaded into any tracking algorithm. Avoiding the overhead of repeatedly generating this data.
## Installation
Start with [**Python>=3.8**](https://www.python.org/) environment.
If you want to run the YOLOv8, YOLO-NAS or YOLOX examples:
```
git clone https://github.com/mikel-brostrom/yolo_tracking.git
cd yolo_tracking
pip install poetry
poetry install --with yolo # installed boxmot + yolo dependencies
poetry shell # activates the newly created environment with the installed dependencies
```
but if you only want to import the tracking modules you can simply:
```
pip install boxmot
```
## YOLOv8 | YOLO-NAS | YOLOX examples
<details>
<summary>Tracking</summary>
<details>
<summary>Yolo models</summary>
```bash
$ python tracking/track.py --yolo-model yolov8n # bboxes only
python tracking/track.py --yolo-model yolo_nas_s # bboxes only
python tracking/track.py --yolo-model yolox_n # bboxes only
yolov8n-seg # bboxes + segmentation masks
yolov8n-pose # bboxes + pose estimation
```
</details>
<details>
<summary>Tracking methods</summary>
```bash
$ python tracking/track.py --tracking-method deepocsort
strongsort
ocsort
bytetrack
botsort
```
</details>
<details>
<summary>Tracking sources</summary>
Tracking can be run on most video formats
```bash
$ python tracking/track.py --source 0 # webcam
img.jpg # image
vid.mp4 # video
path/ # directory
path/*.jpg # glob
'https://youtu.be/Zgi9g1ksQHc' # YouTube
'rtsp://example.com/media.mp4' # RTSP, RTMP, HTTP stream
```
</details>
<details>
<summary>Select ReID model</summary>
Some tracking methods combine appearance description and motion in the process of tracking. For those which use appearance, you can choose a ReID model based on your needs from this [ReID model zoo](https://kaiyangzhou.github.io/deep-person-reid/MODEL_ZOO). These model can be further optimized for you needs by the [reid_export.py](https://github.com/mikel-brostrom/yolo_tracking/blob/master/boxmot/deep/reid_export.py) script
```bash
$ python tracking/track.py --source 0 --reid-model lmbn_n_cuhk03_d.pt # lightweight
osnet_x0_25_market1501.pt
mobilenetv2_x1_4_msmt17.engine
resnet50_msmt17.onnx
osnet_x1_0_msmt17.pt
clip_market1501.pt # heavy
clip_vehicleid.pt
...
```
</details>
<details>
<summary>Filter tracked classes</summary>
By default the tracker tracks all MS COCO classes.
If you want to track a subset of the classes that you model predicts, add their corresponding index after the classes flag,
```bash
python tracking/track.py --source 0 --yolo-model yolov8s.pt --classes 16 17 # COCO yolov8 model. Track cats and dogs, only
```
[Here](https://tech.amikelive.com/node-718/what-object-categories-labels-are-in-coco-dataset/) is a list of all the possible objects that a Yolov8 model trained on MS COCO can detect. Notice that the indexing for the classes in this repo starts at zero
</details>
<details>
<summary>MOT compliant results</summary>
Can be saved to your experiment folder `runs/track/exp*/` by
```bash
python tracking/track.py --source ... --save-mot
```
</details>
</details>
<details>
<summary>Evaluation</summary>
Evaluate a combination of detector, tracking method and ReID model on standard MOT dataset or you custom one by
```bash
# saves dets and embs under ./runs/dets_n_embs separately for each selected yolo and reid model
$ python tracking/generate_dets_n_embs.py --source ./assets/MOT17-mini/train --yolo-model yolov8n.pt yolov8s.pt --reid-model weights/osnet_x0_25_msmt17.pt
# generate MOT challenge format results based on pregenerated detections and embeddings for a specific trackign method
$ python tracking/generate_mot_metrics.py --dets yolov8n --embs osnet_x0_25_msmt17 --tracking-method botsort
# uses TrackEval to generate MOT metrics for the tracking results under ./runs/mot/<dets+embs+tracking-method>
$ python tracking/val.py --benchmark MOT17-mini --dets yolov8n --embs osnet_x0_25_msmt17 --tracking-method botsort
```
</details>
<details>
<summary>Evolution</summary>
We use a fast and elitist multiobjective genetic algorithm for tracker hyperparameter tuning. By default the objectives are: HOTA, MOTA, IDF1. Run it by
```bash
# saves dets and embs under ./runs/dets_n_embs separately for each selected yolo and reid model
$ python tracking/generate_dets_n_embs.py --source ./assets/MOT17-mini/train --yolo-model yolov8n.pt yolov8s.pt --reid-model weights/osnet_x0_25_msmt17.pt
# evolve parameters for specified tracking method using the selected detections and embeddings generated in the previous step
$ python tracking/evolve.py --benchmark MOT17-mini --dets yolov8n --embs osnet_x0_25_msmt17 --n-trials 9 --tracking-method botsort
```
The set of hyperparameters leading to the best HOTA result are written to the tracker's config file.
</details>
## Custom tracking examples
<details>
<summary>Detection</summary>
```python
import cv2
import numpy as np
from pathlib import Path
from boxmot import DeepOCSORT
tracker = DeepOCSORT(
model_weights=Path('osnet_x0_25_msmt17.pt'), # which ReID model to use
device='cuda:0',
fp16=False,
)
vid = cv2.VideoCapture(0)
while True:
ret, im = vid.read()
# substitute by your object detector, output has to be N X (x, y, x, y, conf, cls)
dets = np.array([[144, 212, 578, 480, 0.82, 0],
[425, 281, 576, 472, 0.56, 65]])
tracker.update(dets, im) # --> M X (x, y, x, y, id, conf, cls, ind)
tracker.plot_results(im, show_trajectories=True)
# break on pressing q or space
cv2.imshow('BoxMOT detection', im)
key = cv2.waitKey(1) & 0xFF
if key == ord(' ') or key == ord('q'):
break
vid.release()
cv2.destroyAllWindows()
```
</details>
<details>
<summary>Pose & segmentation</summary>
```python
import cv2
import numpy as np
from pathlib import Path
from boxmot import DeepOCSORT
tracker = DeepOCSORT(
model_weights=Path('osnet_x0_25_msmt17.pt'), # which ReID model to use
device='cuda:0',
fp16=True,
)
vid = cv2.VideoCapture(0)
while True:
ret, im = vid.read()
keypoints = np.random.rand(2, 17, 3)
mask = np.random.rand(2, 480, 640)
# substitute by your object detector, input to tracker has to be N X (x, y, x, y, conf, cls)
dets = np.array([[144, 212, 578, 480, 0.82, 0],
[425, 281, 576, 472, 0.56, 65]])
tracks = tracker.update(dets, im) # --> M x (x, y, x, y, id, conf, cls, ind)
# xyxys = tracks[:, 0:4].astype('int') # float64 to int
# ids = tracks[:, 4].astype('int') # float64 to int
# confs = tracks[:, 5]
# clss = tracks[:, 6].astype('int') # float64 to int
inds = tracks[:, 7].astype('int') # float64 to int
# in case you have segmentations or poses alongside with your detections you can use
# the ind variable in order to identify which track is associated to each seg or pose by:
# masks = masks[inds]
# keypoints = keypoints[inds]
# such that you then can: zip(tracks, masks) or zip(tracks, keypoints)
# break on pressing q or space
cv2.imshow('BoxMOT segmentation | pose', im)
key = cv2.waitKey(1) & 0xFF
if key == ord(' ') or key == ord('q'):
break
vid.release()
cv2.destroyAllWindows()
```
</details>
<details>
<summary>Tiled inference</summary>
```py
from sahi import AutoDetectionModel
from sahi.predict import get_sliced_prediction
import cv2
import numpy as np
from pathlib import Path
from boxmot import DeepOCSORT
tracker = DeepOCSORT(
model_weights=Path('osnet_x0_25_msmt17.pt'), # which ReID model to use
device='cpu',
fp16=False,
)
detection_model = AutoDetectionModel.from_pretrained(
model_type='yolov8',
model_path='yolov8n.pt',
confidence_threshold=0.5,
device="cpu", # or 'cuda:0'
)
vid = cv2.VideoCapture(0)
color = (0, 0, 255) # BGR
thickness = 2
fontscale = 0.5
while True:
ret, im = vid.read()
# get sliced predictions
result = get_sliced_prediction(
im,
detection_model,
slice_height=256,
slice_width=256,
overlap_height_ratio=0.2,
overlap_width_ratio=0.2
)
num_predictions = len(result.object_prediction_list)
dets = np.zeros([num_predictions, 6], dtype=np.float32)
for ind, object_prediction in enumerate(result.object_prediction_list):
dets[ind, :4] = np.array(object_prediction.bbox.to_xyxy(), dtype=np.float32)
dets[ind, 4] = object_prediction.score.value
dets[ind, 5] = object_prediction.category.id
tracks = tracker.update(dets, im) # --> (x, y, x, y, id, conf, cls, ind)
tracker.plot_results(im, show_trajectories=True)
# break on pressing q or space
cv2.imshow('BoxMOT tiled inference', im)
key = cv2.waitKey(1) & 0xFF
if key == ord(' ') or key == ord('q'):
break
vid.release()
cv2.destroyAllWindows()
```
</details>
## Contributors
<a href="https://github.com/mikel-brostrom/yolo_tracking/graphs/contributors ">
<img src="https://contrib.rocks/image?repo=mikel-brostrom/yolo_tracking" />
</a>
## Contact
For Yolo tracking bugs and feature requests please visit [GitHub Issues](https://github.com/mikel-brostrom/yolo_tracking/issues).
For business inquiries or professional support requests please send an email to: yolov5.deepsort.pytorch@gmail.com
Raw data
{
"_id": null,
"home_page": null,
"name": "boxmot",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.8",
"maintainer_email": null,
"keywords": "tracking, tracking-by-detection, machine-learning, deep-learning, vision, ML, DL, AI, YOLO",
"author": "Mikel Brostr\u00f6m",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/dc/e5/d7be9a7226505f5c8f62cf418e0b4eb8ccabe15604b97b40a8ae565d5a17/boxmot-10.0.63.tar.gz",
"platform": null,
"description": "# BoxMOT: pluggable SOTA tracking modules for segmentation, object detection and pose estimation models\n\n<div align=\"center\">\n <p>\n <img src=\"assets/images/track_all_seg_1280_025conf.gif\" width=\"400\"/>\n </p>\n <br>\n <div>\n <a href=\"https://github.com/mikel-brostrom/yolov8_tracking/actions/workflows/ci.yml\"><img src=\"https://github.com/mikel-brostrom/yolov8_tracking/actions/workflows/ci.yml/badge.svg\" alt=\"CI CPU testing\"></a>\n <a href=\"https://pepy.tech/project/boxmot\"><img src=\"https://static.pepy.tech/badge/boxmot\"></a>\n <br>\n <a href=\"https://colab.research.google.com/drive/18nIqkBr68TkK8dHdarxTco6svHUJGggY?usp=sharing\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"></a>\n<a href=\"https://doi.org/10.5281/zenodo.8132989\"><img src=\"https://zenodo.org/badge/DOI/10.5281/zenodo.8132989.svg\" alt=\"DOI\"></a>\n\n </div>\n</div>\n\n## Introduction\n\nThis repo contains a collections of pluggable state-of-the-art multi-object trackers for segmentation, object detection and pose estimation models. For the methods using appearance description, both heavy ([CLIPReID](https://arxiv.org/pdf/2211.13977.pdf)) and lightweight state-of-the-art ReID models ([LightMBN](https://arxiv.org/pdf/2101.10774.pdf), [OSNet](https://arxiv.org/pdf/1905.00953.pdf) and more) are available for automatic download. We provide examples on how to use this package together with popular object detection models such as: [Yolov8](https://github.com/ultralytics), [Yolo-NAS](https://github.com/Deci-AI/super-gradients) and [YOLOX](https://github.com/Megvii-BaseDetection/YOLOX).\n\n<div align=\"center\">\n\n| Tracker | HOTA\u2191 | MOTA\u2191 | IDF1\u2191 |\n| -------- | ----- | ----- | ----- |\n| [BoTSORT](https://arxiv.org/pdf/2206.14651.pdf) | 77.8 | 78.9 | 88.9 |\n| [DeepOCSORT](https://arxiv.org/pdf/2302.11813.pdf) | 77.4 | 78.4 | 89.0 |\n| [OCSORT](https://arxiv.org/pdf/2203.14360.pdf) | 77.4 | 78.4 | 89.0 |\n| [HybridSORT](https://arxiv.org/pdf/2308.00783.pdf) | 77.3 | 77.9 | 88.8 |\n| [ByteTrack](https://arxiv.org/pdf/2110.06864.pdf) | 75.6 | 74.6 | 86.0 |\n| [StrongSORT](https://arxiv.org/pdf/2202.13514.pdf) | | | |\n| <img width=200/> | <img width=100/> | <img width=100/> | <img width=100/> |\n\n<sub> NOTES: performed on the 10 first frames of each MOT17 sequence. The detector used is ByteTrack's YoloXm, trained on: CrowdHuman, MOT17, Cityperson and ETHZ. Each tracker is configured with its original parameters found in their respective official repository.</sub>\n\n</div>\n\n</details>\n\n<details>\n<summary>Tutorials</summary>\n \n* [Yolov8 training (link to external repository)](https://docs.ultralytics.com/modes/train/) \n* [Deep appearance descriptor training (link to external repository)](https://kaiyangzhou.github.io/deep-person-reid/user_guide.html) \n* [ReID model export to ONNX, OpenVINO, TensorRT and TorchScript](https://github.com/mikel-brostrom/yolo_tracking/wiki/ReID-multi-framework-model-export) \n* [Evaluation on custom tracking dataset](https://github.com/mikel-brostrom/yolo_tracking/wiki/How-to-evaluate-on-custom-tracking-dataset) \n* [ReID inference acceleration with Nebullvm](https://colab.research.google.com/drive/1APUZ1ijCiQFBR9xD0gUvFUOC8yOJIvHm?usp=sharing) \n\n </details>\n\n<details>\n<summary>Experiments</summary>\n\nIn inverse chronological order:\n\n* [Evaluation of the params evolved for first half of MOT17 on the complete MOT17](https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/wiki/Evaluation-of-the-params-evolved-for-first-half-of-MOT17-on-the-complete-MOT17)\n\n* [Segmentation model vs object detetion model on MOT metrics](https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/wiki/Segmentation-model-vs-object-detetion-model-on-MOT-metrics)\n\n* [Effect of masking objects before feature extraction](https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/wiki/Masked-detection-crops-vs-regular-detection-crops-for-ReID-feature-extraction)\n\n* [conf-thres vs HOTA, MOTA and IDF1](https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/wiki/conf-thres-vs-MOT-metrics)\n\n* [Effect of KF updates ahead for tracks with no associations on MOT17](https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/wiki/Effect-of-KF-updates-ahead-for-tracks-with-no-associations,-on-MOT17)\n\n* [Effect of full images vs 1280 input to StrongSORT on MOT17](https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/wiki/Effect-of-passing-full-image-input-vs-1280-re-scaled-to-StrongSORT-on-MOT17)\n\n* [Effect of different OSNet architectures on MOT16](https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/wiki/OSNet-architecture-performances-on-MOT16)\n\n* [Yolov5 StrongSORT vs BoTSORT vs OCSORT](https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/wiki/StrongSORT-vs-BoTSORT-vs-OCSORT)\n * Yolov5 [BoTSORT](https://arxiv.org/abs/2206.14651) branch: https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/tree/botsort\n\n* [Yolov5 StrongSORT OSNet vs other trackers MOT17](https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/wiki/MOT-17-evaluation-(private-detector)) \n\n* [StrongSORT MOT16 ablation study](https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/wiki/Yolov5DeepSORTwithOSNet-vs-Yolov5StrongSORTwithOSNet-ablation-study-on-MOT16) \n\n* [Yolov5 StrongSORT OSNet vs other trackers MOT16 (deprecated)](https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/wiki/MOT-16-evaluation) \n\n </details>\n\n#### News\n\n* Enabled tracking per class for all trackers besides StrongSORT by `--per-class` (March 2024)\n* Enabled trajectory plotting for all trackers besides StrongSORT by `--show-trajectories` (March 2024)\n* All trackers inherit from BaseTracker (Mars 2024)\n* Switched from setuptools to poetry for unified: dependency resolution, packaging and publishing management (March 2024)\n* ~x3 pipeline speedup by: using pregenerated detections + embeddings and jobs parallelization (March 2024)\n* Ultra fast exerimentation enabled by allowing local detections and embeddings saving. This data can then be loaded into any tracking algorithm, avoiding the overhead of repeatedly generating it (February 2024)\n* Centroid-based cost function added to OCSORT and DeepOCSORT (suitable for: small and/or high speed objects and low FPS videos) (January 2024)\n* Custom Ultralytics package updated from 8.0.124 to 8.0.224 (December 2023)\n* HybridSORT available (August 2023)\n* SOTA CLIP-ReID people and vehicle models available (August 2023)\n\n\n## Why BOXMOT?\n\nToday's multi-object tracking options are heavily dependant on the computation capabilities of the underlaying hardware. BoxMOT provides a great variety of tracking methods that meet different hardware limitations, all the way from CPU only to larger GPUs. Morover, we provide scripts for ultra fast experimentation by saving detections and embeddings, which then be loaded into any tracking algorithm. Avoiding the overhead of repeatedly generating this data.\n\n## Installation\n\nStart with [**Python>=3.8**](https://www.python.org/) environment.\n\nIf you want to run the YOLOv8, YOLO-NAS or YOLOX examples:\n\n```\ngit clone https://github.com/mikel-brostrom/yolo_tracking.git\ncd yolo_tracking\npip install poetry\npoetry install --with yolo # installed boxmot + yolo dependencies\npoetry shell # activates the newly created environment with the installed dependencies\n```\n\nbut if you only want to import the tracking modules you can simply:\n\n```\npip install boxmot\n```\n\n## YOLOv8 | YOLO-NAS | YOLOX examples\n\n<details>\n<summary>Tracking</summary>\n\n<details>\n<summary>Yolo models</summary>\n\n\n\n```bash\n$ python tracking/track.py --yolo-model yolov8n # bboxes only\n python tracking/track.py --yolo-model yolo_nas_s # bboxes only\n python tracking/track.py --yolo-model yolox_n # bboxes only\n yolov8n-seg # bboxes + segmentation masks\n yolov8n-pose # bboxes + pose estimation\n\n```\n\n </details>\n\n<details>\n<summary>Tracking methods</summary>\n\n```bash\n$ python tracking/track.py --tracking-method deepocsort\n strongsort\n ocsort\n bytetrack\n botsort\n```\n\n</details>\n\n<details>\n<summary>Tracking sources</summary>\n\nTracking can be run on most video formats\n\n```bash\n$ python tracking/track.py --source 0 # webcam\n img.jpg # image\n vid.mp4 # video\n path/ # directory\n path/*.jpg # glob\n 'https://youtu.be/Zgi9g1ksQHc' # YouTube\n 'rtsp://example.com/media.mp4' # RTSP, RTMP, HTTP stream\n```\n\n</details>\n\n<details>\n<summary>Select ReID model</summary>\n\nSome tracking methods combine appearance description and motion in the process of tracking. For those which use appearance, you can choose a ReID model based on your needs from this [ReID model zoo](https://kaiyangzhou.github.io/deep-person-reid/MODEL_ZOO). These model can be further optimized for you needs by the [reid_export.py](https://github.com/mikel-brostrom/yolo_tracking/blob/master/boxmot/deep/reid_export.py) script\n\n```bash\n$ python tracking/track.py --source 0 --reid-model lmbn_n_cuhk03_d.pt # lightweight\n osnet_x0_25_market1501.pt\n mobilenetv2_x1_4_msmt17.engine\n resnet50_msmt17.onnx\n osnet_x1_0_msmt17.pt\n clip_market1501.pt # heavy\n clip_vehicleid.pt\n ...\n```\n\n</details>\n\n<details>\n<summary>Filter tracked classes</summary>\n\nBy default the tracker tracks all MS COCO classes.\n\nIf you want to track a subset of the classes that you model predicts, add their corresponding index after the classes flag,\n\n```bash\npython tracking/track.py --source 0 --yolo-model yolov8s.pt --classes 16 17 # COCO yolov8 model. Track cats and dogs, only\n```\n\n[Here](https://tech.amikelive.com/node-718/what-object-categories-labels-are-in-coco-dataset/) is a list of all the possible objects that a Yolov8 model trained on MS COCO can detect. Notice that the indexing for the classes in this repo starts at zero\n\n</details>\n\n<details>\n<summary>MOT compliant results</summary>\n\nCan be saved to your experiment folder `runs/track/exp*/` by\n\n```bash\npython tracking/track.py --source ... --save-mot\n```\n\n</details>\n\n</details>\n\n<details>\n<summary>Evaluation</summary>\n\nEvaluate a combination of detector, tracking method and ReID model on standard MOT dataset or you custom one by\n\n```bash\n# saves dets and embs under ./runs/dets_n_embs separately for each selected yolo and reid model\n$ python tracking/generate_dets_n_embs.py --source ./assets/MOT17-mini/train --yolo-model yolov8n.pt yolov8s.pt --reid-model weights/osnet_x0_25_msmt17.pt\n# generate MOT challenge format results based on pregenerated detections and embeddings for a specific trackign method\n$ python tracking/generate_mot_metrics.py --dets yolov8n --embs osnet_x0_25_msmt17 --tracking-method botsort\n# uses TrackEval to generate MOT metrics for the tracking results under ./runs/mot/<dets+embs+tracking-method>\n$ python tracking/val.py --benchmark MOT17-mini --dets yolov8n --embs osnet_x0_25_msmt17 --tracking-method botsort\n```\n\n</details>\n\n\n<details>\n<summary>Evolution</summary>\n\nWe use a fast and elitist multiobjective genetic algorithm for tracker hyperparameter tuning. By default the objectives are: HOTA, MOTA, IDF1. Run it by\n\n```bash\n# saves dets and embs under ./runs/dets_n_embs separately for each selected yolo and reid model\n$ python tracking/generate_dets_n_embs.py --source ./assets/MOT17-mini/train --yolo-model yolov8n.pt yolov8s.pt --reid-model weights/osnet_x0_25_msmt17.pt\n# evolve parameters for specified tracking method using the selected detections and embeddings generated in the previous step\n$ python tracking/evolve.py --benchmark MOT17-mini --dets yolov8n --embs osnet_x0_25_msmt17 --n-trials 9 --tracking-method botsort\n```\n\nThe set of hyperparameters leading to the best HOTA result are written to the tracker's config file.\n\n</details>\n\n\n## Custom tracking examples\n\n<details>\n<summary>Detection</summary>\n\n```python\nimport cv2\nimport numpy as np\nfrom pathlib import Path\n\nfrom boxmot import DeepOCSORT\n\n\ntracker = DeepOCSORT(\n model_weights=Path('osnet_x0_25_msmt17.pt'), # which ReID model to use\n device='cuda:0',\n fp16=False,\n)\n\nvid = cv2.VideoCapture(0)\n\nwhile True:\n ret, im = vid.read()\n\n # substitute by your object detector, output has to be N X (x, y, x, y, conf, cls)\n dets = np.array([[144, 212, 578, 480, 0.82, 0],\n [425, 281, 576, 472, 0.56, 65]])\n\n tracker.update(dets, im) # --> M X (x, y, x, y, id, conf, cls, ind)\n tracker.plot_results(im, show_trajectories=True)\n\n # break on pressing q or space\n cv2.imshow('BoxMOT detection', im) \n key = cv2.waitKey(1) & 0xFF\n if key == ord(' ') or key == ord('q'):\n break\n\nvid.release()\ncv2.destroyAllWindows()\n```\n\n</details>\n\n\n<details>\n<summary>Pose & segmentation</summary>\n\n```python\nimport cv2\nimport numpy as np\nfrom pathlib import Path\n\nfrom boxmot import DeepOCSORT\n\n\ntracker = DeepOCSORT(\n model_weights=Path('osnet_x0_25_msmt17.pt'), # which ReID model to use\n device='cuda:0',\n fp16=True,\n)\n\nvid = cv2.VideoCapture(0)\n\nwhile True:\n ret, im = vid.read()\n\n keypoints = np.random.rand(2, 17, 3)\n mask = np.random.rand(2, 480, 640)\n # substitute by your object detector, input to tracker has to be N X (x, y, x, y, conf, cls)\n dets = np.array([[144, 212, 578, 480, 0.82, 0],\n [425, 281, 576, 472, 0.56, 65]])\n\n tracks = tracker.update(dets, im) # --> M x (x, y, x, y, id, conf, cls, ind)\n\n # xyxys = tracks[:, 0:4].astype('int') # float64 to int\n # ids = tracks[:, 4].astype('int') # float64 to int\n # confs = tracks[:, 5]\n # clss = tracks[:, 6].astype('int') # float64 to int\n inds = tracks[:, 7].astype('int') # float64 to int\n\n # in case you have segmentations or poses alongside with your detections you can use\n # the ind variable in order to identify which track is associated to each seg or pose by:\n # masks = masks[inds]\n # keypoints = keypoints[inds]\n # such that you then can: zip(tracks, masks) or zip(tracks, keypoints)\n\n # break on pressing q or space\n cv2.imshow('BoxMOT segmentation | pose', im) \n key = cv2.waitKey(1) & 0xFF\n if key == ord(' ') or key == ord('q'):\n break\n\nvid.release()\ncv2.destroyAllWindows()\n```\n\n</details>\n\n<details>\n<summary>Tiled inference</summary>\n \n```py\nfrom sahi import AutoDetectionModel\nfrom sahi.predict import get_sliced_prediction\nimport cv2\nimport numpy as np\nfrom pathlib import Path\nfrom boxmot import DeepOCSORT\n\n\ntracker = DeepOCSORT(\n model_weights=Path('osnet_x0_25_msmt17.pt'), # which ReID model to use\n device='cpu',\n fp16=False,\n)\n\ndetection_model = AutoDetectionModel.from_pretrained(\n model_type='yolov8',\n model_path='yolov8n.pt',\n confidence_threshold=0.5,\n device=\"cpu\", # or 'cuda:0'\n)\n\nvid = cv2.VideoCapture(0)\ncolor = (0, 0, 255) # BGR\nthickness = 2\nfontscale = 0.5\n\nwhile True:\n ret, im = vid.read()\n\n # get sliced predictions\n result = get_sliced_prediction(\n im,\n detection_model,\n slice_height=256,\n slice_width=256,\n overlap_height_ratio=0.2,\n overlap_width_ratio=0.2\n )\n num_predictions = len(result.object_prediction_list)\n dets = np.zeros([num_predictions, 6], dtype=np.float32)\n for ind, object_prediction in enumerate(result.object_prediction_list):\n dets[ind, :4] = np.array(object_prediction.bbox.to_xyxy(), dtype=np.float32)\n dets[ind, 4] = object_prediction.score.value\n dets[ind, 5] = object_prediction.category.id\n\n tracks = tracker.update(dets, im) # --> (x, y, x, y, id, conf, cls, ind)\n\n tracker.plot_results(im, show_trajectories=True)\n\n # break on pressing q or space\n cv2.imshow('BoxMOT tiled inference', im) \n key = cv2.waitKey(1) & 0xFF\n if key == ord(' ') or key == ord('q'):\n break\n\nvid.release()\ncv2.destroyAllWindows()\n```\n\n</details>\n\n## Contributors\n\n<a href=\"https://github.com/mikel-brostrom/yolo_tracking/graphs/contributors \">\n <img src=\"https://contrib.rocks/image?repo=mikel-brostrom/yolo_tracking\" />\n</a>\n\n## Contact\n\nFor Yolo tracking bugs and feature requests please visit [GitHub Issues](https://github.com/mikel-brostrom/yolo_tracking/issues).\nFor business inquiries or professional support requests please send an email to: yolov5.deepsort.pytorch@gmail.com\n",
"bugtrack_url": null,
"license": "AGPL-3.0",
"summary": "BoxMOT: pluggable SOTA tracking modules for segmentation, object detection and pose estimation models",
"version": "10.0.63",
"project_urls": null,
"split_keywords": [
"tracking",
" tracking-by-detection",
" machine-learning",
" deep-learning",
" vision",
" ml",
" dl",
" ai",
" yolo"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ea34e8bba5d90832e4ed8b697ddf749d5656361eef77ad9c87fe88eabd3c9abb",
"md5": "d3dfe9b05c44690481dc06e96f63ddfb",
"sha256": "e7aac111dd2d86d26ac9098a02afdb551c56088ba5fd78945fc646d3600a3281"
},
"downloads": -1,
"filename": "boxmot-10.0.63-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d3dfe9b05c44690481dc06e96f63ddfb",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.8",
"size": 1560742,
"upload_time": "2024-03-22T10:59:23",
"upload_time_iso_8601": "2024-03-22T10:59:23.463584Z",
"url": "https://files.pythonhosted.org/packages/ea/34/e8bba5d90832e4ed8b697ddf749d5656361eef77ad9c87fe88eabd3c9abb/boxmot-10.0.63-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "dce5d7be9a7226505f5c8f62cf418e0b4eb8ccabe15604b97b40a8ae565d5a17",
"md5": "396c22d54f05a6270c7f3a1863815017",
"sha256": "53bf3a4315c4ee921cb0a45eeeca2533391c40d79f8ab65cca755817149201fd"
},
"downloads": -1,
"filename": "boxmot-10.0.63.tar.gz",
"has_sig": false,
"md5_digest": "396c22d54f05a6270c7f3a1863815017",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.8",
"size": 1515065,
"upload_time": "2024-03-22T10:59:25",
"upload_time_iso_8601": "2024-03-22T10:59:25.371068Z",
"url": "https://files.pythonhosted.org/packages/dc/e5/d7be9a7226505f5c8f62cf418e0b4eb8ccabe15604b97b40a8ae565d5a17/boxmot-10.0.63.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-22 10:59:25",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "boxmot"
}
JSON