# PyTorch Lightning Optical Flow
## Introduction
This is a collection of state-of-the-art deep model for estimating optical flow. The main goal is to provide a unified framework where multiple models can be trained and tested more easily.
The work and code from many others are present here. I tried to make sure everything is properly referenced, but please let me know if I missed something.
This is still under development, so some things may not work as intended. I plan to add more models in the future, as well keep improving the platform.
- [What's new](#whats-new)
- [Available models](#available-models)
- [Results](#results)
- [Getting started](#getting-started)
- [Licenses](#licenses)
- [Contributing](#contributing)
- [Citing](#citing)
- [Acknowledgements](#acknowledgements)
## What's new
### - v0.3.0
- Added new models:
- DIP [https://arxiv.org/abs/2204.00330](https://arxiv.org/abs/2204.00330)
- Flow1D [https://arxiv.org/abs/2103.04524](https://arxiv.org/abs/2103.04524)
- FlowFormer++ [https://arxiv.org/abs/2303.01237](https://arxiv.org/abs/2303.01237)
- GMFlow+, UniMatch [https://arxiv.org/abs/2211.05783](https://arxiv.org/abs/2211.05783)
- MatchFlow [https://arxiv.org/abs/2303.08384](https://arxiv.org/abs/2303.08384)
- MS-RAFT+ [https://arxiv.org/abs/2210.16900](https://arxiv.org/abs/2210.16900)
- RPKNet [https://hmorimitsu.com/publication/2024-aaai-rpknet](https://hmorimitsu.com/publication/2024-aaai-rpknet)
- SeparableFlow [https://openaccess.thecvf.com/content/ICCV2021/papers/Zhang_Separable_Flow_Learning_Motion_Cost_Volumes_for_Optical_Flow_Estimation_ICCV_2021_paper.pdf](https://openaccess.thecvf.com/content/ICCV2021/papers/Zhang_Separable_Flow_Learning_Motion_Cost_Volumes_for_Optical_Flow_Estimation_ICCV_2021_paper.pdf)
- SKFlow [https://arxiv.org/abs/2205.14623](https://arxiv.org/abs/2205.14623)
- VideoFlow [https://arxiv.org/abs/2303.08340](https://arxiv.org/abs/2303.08340)
- `speed_benchmark.py` becomes `model_benchmark.py` and records more metrics
- Fix compatibility with PyTorch 2.0
- Fix compatibility with PyTorch Lightning 1.9
- Fix resizing augmentation when the valid mask is sparse
- Add support for more datasets:
- Middlebury [https://vision.middlebury.edu/flow/](https://vision.middlebury.edu/flow/)
- Monkaa [https://lmb.informatik.uni-freiburg.de/resources/datasets/SceneFlowDatasets.en.html](https://lmb.informatik.uni-freiburg.de/resources/datasets/SceneFlowDatasets.en.html)
- Spring [https://spring-benchmark.org/](https://spring-benchmark.org/)
## Available models
- CRAFT [https://arxiv.org/abs/2203.16896](https://arxiv.org/abs/2203.16896)
- CSFlow [https://arxiv.org/abs/2202.00909](https://arxiv.org/abs/2202.00909)
- DICL-Flow [https://arxiv.org/abs/2010.14851](https://arxiv.org/abs/2010.14851)
- DIP [https://arxiv.org/abs/2204.00330](https://arxiv.org/abs/2204.00330)
- FastFlowNet [https://arxiv.org/abs/2103.04524](https://arxiv.org/abs/2103.04524)
- Flow1D [https://arxiv.org/abs/2103.04524](https://arxiv.org/abs/2103.04524)
- FlowFormer [https://arxiv.org/abs/2203.16194](https://arxiv.org/abs/2203.16194)
- FlowFormer++ [https://arxiv.org/abs/2303.01237](https://arxiv.org/abs/2303.01237)
- FlowNet [https://arxiv.org/abs/1504.06852](https://arxiv.org/abs/1504.06852)
- FlowNet2 [https://arxiv.org/abs/1612.01925](https://arxiv.org/abs/1612.01925)
- GMA [https://arxiv.org/abs/2104.02409](https://arxiv.org/abs/2104.02409)
- GMFlow [https://arxiv.org/abs/2111.13680](https://arxiv.org/abs/2111.13680)
- GMFlow+, UniMatch [https://arxiv.org/abs/2211.05783](https://arxiv.org/abs/2211.05783)
- GMFlowNet [https://arxiv.org/abs/2203.11335](https://arxiv.org/abs/2203.11335)
- HD3 [https://arxiv.org/abs/1812.06264](https://arxiv.org/abs/1812.06264)
- IRR [https://arxiv.org/abs/1904.05290](https://arxiv.org/abs/1904.05290)
- LCV [https://arxiv.org/abs/2007.11431](https://arxiv.org/abs/2007.11431)
- LiteFlowNet [https://arxiv.org/abs/1805.07036](https://arxiv.org/abs/1805.07036)
- LiteFlowNet2 [https://arxiv.org/abs/1903.07414](https://arxiv.org/abs/1903.07414)
- LiteFlowNet3 [https://arxiv.org/abs/2007.09319](https://arxiv.org/abs/2007.09319)
- MaskFlownet [https://arxiv.org/abs/2003.10955](https://arxiv.org/abs/2003.10955)
- MatchFlow [https://arxiv.org/abs/2303.08384](https://arxiv.org/abs/2303.08384)
- MS-RAFT+ [https://arxiv.org/abs/2210.16900](https://arxiv.org/abs/2210.16900)
- PWCNet [https://arxiv.org/abs/1709.02371](https://arxiv.org/abs/1709.02371)
- RAFT [https://arxiv.org/abs/2003.12039](https://arxiv.org/abs/2003.12039)
- RPKNet [https://hmorimitsu.com/publication/2024-aaai-rpknet](https://hmorimitsu.com/publication/2024-aaai-rpknet)
- ScopeFlow [https://arxiv.org/abs/2002.10770](https://arxiv.org/abs/2002.10770)
- SCV [https://arxiv.org/abs/2104.02166](https://arxiv.org/abs/2104.02166)
- SeparableFlow [https://openaccess.thecvf.com/content/ICCV2021/papers/Zhang_Separable_Flow_Learning_Motion_Cost_Volumes_for_Optical_Flow_Estimation_ICCV_2021_paper.pdf](https://openaccess.thecvf.com/content/ICCV2021/papers/Zhang_Separable_Flow_Learning_Motion_Cost_Volumes_for_Optical_Flow_Estimation_ICCV_2021_paper.pdf)
- SKFlow [https://arxiv.org/abs/2205.14623](https://arxiv.org/abs/2205.14623)
- STaRFlow [https://arxiv.org/abs/2007.05481](https://arxiv.org/abs/2007.05481)
- VCN [https://papers.nips.cc/paper/2019/file/bbf94b34eb32268ada57a3be5062fe7d-Paper.pdf](https://papers.nips.cc/paper/2019/file/bbf94b34eb32268ada57a3be5062fe7d-Paper.pdf)
- VideoFlow [https://arxiv.org/abs/2303.08340](https://arxiv.org/abs/2303.08340)
Read more details about the models on [https://ptlflow.readthedocs.io/en/latest/models/models_list.html](https://ptlflow.readthedocs.io/en/latest/models/models_list.html).
# Results
You can see a table with main evaluation results of the available models [here](https://ptlflow.readthedocs.io/en/latest/results/accuracy_epe.html). More results are also available in the folder [docs/source/results](docs/source/results).
**Disclaimer**: These results are the ones obtained by evaluating the available models in this framework in my machine. Your results may be different due to differences in hardware and software. I also do not guarantee that the results of each model will be similar to the ones presented in the respective papers or other original sources. If you need to replicate the original results from a paper, you should use the original implementations.
## Getting started
Please take a look at the [documentation](https://ptlflow.readthedocs.io/) to learn how to install and use PTLFlow.
You can also check the notebooks below running on Google Colab for some practical examples:
- [Inference with a pretrained model](https://colab.research.google.com/drive/1YARBRUGplqTRnRuY9sKIs6LY_2kWAWZJ?usp=sharing).
- [Training and using the learned weights for inference](https://colab.research.google.com/drive/1mbuAEF728_jZpFEsQHXDxjIGAcB1-nVs?usp=sharing).
## Licenses
The original code of this repository is licensed under the [Apache 2.0 license](LICENSE).
Each model may be subjected to different licenses. The license of each model is included in their respective folders. It is your responsibility to make sure that your project is in compliance with all the licenses and conditions involved.
The external pretrained weights all have different licenses, which are listed in their respective folders.
The pretrained weights that were trained within this project are available under the [CC BY-NC-SA 4.0 license](https://creativecommons.org/licenses/by-nc-sa/4.0/), which I believe that covers the licenses of the datasets used in the training. That being said, I am not a legal expert so if you plan to use them to any purpose other than research, you should check all the involved licenses by yourself. Additionally, the datasets used for the training usually require the user to cite the original papers, so be sure to include their respective references in your work.
## Contributing
Contribution are welcome! Please check [CONTRIBUTING.md](CONTRIBUTING.md) to see how to contribute.
## Citing
### BibTeX
```
@misc{morimitsu2021ptlflow,
author = {Henrique Morimitsu},
title = {PyTorch Lightning Optical Flow},
year = {2021},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/hmorimitsu/ptlflow}}
}
```
## Acknowledgements
- This README file is heavily inspired by the one from the [timm](https://github.com/rwightman/pytorch-image-models) repository.
- Some parts of the code were inspired by or taken from [FlowNetPytorch](https://github.com/ClementPinard/FlowNetPytorch).
- [flownet2-pytorch](https://github.com/NVIDIA/flownet2-pytorch) was also another important source.
- The current main training routine is based on [RAFT](https://github.com/princeton-vl/RAFT).
Raw data
{
"_id": null,
"home_page": "",
"name": "ptlflow",
"maintainer": "",
"docs_url": null,
"requires_python": "<3.12,>=3.8",
"maintainer_email": "",
"keywords": "torch,pytorch lightning,optical flow,models",
"author": "",
"author_email": "Henrique Morimitsu <henriquem87@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/12/41/f641814568237064a730e70ae1fa362800308a67c87e2a252825ee759a0f/ptlflow-0.3.0.tar.gz",
"platform": null,
"description": "# PyTorch Lightning Optical Flow\n\n\n\n\n\n\n## Introduction\n\nThis is a collection of state-of-the-art deep model for estimating optical flow. The main goal is to provide a unified framework where multiple models can be trained and tested more easily.\n\nThe work and code from many others are present here. I tried to make sure everything is properly referenced, but please let me know if I missed something.\n\nThis is still under development, so some things may not work as intended. I plan to add more models in the future, as well keep improving the platform.\n\n- [What's new](#whats-new)\n- [Available models](#available-models)\n- [Results](#results)\n- [Getting started](#getting-started)\n- [Licenses](#licenses)\n- [Contributing](#contributing)\n- [Citing](#citing)\n- [Acknowledgements](#acknowledgements)\n\n## What's new\n\n### - v0.3.0\n\n- Added new models:\n - DIP [https://arxiv.org/abs/2204.00330](https://arxiv.org/abs/2204.00330)\n - Flow1D [https://arxiv.org/abs/2103.04524](https://arxiv.org/abs/2103.04524)\n - FlowFormer++ [https://arxiv.org/abs/2303.01237](https://arxiv.org/abs/2303.01237)\n - GMFlow+, UniMatch [https://arxiv.org/abs/2211.05783](https://arxiv.org/abs/2211.05783)\n - MatchFlow [https://arxiv.org/abs/2303.08384](https://arxiv.org/abs/2303.08384)\n - MS-RAFT+ [https://arxiv.org/abs/2210.16900](https://arxiv.org/abs/2210.16900)\n - RPKNet [https://hmorimitsu.com/publication/2024-aaai-rpknet](https://hmorimitsu.com/publication/2024-aaai-rpknet)\n - SeparableFlow [https://openaccess.thecvf.com/content/ICCV2021/papers/Zhang_Separable_Flow_Learning_Motion_Cost_Volumes_for_Optical_Flow_Estimation_ICCV_2021_paper.pdf](https://openaccess.thecvf.com/content/ICCV2021/papers/Zhang_Separable_Flow_Learning_Motion_Cost_Volumes_for_Optical_Flow_Estimation_ICCV_2021_paper.pdf)\n - SKFlow [https://arxiv.org/abs/2205.14623](https://arxiv.org/abs/2205.14623)\n - VideoFlow [https://arxiv.org/abs/2303.08340](https://arxiv.org/abs/2303.08340)\n- `speed_benchmark.py` becomes `model_benchmark.py` and records more metrics\n- Fix compatibility with PyTorch 2.0\n- Fix compatibility with PyTorch Lightning 1.9\n- Fix resizing augmentation when the valid mask is sparse\n- Add support for more datasets:\n - Middlebury [https://vision.middlebury.edu/flow/](https://vision.middlebury.edu/flow/)\n - Monkaa [https://lmb.informatik.uni-freiburg.de/resources/datasets/SceneFlowDatasets.en.html](https://lmb.informatik.uni-freiburg.de/resources/datasets/SceneFlowDatasets.en.html)\n - Spring [https://spring-benchmark.org/](https://spring-benchmark.org/)\n\n## Available models\n\n- CRAFT [https://arxiv.org/abs/2203.16896](https://arxiv.org/abs/2203.16896)\n- CSFlow [https://arxiv.org/abs/2202.00909](https://arxiv.org/abs/2202.00909)\n- DICL-Flow [https://arxiv.org/abs/2010.14851](https://arxiv.org/abs/2010.14851)\n- DIP [https://arxiv.org/abs/2204.00330](https://arxiv.org/abs/2204.00330)\n- FastFlowNet [https://arxiv.org/abs/2103.04524](https://arxiv.org/abs/2103.04524)\n- Flow1D [https://arxiv.org/abs/2103.04524](https://arxiv.org/abs/2103.04524)\n- FlowFormer [https://arxiv.org/abs/2203.16194](https://arxiv.org/abs/2203.16194)\n- FlowFormer++ [https://arxiv.org/abs/2303.01237](https://arxiv.org/abs/2303.01237)\n- FlowNet [https://arxiv.org/abs/1504.06852](https://arxiv.org/abs/1504.06852)\n- FlowNet2 [https://arxiv.org/abs/1612.01925](https://arxiv.org/abs/1612.01925)\n- GMA [https://arxiv.org/abs/2104.02409](https://arxiv.org/abs/2104.02409)\n- GMFlow [https://arxiv.org/abs/2111.13680](https://arxiv.org/abs/2111.13680)\n- GMFlow+, UniMatch [https://arxiv.org/abs/2211.05783](https://arxiv.org/abs/2211.05783)\n- GMFlowNet [https://arxiv.org/abs/2203.11335](https://arxiv.org/abs/2203.11335)\n- HD3 [https://arxiv.org/abs/1812.06264](https://arxiv.org/abs/1812.06264)\n- IRR [https://arxiv.org/abs/1904.05290](https://arxiv.org/abs/1904.05290)\n- LCV [https://arxiv.org/abs/2007.11431](https://arxiv.org/abs/2007.11431)\n- LiteFlowNet [https://arxiv.org/abs/1805.07036](https://arxiv.org/abs/1805.07036)\n- LiteFlowNet2 [https://arxiv.org/abs/1903.07414](https://arxiv.org/abs/1903.07414)\n- LiteFlowNet3 [https://arxiv.org/abs/2007.09319](https://arxiv.org/abs/2007.09319)\n- MaskFlownet [https://arxiv.org/abs/2003.10955](https://arxiv.org/abs/2003.10955)\n- MatchFlow [https://arxiv.org/abs/2303.08384](https://arxiv.org/abs/2303.08384)\n- MS-RAFT+ [https://arxiv.org/abs/2210.16900](https://arxiv.org/abs/2210.16900)\n- PWCNet [https://arxiv.org/abs/1709.02371](https://arxiv.org/abs/1709.02371)\n- RAFT [https://arxiv.org/abs/2003.12039](https://arxiv.org/abs/2003.12039)\n- RPKNet [https://hmorimitsu.com/publication/2024-aaai-rpknet](https://hmorimitsu.com/publication/2024-aaai-rpknet)\n- ScopeFlow [https://arxiv.org/abs/2002.10770](https://arxiv.org/abs/2002.10770)\n- SCV [https://arxiv.org/abs/2104.02166](https://arxiv.org/abs/2104.02166)\n- SeparableFlow [https://openaccess.thecvf.com/content/ICCV2021/papers/Zhang_Separable_Flow_Learning_Motion_Cost_Volumes_for_Optical_Flow_Estimation_ICCV_2021_paper.pdf](https://openaccess.thecvf.com/content/ICCV2021/papers/Zhang_Separable_Flow_Learning_Motion_Cost_Volumes_for_Optical_Flow_Estimation_ICCV_2021_paper.pdf)\n- SKFlow [https://arxiv.org/abs/2205.14623](https://arxiv.org/abs/2205.14623)\n- STaRFlow [https://arxiv.org/abs/2007.05481](https://arxiv.org/abs/2007.05481)\n- VCN [https://papers.nips.cc/paper/2019/file/bbf94b34eb32268ada57a3be5062fe7d-Paper.pdf](https://papers.nips.cc/paper/2019/file/bbf94b34eb32268ada57a3be5062fe7d-Paper.pdf)\n- VideoFlow [https://arxiv.org/abs/2303.08340](https://arxiv.org/abs/2303.08340)\n\nRead more details about the models on [https://ptlflow.readthedocs.io/en/latest/models/models_list.html](https://ptlflow.readthedocs.io/en/latest/models/models_list.html).\n\n# Results\n\nYou can see a table with main evaluation results of the available models [here](https://ptlflow.readthedocs.io/en/latest/results/accuracy_epe.html). More results are also available in the folder [docs/source/results](docs/source/results).\n\n**Disclaimer**: These results are the ones obtained by evaluating the available models in this framework in my machine. Your results may be different due to differences in hardware and software. I also do not guarantee that the results of each model will be similar to the ones presented in the respective papers or other original sources. If you need to replicate the original results from a paper, you should use the original implementations.\n\n## Getting started\n\nPlease take a look at the [documentation](https://ptlflow.readthedocs.io/) to learn how to install and use PTLFlow.\n\nYou can also check the notebooks below running on Google Colab for some practical examples:\n\n- [Inference with a pretrained model](https://colab.research.google.com/drive/1YARBRUGplqTRnRuY9sKIs6LY_2kWAWZJ?usp=sharing).\n- [Training and using the learned weights for inference](https://colab.research.google.com/drive/1mbuAEF728_jZpFEsQHXDxjIGAcB1-nVs?usp=sharing).\n\n## Licenses\n\nThe original code of this repository is licensed under the [Apache 2.0 license](LICENSE).\n\nEach model may be subjected to different licenses. The license of each model is included in their respective folders. It is your responsibility to make sure that your project is in compliance with all the licenses and conditions involved.\n\nThe external pretrained weights all have different licenses, which are listed in their respective folders.\n\nThe pretrained weights that were trained within this project are available under the [CC BY-NC-SA 4.0 license](https://creativecommons.org/licenses/by-nc-sa/4.0/), which I believe that covers the licenses of the datasets used in the training. That being said, I am not a legal expert so if you plan to use them to any purpose other than research, you should check all the involved licenses by yourself. Additionally, the datasets used for the training usually require the user to cite the original papers, so be sure to include their respective references in your work.\n\n## Contributing\n\nContribution are welcome! Please check [CONTRIBUTING.md](CONTRIBUTING.md) to see how to contribute.\n\n## Citing\n\n### BibTeX\n\n```\n@misc{morimitsu2021ptlflow,\n author = {Henrique Morimitsu},\n title = {PyTorch Lightning Optical Flow},\n year = {2021},\n publisher = {GitHub},\n journal = {GitHub repository},\n howpublished = {\\url{https://github.com/hmorimitsu/ptlflow}}\n}\n```\n\n## Acknowledgements\n\n- This README file is heavily inspired by the one from the [timm](https://github.com/rwightman/pytorch-image-models) repository.\n- Some parts of the code were inspired by or taken from [FlowNetPytorch](https://github.com/ClementPinard/FlowNetPytorch).\n- [flownet2-pytorch](https://github.com/NVIDIA/flownet2-pytorch) was also another important source.\n- The current main training routine is based on [RAFT](https://github.com/princeton-vl/RAFT).\n",
"bugtrack_url": null,
"license": "Apache 2.0 License",
"summary": "PyTorch Lightning Optical Flow",
"version": "0.3.0",
"project_urls": {
"Documentation": "https://ptlflow.readthedocs.io/",
"Source Code": "https://github.com/hmorimitsu/ptlflow"
},
"split_keywords": [
"torch",
"pytorch lightning",
"optical flow",
"models"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e3e94457f2f4bb03e47ec9a316b89d70570d346cfb44abe248868de4faeb3b9e",
"md5": "92d7466c9e4d5c231abe751e6d7be11a",
"sha256": "b54dca038cfb76902c9c987400dca93eaabae1ed5382ecaf9cdc9df64102e0fa"
},
"downloads": -1,
"filename": "ptlflow-0.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "92d7466c9e4d5c231abe751e6d7be11a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.12,>=3.8",
"size": 732216,
"upload_time": "2024-01-13T15:46:10",
"upload_time_iso_8601": "2024-01-13T15:46:10.013995Z",
"url": "https://files.pythonhosted.org/packages/e3/e9/4457f2f4bb03e47ec9a316b89d70570d346cfb44abe248868de4faeb3b9e/ptlflow-0.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1241f641814568237064a730e70ae1fa362800308a67c87e2a252825ee759a0f",
"md5": "426d611c06b034c020e28bca44507688",
"sha256": "36a9e33d3df10f5852169a6251977f0cddc3cb94b202475ec1c69df898203ab9"
},
"downloads": -1,
"filename": "ptlflow-0.3.0.tar.gz",
"has_sig": false,
"md5_digest": "426d611c06b034c020e28bca44507688",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.12,>=3.8",
"size": 597991,
"upload_time": "2024-01-13T15:46:11",
"upload_time_iso_8601": "2024-01-13T15:46:11.922778Z",
"url": "https://files.pythonhosted.org/packages/12/41/f641814568237064a730e70ae1fa362800308a67c87e2a252825ee759a0f/ptlflow-0.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-13 15:46:11",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "hmorimitsu",
"github_project": "ptlflow",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"tox": true,
"lcname": "ptlflow"
}
JSON
