# Python Edge Evaluation Tools
Edge detection tasks heavily rely on the original codes introduced in the [BSDS300/500 benchmark](https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/resources.html) which were targeted for MATLAB users.
In the field of computer vision, various edge detection algorithms are now resorting to Python which supports various machine learning libraries such as PyTorch and Tensorflow.
However, not everyone has access to MATLAB and the original benchmark codes are outdated.
I created this open-source library, `pyEdgeEval`, to make it easier to evaluate and reproduce recent deep learning models for edge and boundary detection.
The original C++ codes used in the MATLAB benchmarks are ported with Cython and the evaluation scripts are rewritten in Python3.
This is especially useful for evaluating algorithms on remote linux servers (just run `pip install pyEdgeEval`) and docker containers, which have been difficult before due to MATLAB's constrains.
The codebase is designed to be extensible and supports various tasks and datasets as well as different evaluation protocols.
To test the validity of the evaluation code, `pyEdgeEval`'s results are compared with the results of the original MATLAB codes.
Besides benchmarking, `pyEdgeEval` adds various tools for edge detection such as `mask2edge` transformation.
`pyEdgeEval` is:
- a Python alternative to the original MATLAB benchmark
- light with minimal dependencies
- modular architecture and easily customizable
- relatively fast (uses multiprocessing and Cython)
- implements common preprocessing algorithms
- supports various tasks and datasets (extensible to other datasets)
- supports various evaluation protocols
- edge generation tools
- etc...
**Supported tasks**:
- Edge Detection
- Semantic Boundary Detection
**Supported datasets**:
- [BSDS500](https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/resources.html)
- [SBD](http://home.bharathh.info/pubs/codes/SBD/download.html)
- [Cityscapes](https://www.cityscapes-dataset.com) (semantic boundary detection)
*Disclaimers*:
- The evaluation code does not output results that exactly match the original MATLAB benchmark. This could be for various reasons such as random seeds for matching algorithm. The results are, for the most part, close enough (around 0.01% difference).
- The evaluation speed should, in theory, be around the same as the original MATLAB benchmark as we use the same C/C++ code. However, from my experience, using `pyEdgeEval` is around __3x ~ 5x__ faster than the original MATLAB benchmark (tested using 10-threads). This is a night-and-day improvement in quality-of-life ('raw' protocol evaluation for SBD on MATLAB took around 35 hours, whereas `pyEdgeEval` took around 9 hours).
- The codes and algorithms are not perfect. I will not take responsibility for how the code is used (check the license(s)).
- If you find some bugs or want to improve this project, please submit issues or pull requests.
Please cite the following if you found this project useful:
```
@misc{pyEdgeEval,
author = {Haruya Ishikawa},
title = {pyEdgeEval: Python Edge Evaluation Tools},
year = {2022},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/haruishi43/py-edge-eval}}
}
```
*Help Wanted*:
- If anyone knows a better way to package Cython codes, please let me know. I'm not a Cython expert and I'm not sure if I'm doing it right. I'm having trouble supporting various platforms since I'm only creating a wheel for Linux (`manylinux`).
- I'm aware that the documentation is a little rough. If anyone is willing to help, please let me know.
- For the other TODOs, check out [dev.md](./.readme/dev.md).
# Installation
## Dependencies
- `python >= 3.8` (tested on 3.8.x)
- `cv2`
## Installation guide
```Bash
# Install dependencies
pip install -r requirements.txt
# install cv2 (e.g. use pip)
pip install opencv-python
# Option 1. install without cloning the project (only tested on ubuntu with python 3.8)
pip install pyEdgeEval
# Option 2. install as a pip package (install as a package)
git clone https://github.com/haruishi43/py-edge-eval.git
pip install -e .
```
# Converting Cityscapes Dataset for SBD
Script:
```Bash
python scripts/convert_datasets/cityscapes.py --nproc=8
```
NOTE:
- Beaware that using multi-processing will consume at most 10GB per process (I'm working on debugging memory allocation issues).
- `--nonIS` will generate non-IS boundaries.
- The script will generate full resolution training dataset, full resolution validation dataset, and half resolution validation dataset (both raw/thin for validation).
# Evaluation for each datasets
## BSDS500
```Bash
# get complete list of options
$ python scripts/evaluate/bsds500.py -h
```
Options:
```
Evaluate BSDS output
positional arguments:
bsds_path the root path of the BSDS-500 dataset
pred_path the root path of the predictions
optional arguments:
-h, --help show this help message and exit
--output-path OUTPUT_PATH
the root path of where the results are populated
--use-val val or test
--max-dist MAX_DIST tolerance distance (default: 0.0075)
--thresholds THRESHOLDS
the number of thresholds (could be a
list of floats); use 99 for eval
--raw option to remove the thinning process (i.e. uses raw predition)
--apply-nms applies NMS before evaluation
--nproc NPROC the number of parallel threads
```
```Bash
# Example:
python scripts/evaluate/bsds500.py <path/to/bsds500> <path/to/pred> <path/to/output> \
--thresholds=5 --nproc=8
```
- Tested with [@xwjabc's HED implementation](https://github.com/xwjabc/hed).
- Due to the randomness in the original MATLAB (C++) codebase, the results will be different (at most +-0.001 difference).
- Setting `--nproc` to the number of available cores will drastically improve the speed of evaluation.
## SBD
First, you need to generate the GT using the `scripts/convert_dataset/sbd.py` script.
Example evaluation commands:
```Bash
python scripts/evaluate/sbd.py <path/to/sbd> <path/to/pred> --categories=15 --thresholds=5 --nproc=8
```
## Cityscapes
First, create GT data using this script:
```Bash
# if you plan on evaluating with instance-sensitive edges (IS edges)
python scripts/convert_dataset/cityscapes.py --nproc 8
# if you plan on evaluating with non-instance-sensitive edges
python scripts/convert_dataset/cityscapes.py --nonIS --nproc 8
```
The scripts will create two types of edges (raw and thin) for two different scales (half and full).
Evaluation script:
```Bash
# thin protocol
python scripts/evaluate/cityscapes_thin.py <path/to/cityscapes> <path/to/predictions> <path/to/output> --categories='[1, 14]' --thresholds 99 --nproc 8
# raw protocol
python scripts/evaluate/cityscapes_raw.py <path/to/cityscapes> <path/to/predictions> <path/to/output> --categories='[1, 14]' --thresholds 99 --nproc 8
```
- For instance-insensitive edges, you would need to supply `--pre-seal` argument.
- You can also preprocess the predictions by passing `--apply-thinning` and/or `--apply-nms` for thinning and NMS respectively.
# License
- The code is released under the MIT License (please refer to the LICENSE file for details).
- I modified codes from other projects and their licenses applies to those files (please refer to [Licenses](./LICENSES.md)).
# Development and Contribution
See [dev.md](./.readme/dev.md).
Raw data
{
"_id": null,
"home_page": "https://github.com/haruishi43/py-edge-eval",
"name": "pyEdgeEval",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": null,
"keywords": "Edge Detection, Semantic Boundary Detection, Computer Vision",
"author": "haruishi43",
"author_email": "haruyaishikawa@keio.jp",
"download_url": "https://files.pythonhosted.org/packages/77/b9/a7294370ebccd4b9b2fc3f17f8e3578a1e11da7188f8bcd23ab139f27e50/pyedgeeval-0.2.8.tar.gz",
"platform": null,
"description": "# Python Edge Evaluation Tools\n\nEdge detection tasks heavily rely on the original codes introduced in the [BSDS300/500 benchmark](https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/resources.html) which were targeted for MATLAB users.\nIn the field of computer vision, various edge detection algorithms are now resorting to Python which supports various machine learning libraries such as PyTorch and Tensorflow.\nHowever, not everyone has access to MATLAB and the original benchmark codes are outdated.\nI created this open-source library, `pyEdgeEval`, to make it easier to evaluate and reproduce recent deep learning models for edge and boundary detection.\nThe original C++ codes used in the MATLAB benchmarks are ported with Cython and the evaluation scripts are rewritten in Python3.\nThis is especially useful for evaluating algorithms on remote linux servers (just run `pip install pyEdgeEval`) and docker containers, which have been difficult before due to MATLAB's constrains.\nThe codebase is designed to be extensible and supports various tasks and datasets as well as different evaluation protocols.\nTo test the validity of the evaluation code, `pyEdgeEval`'s results are compared with the results of the original MATLAB codes.\nBesides benchmarking, `pyEdgeEval` adds various tools for edge detection such as `mask2edge` transformation.\n\n`pyEdgeEval` is:\n- a Python alternative to the original MATLAB benchmark\n- light with minimal dependencies\n- modular architecture and easily customizable\n- relatively fast (uses multiprocessing and Cython)\n- implements common preprocessing algorithms\n- supports various tasks and datasets (extensible to other datasets)\n- supports various evaluation protocols\n- edge generation tools\n- etc...\n\n**Supported tasks**:\n- Edge Detection\n- Semantic Boundary Detection\n\n**Supported datasets**:\n- [BSDS500](https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/resources.html)\n- [SBD](http://home.bharathh.info/pubs/codes/SBD/download.html)\n- [Cityscapes](https://www.cityscapes-dataset.com) (semantic boundary detection)\n\n*Disclaimers*:\n- The evaluation code does not output results that exactly match the original MATLAB benchmark. This could be for various reasons such as random seeds for matching algorithm. The results are, for the most part, close enough (around 0.01% difference).\n- The evaluation speed should, in theory, be around the same as the original MATLAB benchmark as we use the same C/C++ code. However, from my experience, using `pyEdgeEval` is around __3x ~ 5x__ faster than the original MATLAB benchmark (tested using 10-threads). This is a night-and-day improvement in quality-of-life ('raw' protocol evaluation for SBD on MATLAB took around 35 hours, whereas `pyEdgeEval` took around 9 hours).\n- The codes and algorithms are not perfect. I will not take responsibility for how the code is used (check the license(s)).\n- If you find some bugs or want to improve this project, please submit issues or pull requests.\n\nPlease cite the following if you found this project useful:\n```\n@misc{pyEdgeEval,\n author = {Haruya Ishikawa},\n title = {pyEdgeEval: Python Edge Evaluation Tools},\n year = {2022},\n publisher = {GitHub},\n journal = {GitHub repository},\n howpublished = {\\url{https://github.com/haruishi43/py-edge-eval}}\n}\n```\n\n*Help Wanted*:\n- If anyone knows a better way to package Cython codes, please let me know. I'm not a Cython expert and I'm not sure if I'm doing it right. I'm having trouble supporting various platforms since I'm only creating a wheel for Linux (`manylinux`).\n- I'm aware that the documentation is a little rough. If anyone is willing to help, please let me know.\n- For the other TODOs, check out [dev.md](./.readme/dev.md).\n\n# Installation\n\n## Dependencies\n\n- `python >= 3.8` (tested on 3.8.x)\n- `cv2`\n\n## Installation guide\n\n```Bash\n# Install dependencies\npip install -r requirements.txt\n\n# install cv2 (e.g. use pip)\npip install opencv-python\n\n# Option 1. install without cloning the project (only tested on ubuntu with python 3.8)\npip install pyEdgeEval\n\n# Option 2. install as a pip package (install as a package)\ngit clone https://github.com/haruishi43/py-edge-eval.git\npip install -e .\n```\n\n# Converting Cityscapes Dataset for SBD\n\nScript:\n\n```Bash\npython scripts/convert_datasets/cityscapes.py --nproc=8\n```\n\nNOTE:\n- Beaware that using multi-processing will consume at most 10GB per process (I'm working on debugging memory allocation issues).\n- `--nonIS` will generate non-IS boundaries.\n- The script will generate full resolution training dataset, full resolution validation dataset, and half resolution validation dataset (both raw/thin for validation).\n\n# Evaluation for each datasets\n\n## BSDS500\n\n```Bash\n# get complete list of options\n$ python scripts/evaluate/bsds500.py -h\n```\n\nOptions:\n```\nEvaluate BSDS output\n\npositional arguments:\n bsds_path the root path of the BSDS-500 dataset\n pred_path the root path of the predictions\n\noptional arguments:\n -h, --help show this help message and exit\n --output-path OUTPUT_PATH\n the root path of where the results are populated\n --use-val val or test\n --max-dist MAX_DIST tolerance distance (default: 0.0075)\n --thresholds THRESHOLDS\n the number of thresholds (could be a\n list of floats); use 99 for eval\n --raw option to remove the thinning process (i.e. uses raw predition)\n --apply-nms applies NMS before evaluation\n --nproc NPROC the number of parallel threads\n```\n\n```Bash\n# Example:\npython scripts/evaluate/bsds500.py <path/to/bsds500> <path/to/pred> <path/to/output> \\\n --thresholds=5 --nproc=8\n```\n\n- Tested with [@xwjabc's HED implementation](https://github.com/xwjabc/hed).\n- Due to the randomness in the original MATLAB (C++) codebase, the results will be different (at most +-0.001 difference).\n- Setting `--nproc` to the number of available cores will drastically improve the speed of evaluation.\n\n\n## SBD\n\nFirst, you need to generate the GT using the `scripts/convert_dataset/sbd.py` script.\n\nExample evaluation commands:\n```Bash\npython scripts/evaluate/sbd.py <path/to/sbd> <path/to/pred> --categories=15 --thresholds=5 --nproc=8\n```\n\n## Cityscapes\n\nFirst, create GT data using this script:\n```Bash\n# if you plan on evaluating with instance-sensitive edges (IS edges)\npython scripts/convert_dataset/cityscapes.py --nproc 8\n# if you plan on evaluating with non-instance-sensitive edges\npython scripts/convert_dataset/cityscapes.py --nonIS --nproc 8\n```\nThe scripts will create two types of edges (raw and thin) for two different scales (half and full).\n\nEvaluation script:\n```Bash\n# thin protocol\npython scripts/evaluate/cityscapes_thin.py <path/to/cityscapes> <path/to/predictions> <path/to/output> --categories='[1, 14]' --thresholds 99 --nproc 8\n\n# raw protocol\npython scripts/evaluate/cityscapes_raw.py <path/to/cityscapes> <path/to/predictions> <path/to/output> --categories='[1, 14]' --thresholds 99 --nproc 8\n```\n\n- For instance-insensitive edges, you would need to supply `--pre-seal` argument.\n- You can also preprocess the predictions by passing `--apply-thinning` and/or `--apply-nms` for thinning and NMS respectively.\n\n\n# License\n\n- The code is released under the MIT License (please refer to the LICENSE file for details).\n- I modified codes from other projects and their licenses applies to those files (please refer to [Licenses](./LICENSES.md)).\n\n# Development and Contribution\n\nSee [dev.md](./.readme/dev.md).\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Python Edge Evaluation Tools",
"version": "0.2.8",
"project_urls": {
"Homepage": "https://github.com/haruishi43/py-edge-eval"
},
"split_keywords": [
"edge detection",
" semantic boundary detection",
" computer vision"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ea848a26ca20185b496fb65487e642c773ccad7592117a82e9a5e7f1de66eac7",
"md5": "dfe605136ec3dccfb36212d4a98f1693",
"sha256": "14fd0fe2a7ab698083f0a74d9b85d9cae4f07070ba8602f555981bb886c54f82"
},
"downloads": -1,
"filename": "pyEdgeEval-0.2.8-cp38-cp38-manylinux_2_31_x86_64.whl",
"has_sig": false,
"md5_digest": "dfe605136ec3dccfb36212d4a98f1693",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.7",
"size": 1860190,
"upload_time": "2024-06-19T01:38:53",
"upload_time_iso_8601": "2024-06-19T01:38:53.864137Z",
"url": "https://files.pythonhosted.org/packages/ea/84/8a26ca20185b496fb65487e642c773ccad7592117a82e9a5e7f1de66eac7/pyEdgeEval-0.2.8-cp38-cp38-manylinux_2_31_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "77b9a7294370ebccd4b9b2fc3f17f8e3578a1e11da7188f8bcd23ab139f27e50",
"md5": "aab86f65083f504ff0873804c0b36335",
"sha256": "dd94dbef7acd499c7266704fa34c01a1fcecab38935253db1b3748849890403c"
},
"downloads": -1,
"filename": "pyedgeeval-0.2.8.tar.gz",
"has_sig": false,
"md5_digest": "aab86f65083f504ff0873804c0b36335",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 386244,
"upload_time": "2024-06-19T01:38:58",
"upload_time_iso_8601": "2024-06-19T01:38:58.488461Z",
"url": "https://files.pythonhosted.org/packages/77/b9/a7294370ebccd4b9b2fc3f17f8e3578a1e11da7188f8bcd23ab139f27e50/pyedgeeval-0.2.8.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-06-19 01:38:58",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "haruishi43",
"github_project": "py-edge-eval",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "cityscapesscripts",
"specs": []
},
{
"name": "matplotlib",
"specs": []
},
{
"name": "mat73",
"specs": []
},
{
"name": "numpy",
"specs": []
},
{
"name": "scipy",
"specs": []
},
{
"name": "prettytable",
"specs": []
},
{
"name": "pymatreader",
"specs": []
}
],
"lcname": "pyedgeeval"
}
JSON
