<h1 align="center">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="./assets/bench-logo-dark.png">
<source media="(prefers-color-scheme: light)" srcset="./assets/bench-logo-light.png">
<img alt="EvoXBench Logo" height="48em" src="./assets/bench-logo-light.png">
</picture>
<br>
Neural Architecture Search as Multiobjective Optimization Benchmarks: Problem Formulation and Performance Assessment <a href=https://arxiv.org/abs/2208.04321>arXiv</a>
</h1>
---
EvoXBench is a platfrom offering **instant benchmarking** of evolutionary multi-objective optimization (EMO) algorithms in neural architecture search (NAS), with ready to use test suites. It facilitates efficient performance assessments **with NO requirement of GPUs or PyTorch/TensorFlow**, enhancing accessibility for a broader range of research applications. It encompasses extensive test suites that cover a variety of datasets (CIFAR10, ImageNet, Cityscapes, etc.), search spaces (NASBench101, NASBench201, NATS, DARTS, ResNet50, Transformer, MNV3, MoSegNAS, etc.), and hardware devices (Eyeriss, GPUs, Samsung Note10, etc.). It provides a versatile interface **compatible with multiple programming languages** (Java, Matlab, Python, etc.).
---
## 📢 Latest News & Updates
- EvoXBench has been updated to version **1.0.4**! This latest release adds new test suite **CitySeg/MOP**.
If you're already onboard with EvoXBench, give this command a spin: `pip install evoxbench==1.0.4`.
## ⭐️ Key Features
### 📐 General NAS Problem Formulation
- Formulating NAS tasks into general multi-objective optimization problems.
- Exploring NAS's nuanced traits through the prism of evolutionary optimization.
### 🛠️ Efficient Benchmarking Pipeline
- Presenting an end-to-end worflow for instant benchmark assessments of EMO algorithms.
- Providing instant fitness evaluations as numerical optimization.
### 📊 Comprehensive Test Suites
- Encompassing a wide spectrum of datasets, search spaces, and hardware devices.
- Ready-to-use test multi-objective optimization suites with up to eight objectives.
## Get Started
<p align="center">
<a href="https://www.emigroup.tech/wp-content/uploads/2023/02/tutorial.mp4">
<img src="https://github.com/EMI-Group/evoxbench/blob/main/assets/video%20cover.png" alt="Dive into the tutorial" width="450"/>
</a>
<br>
<small>Tap the image to embark on the introductory video voyage.</small>
</p>
## Setup & Installation
1. Download requisite files:
- ``database.zip``
via [Google Drive](https://drive.google.com/file/d/11bQ1paHEWHDnnTPtxs2OyVY_Re-38DiO/view?usp=sharing)
or [Baidu云盘（提取码：mhgs）](https://pan.baidu.com/s/1PwWloA543-81O-GFkA7GKg)
- ``data.zip``
via [Google Drive](https://drive.google.com/file/d/1r0iSCq1gLFs5xnmp1MDiqcqxNcY5q6Hp/view?usp=sharing)
or [Baidu云盘（提取码：9z45）](https://pan.baidu.com/s/17dUpiIosSCZoSgKXwSBlVg)
2. Run `pip install evoxbench` to get the benchmark.
3. Configure the benchmark:
```python
from evoxbench.database.init import config
config("Path to database", "Path to data")
# For instance:
# With this structure:
# /home/Downloads/
# └─ database/
# | | __init__.py
# | | db.sqlite3
# | | ...
# |
# └─ data/
# └─ darts/
# └─ mnv3/
# └─ ...
# Then, execute:
# config("/home/Downloads/database", "/home/Downloads/data")
```
## About the Database
Explore our comprehensive database and understand its structure and content. Check it out [here](https://github.com/liuxukun2000/evoxdatabase).
## Community & Support
- Use the issue tracker for bugs or questions.
- Submit your enhancements through a pull request (PR).
- We have an active QQ group (ID: 297969717).
- Official Website: https://evox.group/
## Sister Projects
- EvoX: A computing framework for distributed GPU-aceleration of evolutionary computation, supporting a wide spectrum of evolutionary algorithms and test problems. Check out [here](https://github.com/EMI-Group/evox).
## Citing EvoXBench
If you use EvoXBench in your research and want to cite it in your work, please use:
```
@article{EvoXBench,
title={Neural Architecture Search as Multiobjective Optimization Benchmarks: Problem Formulation and Performance Assessment},
author={Lu, Zhichao and Cheng, Ran and Jin, Yaochu and Tan, Kay Chen and Deb, Kalyanmoy},
journal={IEEE Transactions on Evolutionary Computation},
year={2023},
publisher={IEEE}
}
```
## Acknowledgements
A big shoutout to the following projects that have made EvoXBench possible:
[NAS-Bench-101](https://github.com/google-research/nasbench),
[NAS-Bench-201](https://github.com/D-X-Y/NAS-Bench-201),
[NAS-Bench-301](https://github.com/automl/nasbench301),
[NATS-Bench](https://xuanyidong.com/assets/projects/NATS-Bench),
[Once for All](https://github.com/mit-han-lab/once-for-all),
[AutoFormer](https://github.com/microsoft/Cream/tree/main/AutoFormer),
[Django](https://www.djangoproject.com/),
[pymoo](https://pymoo.org/).
Raw data
{
"_id": null,
"home_page": "https://github.com/EMI-Group/evoxbench",
"name": "evoxbench",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7, <4",
"maintainer_email": "",
"keywords": "benchmark,evolution algorithm,neural architecture search",
"author": "EMI-Group",
"author_email": "emi-group@outlook.com",
"download_url": "https://files.pythonhosted.org/packages/9b/d5/013ac901fa57dbe4bf8ff1e41f6a1647a35fad3d470105e21e4c2e9b8bb2/evoxbench-1.0.4.tar.gz",
"platform": null,
"description": "<h1 align=\"center\">\r\n <picture>\r\n <source media=\"(prefers-color-scheme: dark)\" srcset=\"./assets/bench-logo-dark.png\">\r\n <source media=\"(prefers-color-scheme: light)\" srcset=\"./assets/bench-logo-light.png\">\r\n <img alt=\"EvoXBench Logo\" height=\"48em\" src=\"./assets/bench-logo-light.png\">\r\n </picture>\r\n <br>\r\n Neural Architecture Search as Multiobjective Optimization Benchmarks: Problem Formulation and Performance Assessment <a href=https://arxiv.org/abs/2208.04321>arXiv</a>\r\n</h1>\r\n\r\n---\r\n\r\nEvoXBench is a platfrom offering **instant benchmarking** of evolutionary multi-objective optimization (EMO) algorithms in neural architecture search (NAS), with ready to use test suites. It facilitates efficient performance assessments **with NO requirement of GPUs or PyTorch/TensorFlow**, enhancing accessibility for a broader range of research applications. It encompasses extensive test suites that cover a variety of datasets (CIFAR10, ImageNet, Cityscapes, etc.), search spaces (NASBench101, NASBench201, NATS, DARTS, ResNet50, Transformer, MNV3, MoSegNAS, etc.), and hardware devices (Eyeriss, GPUs, Samsung Note10, etc.). It provides a versatile interface **compatible with multiple programming languages** (Java, Matlab, Python, etc.).\r\n\r\n---\r\n\r\n\r\n## \ud83d\udce2 Latest News & Updates\r\n\r\n- EvoXBench has been updated to version **1.0.4**! This latest release adds new test suite **CitySeg/MOP**.\r\n\r\n If you're already onboard with EvoXBench, give this command a spin: `pip install evoxbench==1.0.4`.\r\n\r\n\r\n\r\n## \u2b50\ufe0f Key Features\r\n\r\n### \ud83d\udcd0 General NAS Problem Formulation\r\n- Formulating NAS tasks into general multi-objective optimization problems.\r\n- Exploring NAS's nuanced traits through the prism of evolutionary optimization.\r\n\r\n### \ud83d\udee0\ufe0f Efficient Benchmarking Pipeline\r\n- Presenting an end-to-end worflow for instant benchmark assessments of EMO algorithms.\r\n- Providing instant fitness evaluations as numerical optimization.\r\n\r\n### \ud83d\udcca Comprehensive Test Suites\r\n- Encompassing a wide spectrum of datasets, search spaces, and hardware devices.\r\n- Ready-to-use test multi-objective optimization suites with up to eight objectives.\r\n\r\n\r\n## Get Started\r\n\r\n<p align=\"center\">\r\n <a href=\"https://www.emigroup.tech/wp-content/uploads/2023/02/tutorial.mp4\">\r\n <img src=\"https://github.com/EMI-Group/evoxbench/blob/main/assets/video%20cover.png\" alt=\"Dive into the tutorial\" width=\"450\"/>\r\n </a>\r\n <br>\r\n <small>Tap the image to embark on the introductory video voyage.</small>\r\n</p>\r\n\r\n\r\n\r\n## Setup & Installation\r\n\r\n1. Download requisite files:\r\n - ``database.zip``\r\n via [Google Drive](https://drive.google.com/file/d/11bQ1paHEWHDnnTPtxs2OyVY_Re-38DiO/view?usp=sharing)\r\n or [Baidu\u4e91\u76d8\uff08\u63d0\u53d6\u7801\uff1amhgs\uff09](https://pan.baidu.com/s/1PwWloA543-81O-GFkA7GKg)\r\n\r\n - ``data.zip``\r\n via [Google Drive](https://drive.google.com/file/d/1r0iSCq1gLFs5xnmp1MDiqcqxNcY5q6Hp/view?usp=sharing)\r\n or [Baidu\u4e91\u76d8\uff08\u63d0\u53d6\u7801\uff1a9z45\uff09](https://pan.baidu.com/s/17dUpiIosSCZoSgKXwSBlVg)\r\n\r\n2. Run `pip install evoxbench` to get the benchmark.\r\n\r\n3. Configure the benchmark:\r\n\r\n```python\r\n from evoxbench.database.init import config\r\n\r\n config(\"Path to database\", \"Path to data\")\r\n # For instance:\r\n # With this structure:\r\n # /home/Downloads/\r\n # \u2514\u2500 database/\r\n # | | __init__.py\r\n # | | db.sqlite3\r\n # | | ...\r\n # |\r\n # \u2514\u2500 data/\r\n # \u2514\u2500 darts/\r\n # \u2514\u2500 mnv3/\r\n # \u2514\u2500 ...\r\n # Then, execute:\r\n # config(\"/home/Downloads/database\", \"/home/Downloads/data\")\r\n```\r\n\r\n## About the Database\r\n\r\nExplore our comprehensive database and understand its structure and content. Check it out [here](https://github.com/liuxukun2000/evoxdatabase).\r\n\r\n## Community & Support\r\n\r\n- Use the issue tracker for bugs or questions.\r\n- Submit your enhancements through a pull request (PR).\r\n- We have an active QQ group (ID: 297969717).\r\n- Official Website: https://evox.group/\r\n\r\n## Sister Projects\r\n- EvoX: A computing framework for distributed GPU-aceleration of evolutionary computation, supporting a wide spectrum of evolutionary algorithms and test problems. Check out [here](https://github.com/EMI-Group/evox).\r\n\r\n\r\n## Citing EvoXBench\r\n\r\nIf you use EvoXBench in your research and want to cite it in your work, please use:\r\n```\r\n@article{EvoXBench,\r\n title={Neural Architecture Search as Multiobjective Optimization Benchmarks: Problem Formulation and Performance Assessment},\r\n author={Lu, Zhichao and Cheng, Ran and Jin, Yaochu and Tan, Kay Chen and Deb, Kalyanmoy},\r\n journal={IEEE Transactions on Evolutionary Computation},\r\n year={2023},\r\n publisher={IEEE}\r\n}\r\n```\r\n\r\n## Acknowledgements\r\n\r\nA big shoutout to the following projects that have made EvoXBench possible:\r\n\r\n [NAS-Bench-101](https://github.com/google-research/nasbench),\r\n [NAS-Bench-201](https://github.com/D-X-Y/NAS-Bench-201),\r\n [NAS-Bench-301](https://github.com/automl/nasbench301),\r\n [NATS-Bench](https://xuanyidong.com/assets/projects/NATS-Bench),\r\n [Once for All](https://github.com/mit-han-lab/once-for-all),\r\n [AutoFormer](https://github.com/microsoft/Cream/tree/main/AutoFormer),\r\n [Django](https://www.djangoproject.com/),\r\n [pymoo](https://pymoo.org/).\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n",
"bugtrack_url": null,
"license": "",
"summary": "A benchmark for NAS algorithms",
"version": "1.0.4",
"project_urls": {
"Bug Reports": "https://github.com/EMI-Group/evoxbench/issues",
"Homepage": "https://github.com/EMI-Group/evoxbench",
"Source": "https://github.com/EMI-Group/evoxbench"
},
"split_keywords": [
"benchmark",
"evolution algorithm",
"neural architecture search"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "04eef4eba5afc629b761e1a6b9aa4092802d21373319dbfc9e1fa66061d55f6f",
"md5": "4953a493ae3a186be09961e749c888e8",
"sha256": "99784ffc35d9cf21d7ebb43be21b200fd692b513edadf7c9c2e488119af1f08f"
},
"downloads": -1,
"filename": "evoxbench-1.0.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "4953a493ae3a186be09961e749c888e8",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7, <4",
"size": 183320,
"upload_time": "2024-02-29T15:47:31",
"upload_time_iso_8601": "2024-02-29T15:47:31.830773Z",
"url": "https://files.pythonhosted.org/packages/04/ee/f4eba5afc629b761e1a6b9aa4092802d21373319dbfc9e1fa66061d55f6f/evoxbench-1.0.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9bd5013ac901fa57dbe4bf8ff1e41f6a1647a35fad3d470105e21e4c2e9b8bb2",
"md5": "a0e26b0bd71141d023c1bdeeba8c3c0d",
"sha256": "ea6dbe1b17a01510e951e31fdd9dcd04a8f5e7655e2b6686b9dd969d7126f089"
},
"downloads": -1,
"filename": "evoxbench-1.0.4.tar.gz",
"has_sig": false,
"md5_digest": "a0e26b0bd71141d023c1bdeeba8c3c0d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7, <4",
"size": 102644,
"upload_time": "2024-02-29T15:47:33",
"upload_time_iso_8601": "2024-02-29T15:47:33.455959Z",
"url": "https://files.pythonhosted.org/packages/9b/d5/013ac901fa57dbe4bf8ff1e41f6a1647a35fad3d470105e21e4c2e9b8bb2/evoxbench-1.0.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-29 15:47:33",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "EMI-Group",
"github_project": "evoxbench",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "evoxbench"
}
JSON
