| Name | iohblade JSON |
| Version |
0.2.1
JSON |
| download |
| home_page | None |
| Summary | Benchmarking Llm Assisted Design and Evolution of algorithms. |
| upload_time | 2025-07-10 12:50:54 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | <4,>=3.11 |
| license | None |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
|
<p align="center">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="logo-dark.png">
<source media="(prefers-color-scheme: light)" srcset="logo.png">
<img alt="Shows the BLADE logo." src="logo.png" width="200px">
</picture>
</p>
<h1 align="center">IOH-BLADE: Benchmarking LLM-driven Automated Design and Evolution of Iterative Optimization Heuristics</h1>
<p align="center">
<a href="https://pypi.org/project/iohblade/">
<img src="https://badge.fury.io/py/iohblade.svg" alt="PyPI version" height="18">
</a>
<img src="https://img.shields.io/badge/Maintained%3F-yes-brightgreen.svg" alt="Maintenance" height="18">
<img src="https://img.shields.io/badge/Python-3.11+-blue" alt="Python 3.11+" height="18">
<a href="https://codecov.io/gh/XAI-liacs/BLADE" >
<img src="https://codecov.io/gh/XAI-liacs/BLADE/graph/badge.svg?token=ZOT67R1TP7" alt="CodeCov" height="18"/>
</a>
</p>
> [!TIP]
> See also the [Documentation](https://xai-liacs.github.io/BLADE/).
## Table of Contents
- [News](#-news)
- [Introduction](#introduction)
- [Installation](#-installation)
- [Quick Start](#-quick-start)
- [Webapp](#-webapp)
- [Contributing](#-contributing)
- [License](#-license)
- [Citation](#-citation)
## 🔥 News
- 2025.03 ✨✨ **BLADE v0.0.1 released**!
## Introduction
**BLADE** (Benchmark suite for LLM-driven Automated Design and Evolution) provides a standardized benchmark suite for evaluating automatic algorithm design algorithms, particularly those generating metaheuristics by large language models (LLMs). It focuses on **continuous black-box optimization** and integrates a diverse set of **problems** and **methods**, facilitating fair and comprehensive benchmarking.
### Features
- **Comprehensive Benchmark Suite:** Covers various classes of black-box optimization problems.
- **LLM-Driven Evaluation:** Supports algorithm evolution and design using large language models.
- **Built-In Baselines:** Includes state-of-the-art metaheuristics for comparison.
- **Automatic Logging & Visualization:** Integrated with **IOHprofiler** for performance tracking.
#### Included Benchmark Function Sets
BLADE incorporates several benchmark function sets to provide a comprehensive evaluation environment:
| Name | Short Description | Number of Functions | Multiple Instances |
|------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------|---------------------|--------------------|
| **BBOB** (Black-Box Optimization Benchmarking) | A suite of 24 noiseless functions designed for benchmarking continuous optimization algorithms. [Reference](https://arxiv.org/pdf/1903.06396) | 24 | Yes |
| **SBOX-COST** | A set of 24 boundary-constrained functions focusing on strict box-constraint optimization scenarios. [Reference](https://inria.hal.science/hal-04403658/file/sboxcost-cmacomparison-authorversion.pdf) | 24 | Yes |
| **MA-BBOB** (Many-Affine BBOB) | An extension of the BBOB suite, generating functions through affine combinations and shifts. [Reference](https://dl.acm.org/doi/10.1145/3673908) | Generator-Based | Yes |
| **GECCO MA-BBOB Competition Instances** | A collection of 1,000 pre-defined instances from the GECCO MA-BBOB competition, evaluating algorithm performance on diverse affine-combined functions. [Reference](https://iohprofiler.github.io/competitions) | 1,000 | Yes |
In addition, several real-world applications are included such as several photonics problems.
### Included Search Methods
The suite contains the state-of-the-art LLM-assisted search algorithms:
| Algorithm | Description | Link
|--------------------------|-------------------------------------------------|--------------|
| **LLaMEA** | Large Langugage Model Evolutionary Algorithm | [code](https://github.com/nikivanstein/LLaMEA) [paper](https://arxiv.org/abs/2405.20132) |
| **EoH** | Evolution of Heuristics | [code](https://github.com/FeiLiu36/EoH) [paper](https://arxiv.org/abs/2401.02051) |
| **FunSearch** | Google's GA-like algorithm | [code](https://github.com/google-deepmind/funsearch) [paper](https://www.nature.com/articles/s41586-023-06924-6) |
| **ReEvo** | Large Language Models as Hyper-Heuristics with Reflective Evolution | [code](https://github.com/ai4co/LLM-as-HH) [paper](https://arxiv.org/abs/2402.01145) |
> Note, FunSearch is currently not yet integrated.
### Supported LLM APIs
BLADE supports integration with various LLM APIs to facilitate automated design of algorithms:
| LLM Provider | Description | Integration Notes |
|--------------|---------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------|
| **Gemini** | Google's multimodal LLM designed to process text, images, audio, and more. [Reference](https://en.wikipedia.org/wiki/Gemini_%28language_model%29) | Accessible via the Gemini API, compatible with OpenAI libraries. [Reference](https://ai.google.dev/gemini-api/docs/openai) |
| **OpenAI** | Developer of GPT series models, including GPT-4, widely used for natural language understanding and generation. [Reference](https://openai.com/) | Integration through OpenAI's REST API and client libraries. |
| **Ollama** | A platform offering access to various LLMs, enabling local and cloud-based model deployment. [Reference](https://www.ollama.ai/) | Integration details can be found in their official documentation. |
### Evaluating against Human Designed baselines
An important part of BLADE is the final evaluation of generated algorithms against state-of-the-art human designed algorithms.
In the `iohblade.baselines` part of the package, several well known SOTA black-box optimizers are imolemented to compare against.
Including but not limited to CMA-ES and DE variants.
For the final validation **BLADE** uses [**IOHprofiler**](https://iohprofiler.github.io/), providing detailed tracking and visualization of performance metrics.
## 🎁 Installation
It is the easiest to use BLADE from the pypi package (`iohblade`).
```bash
pip install iohblade
```
> [!Important]
> The Python version **must** be larger or equal to Python 3.11.
> You need an OpenAI/Gemini/Ollama API key for using LLM models.
You can also install the package from source using <a href="https://docs.astral.sh/uv/" target="_blank">uv</a> (0.7.19).
make sure you have `uv` installed.
1. Clone the repository:
```bash
git clone https://github.com/XAI-liacs/BLADE.git
cd BLADE
```
2. Install the required dependencies via uv:
```bash
uv sync
```
3. *(Optional)* Install additional packages:
```bash
uv sync --group kerneltuner --group dev --group docs
```
This will install additional dependencies for development and building documentation.
The (experimental) auto-kernel application is also under a separate group for now.
## 💻 Quick Start
1. Set up an OpenAI API key:
- Obtain an API key from [OpenAI](https://openai.com/) or Gemini or another LLM provider.
- Set the API key in your environment variables:
```bash
export OPENAI_API_KEY='your_api_key_here'
```
2. Running an Experiment
To run a benchmarking experiment using BLADE:
```python
from iohblade import Experiment
from iohblade import Ollama_LLM
from iohblade.methods import LLaMEA, RandomSearch
from iohblade.problems import BBOB_SBOX
from iohblade.loggers import ExperimentLogger
import os
llm = Ollama_LLM("qwen2.5-coder:14b") #qwen2.5-coder:14b, deepseek-coder-v2:16b
budget = 50 #short budget for testing
RS = RandomSearch(llm, budget=budget) #Random Search baseline
LLaMEA_method = LLaMEA(llm, budget=budget, name="LLaMEA", n_parents=4, n_offspring=12, elitism=False) #LLamEA with 4,12 strategy
methods = [RS, LLaMEA_method]
problems = []
# include all SBOX_COST functions with 5 instances for training and 10 for final validation as the benchmark problem.
training_instances = [(f, i) for f in range(1,25) for i in range(1, 6)]
test_instances = [(f, i) for f in range(1,25) for i in range(5, 16)]
problems.append(BBOB_SBOX(training_instances=training_instances, test_instances=test_instances, dims=[5], budget_factor=2000, name=f"SBOX_COST"))
# Set up the experiment object with 5 independent runs per method/problem. (in this case 1 problem)
logger = ExperimentLogger("results/SBOX")
experiment = Experiment(methods=methods, problems=problems, runs=5, show_stdout=True, exp_logger=logger) #normal run
experiment() #run the experiment, all data is logged in the folder results/SBOX/
```
## 🌐 Webapp
After running experiments you can browse them using the built-in Streamlit app:
```bash
uv run iohblade-webapp
```
The app lists available experiments from the `results` directory, displays their progress, and shows convergence plots.
---
## 💻 Examples
See the files in the `examples` folder for examples on experiments and visualisations.
---
## 🤖 Contributing
Contributions to BLADE are welcome! Here are a few ways you can help:
- **Report Bugs**: Use [GitHub Issues](https://github.com/XAI-Liacs/BLADE/issues) to report bugs.
- **Feature Requests**: Suggest new features or improvements.
- **Pull Requests**: Submit PRs for bug fixes or feature additions.
Please refer to CONTRIBUTING.md for more details on contributing guidelines.
## 🪪 License
Distributed under the [MIT](https://choosealicense.com/licenses/mit/) License. See `LICENSE` for more information.
## ✨ Citation
TBA
-----
Happy Benchmarking with IOH-BLADE! 🚀
Raw data
{
"_id": null,
"home_page": null,
"name": "iohblade",
"maintainer": null,
"docs_url": null,
"requires_python": "<4,>=3.11",
"maintainer_email": null,
"keywords": null,
"author": null,
"author_email": "Niki van Stein <n.van.stein@liacs.leidenuniv.nl>",
"download_url": "https://files.pythonhosted.org/packages/fc/48/3f8c21c477fc45387959c44f97fa4d2ea27991495e8fc0c650735d147049/iohblade-0.2.1.tar.gz",
"platform": null,
"description": "<p align=\"center\">\n <picture>\n <source media=\"(prefers-color-scheme: dark)\" srcset=\"logo-dark.png\">\n <source media=\"(prefers-color-scheme: light)\" srcset=\"logo.png\">\n <img alt=\"Shows the BLADE logo.\" src=\"logo.png\" width=\"200px\">\n </picture>\n</p>\n\n<h1 align=\"center\">IOH-BLADE: Benchmarking LLM-driven Automated Design and Evolution of Iterative Optimization Heuristics</h1>\n\n<p align=\"center\">\n <a href=\"https://pypi.org/project/iohblade/\">\n <img src=\"https://badge.fury.io/py/iohblade.svg\" alt=\"PyPI version\" height=\"18\">\n </a>\n <img src=\"https://img.shields.io/badge/Maintained%3F-yes-brightgreen.svg\" alt=\"Maintenance\" height=\"18\">\n <img src=\"https://img.shields.io/badge/Python-3.11+-blue\" alt=\"Python 3.11+\" height=\"18\">\n <a href=\"https://codecov.io/gh/XAI-liacs/BLADE\" > \n <img src=\"https://codecov.io/gh/XAI-liacs/BLADE/graph/badge.svg?token=ZOT67R1TP7\" alt=\"CodeCov\" height=\"18\"/> \n </a>\n</p>\n\n> [!TIP]\n> See also the [Documentation](https://xai-liacs.github.io/BLADE/).\n\n## Table of Contents\n- [News](#-news)\n- [Introduction](#introduction)\n- [Installation](#-installation)\n- [Quick Start](#-quick-start)\n- [Webapp](#-webapp)\n- [Contributing](#-contributing)\n- [License](#-license)\n- [Citation](#-citation)\n\n## \ud83d\udd25 News \n\n- 2025.03 \u2728\u2728 **BLADE v0.0.1 released**!\n\n\n## Introduction\n**BLADE** (Benchmark suite for LLM-driven Automated Design and Evolution) provides a standardized benchmark suite for evaluating automatic algorithm design algorithms, particularly those generating metaheuristics by large language models (LLMs). It focuses on **continuous black-box optimization** and integrates a diverse set of **problems** and **methods**, facilitating fair and comprehensive benchmarking.\n\n\n### Features\n\n- **Comprehensive Benchmark Suite:** Covers various classes of black-box optimization problems.\n- **LLM-Driven Evaluation:** Supports algorithm evolution and design using large language models.\n- **Built-In Baselines:** Includes state-of-the-art metaheuristics for comparison.\n- **Automatic Logging & Visualization:** Integrated with **IOHprofiler** for performance tracking.\n\n#### Included Benchmark Function Sets\n\nBLADE incorporates several benchmark function sets to provide a comprehensive evaluation environment:\n\n| Name | Short Description | Number of Functions | Multiple Instances |\n|------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------|---------------------|--------------------|\n| **BBOB** (Black-Box Optimization Benchmarking) | A suite of 24 noiseless functions designed for benchmarking continuous optimization algorithms. [Reference](https://arxiv.org/pdf/1903.06396) | 24 | Yes |\n| **SBOX-COST** | A set of 24 boundary-constrained functions focusing on strict box-constraint optimization scenarios. [Reference](https://inria.hal.science/hal-04403658/file/sboxcost-cmacomparison-authorversion.pdf) | 24 | Yes |\n| **MA-BBOB** (Many-Affine BBOB) | An extension of the BBOB suite, generating functions through affine combinations and shifts. [Reference](https://dl.acm.org/doi/10.1145/3673908) | Generator-Based | Yes |\n| **GECCO MA-BBOB Competition Instances** | A collection of 1,000 pre-defined instances from the GECCO MA-BBOB competition, evaluating algorithm performance on diverse affine-combined functions. [Reference](https://iohprofiler.github.io/competitions) | 1,000 | Yes |\n\nIn addition, several real-world applications are included such as several photonics problems.\n\n### Included Search Methods\n\nThe suite contains the state-of-the-art LLM-assisted search algorithms:\n\n| Algorithm | Description | Link\n|--------------------------|-------------------------------------------------|--------------|\n| **LLaMEA** | Large Langugage Model Evolutionary Algorithm | [code](https://github.com/nikivanstein/LLaMEA) [paper](https://arxiv.org/abs/2405.20132) |\n| **EoH** | Evolution of Heuristics | [code](https://github.com/FeiLiu36/EoH) [paper](https://arxiv.org/abs/2401.02051) |\n| **FunSearch** | Google's GA-like algorithm | [code](https://github.com/google-deepmind/funsearch) [paper](https://www.nature.com/articles/s41586-023-06924-6) |\n| **ReEvo** | Large Language Models as Hyper-Heuristics with Reflective Evolution | [code](https://github.com/ai4co/LLM-as-HH) [paper](https://arxiv.org/abs/2402.01145) |\n\n> Note, FunSearch is currently not yet integrated.\n\n### Supported LLM APIs\n\nBLADE supports integration with various LLM APIs to facilitate automated design of algorithms:\n\n| LLM Provider | Description | Integration Notes |\n|--------------|---------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------|\n| **Gemini** | Google's multimodal LLM designed to process text, images, audio, and more. [Reference](https://en.wikipedia.org/wiki/Gemini_%28language_model%29) | Accessible via the Gemini API, compatible with OpenAI libraries. [Reference](https://ai.google.dev/gemini-api/docs/openai) |\n| **OpenAI** | Developer of GPT series models, including GPT-4, widely used for natural language understanding and generation. [Reference](https://openai.com/) | Integration through OpenAI's REST API and client libraries. |\n| **Ollama** | A platform offering access to various LLMs, enabling local and cloud-based model deployment. [Reference](https://www.ollama.ai/) | Integration details can be found in their official documentation. |\n\n\n### Evaluating against Human Designed baselines\n\nAn important part of BLADE is the final evaluation of generated algorithms against state-of-the-art human designed algorithms.\nIn the `iohblade.baselines` part of the package, several well known SOTA black-box optimizers are imolemented to compare against.\nIncluding but not limited to CMA-ES and DE variants.\n\nFor the final validation **BLADE** uses [**IOHprofiler**](https://iohprofiler.github.io/), providing detailed tracking and visualization of performance metrics.\n\n\n## \ud83c\udf81 Installation\n\nIt is the easiest to use BLADE from the pypi package (`iohblade`).\n\n```bash\n pip install iohblade\n```\n> [!Important]\n> The Python version **must** be larger or equal to Python 3.11.\n> You need an OpenAI/Gemini/Ollama API key for using LLM models.\n\nYou can also install the package from source using <a href=\"https://docs.astral.sh/uv/\" target=\"_blank\">uv</a> (0.7.19).\nmake sure you have `uv` installed.\n\n1. Clone the repository:\n ```bash\n git clone https://github.com/XAI-liacs/BLADE.git\n cd BLADE\n ```\n\n2. Install the required dependencies via uv:\n ```bash\n uv sync\n ```\n\n3. *(Optional)* Install additional packages:\n ```bash\n uv sync --group kerneltuner --group dev --group docs\n ```\n This will install additional dependencies for development and building documentation.\n The (experimental) auto-kernel application is also under a separate group for now. \n\n## \ud83d\udcbb Quick Start\n\n1. Set up an OpenAI API key:\n - Obtain an API key from [OpenAI](https://openai.com/) or Gemini or another LLM provider.\n - Set the API key in your environment variables:\n ```bash\n export OPENAI_API_KEY='your_api_key_here'\n ```\n\n2. Running an Experiment\n\n To run a benchmarking experiment using BLADE:\n\n ```python\n from iohblade import Experiment\n from iohblade import Ollama_LLM\n from iohblade.methods import LLaMEA, RandomSearch\n from iohblade.problems import BBOB_SBOX\n from iohblade.loggers import ExperimentLogger\n import os\n\n llm = Ollama_LLM(\"qwen2.5-coder:14b\") #qwen2.5-coder:14b, deepseek-coder-v2:16b\n budget = 50 #short budget for testing\n\n RS = RandomSearch(llm, budget=budget) #Random Search baseline\n LLaMEA_method = LLaMEA(llm, budget=budget, name=\"LLaMEA\", n_parents=4, n_offspring=12, elitism=False) #LLamEA with 4,12 strategy\n methods = [RS, LLaMEA_method]\n\n problems = []\n # include all SBOX_COST functions with 5 instances for training and 10 for final validation as the benchmark problem.\n training_instances = [(f, i) for f in range(1,25) for i in range(1, 6)]\n test_instances = [(f, i) for f in range(1,25) for i in range(5, 16)]\n problems.append(BBOB_SBOX(training_instances=training_instances, test_instances=test_instances, dims=[5], budget_factor=2000, name=f\"SBOX_COST\"))\n # Set up the experiment object with 5 independent runs per method/problem. (in this case 1 problem)\n logger = ExperimentLogger(\"results/SBOX\")\n experiment = Experiment(methods=methods, problems=problems, runs=5, show_stdout=True, exp_logger=logger) #normal run\n experiment() #run the experiment, all data is logged in the folder results/SBOX/\n ```\n\n## \ud83c\udf10 Webapp\n\nAfter running experiments you can browse them using the built-in Streamlit app:\n\n```bash\nuv run iohblade-webapp\n```\n\nThe app lists available experiments from the `results` directory, displays their progress, and shows convergence plots.\n\n---\n\n## \ud83d\udcbb Examples\n\n\nSee the files in the `examples` folder for examples on experiments and visualisations.\n\n---\n\n## \ud83e\udd16 Contributing\n\nContributions to BLADE are welcome! Here are a few ways you can help:\n\n- **Report Bugs**: Use [GitHub Issues](https://github.com/XAI-Liacs/BLADE/issues) to report bugs.\n- **Feature Requests**: Suggest new features or improvements.\n- **Pull Requests**: Submit PRs for bug fixes or feature additions.\n\nPlease refer to CONTRIBUTING.md for more details on contributing guidelines.\n\n## \ud83e\udeaa License\n\nDistributed under the [MIT](https://choosealicense.com/licenses/mit/) License. See `LICENSE` for more information.\n\n\n## \u2728 Citation\n\n\nTBA\n\n-----\nHappy Benchmarking with IOH-BLADE! \ud83d\ude80\n",
"bugtrack_url": null,
"license": null,
"summary": "Benchmarking Llm Assisted Design and Evolution of algorithms.",
"version": "0.2.1",
"project_urls": {
"Homepage": "https://github.com/XAI-liacs/BLADE",
"Repository": "https://github.com/XAI-liacs/BLADE"
},
"split_keywords": [],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "15fa028c7b3ab9b42f0d1e75e838c2268d24ddad50a16f9d6f3eb337bd89f81e",
"md5": "eabf478b37202c737f6180f5540fa0e7",
"sha256": "de925bdc65dd936b4a6cbc9d767d182ea081b5457f703075d530f3e63314a313"
},
"downloads": -1,
"filename": "iohblade-0.2.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "eabf478b37202c737f6180f5540fa0e7",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4,>=3.11",
"size": 364407,
"upload_time": "2025-07-10T12:50:52",
"upload_time_iso_8601": "2025-07-10T12:50:52.566513Z",
"url": "https://files.pythonhosted.org/packages/15/fa/028c7b3ab9b42f0d1e75e838c2268d24ddad50a16f9d6f3eb337bd89f81e/iohblade-0.2.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "fc483f8c21c477fc45387959c44f97fa4d2ea27991495e8fc0c650735d147049",
"md5": "a7b58caac95ed45be07ce9116c5c4954",
"sha256": "55cf67d49308e872a808052b89e6c7a72d6160963e4af385b7f36c87dd0b10f2"
},
"downloads": -1,
"filename": "iohblade-0.2.1.tar.gz",
"has_sig": false,
"md5_digest": "a7b58caac95ed45be07ce9116c5c4954",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4,>=3.11",
"size": 6252826,
"upload_time": "2025-07-10T12:50:54",
"upload_time_iso_8601": "2025-07-10T12:50:54.141828Z",
"url": "https://files.pythonhosted.org/packages/fc/48/3f8c21c477fc45387959c44f97fa4d2ea27991495e8fc0c650735d147049/iohblade-0.2.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-10 12:50:54",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "XAI-liacs",
"github_project": "BLADE",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"lcname": "iohblade"
}
