# Fixstars Amplify Benchmark
[](https://www.fixstars.com/)
[](https://amplify.fixstars.com/)
[](README-ja.md)
## What is Amplify Benchmark?
Fixstars Amplify Benchmark is a framework for benchmarking the performances of solvers for quadratic unconstrained binary optimization problems (QUBO). It provides a command line interface to perform benchmarking and a definition of the benchmark problem.
The [Fixstars Amplify SDK](https://amplify.fixstars.com/sdk) is used as the backend, allowing benchmarks to be run with many solvers such as quantum annealing machines, Ising machines, and mathematical optimization solvers. Benchmarks are run based on a job set file that defines the target problems, the number of runs, and the solvers to be used, making it easy to automate the process from benchmark execution to analyzing results.
The results of this library run can be loaded into the [Amplify Benchmark Viewer](https://github.com/fixstars/amplify-benchmark-viewer) to visualize the results in a web browser. A demo of the benchmark results for Amplify AE is [here](https://amplify.fixstars.com/benchmark/).
### Features
* Easy to run
* Parallel execution
* Automatic evaluation and analysis
* Benchmark result viewer is provided
* Customizable solver and problem parameters
* Formulations for several benchmark sets are pre-defined
* User-define problems can be added
Pre-defined benchmark sets:
* Traveling Salesperson Problem: [TSPLIB](http://comopt.ifi.uni-heidelberg.de/software/TSPLIB95/)
* Quadratic Assignment Problem: [QAPLIB](https://coral.ise.lehigh.edu/data-sets/qaplib/)
* Max-CUT Problem: [Gset](https://web.stanford.edu/~yyye/yyye/Gset/)
* Capacitated Vehicle Routing Problem: [CVRPLIB](http://vrp.galgos.inf.puc-rio.br/index.php/en/)
* Quadratic Problem: [QPLIB](https://qplib.zib.de/)
* Sudoku (logic-based combinatorial number-placement puzzle)
Supported solvers powered by Amplify SDK:
* [Fixstars Amplify AE](https://amplify.fixstars.com/)
* [D-Wave Advantage](https://www.dwavesys.com/solutions-and-products/systems/)
* [Fujitsu Digital Annealer 3/4](https://www.fujitsu.com/global/services/business-services/digital-annealer/)
* [Toshiba SQBM+](https://www.global.toshiba/ww/products-solutions/ai-iot/sbm.html)
* [Gurobi Optimizer](https://www.gurobi.com/solutions/gurobi-optimizer/)
* (See [supported solvers of Amplify SDK](https://amplify.fixstars.com/docs/client.html))
## Gallery
| Objective value for execution time | Time To Solution (TTS) |
| :----------------------------------------------------------: | --------------------------------------------------------------- |
| <img src="docs/screenshots/target_energy.png" width="600px"> | <img src="docs/screenshots/time_to_solution.png" width="600px"> |
| Probability of obtaining a feasible solution | Probability of obtaining the best solution Rate |
| ------------------------------------------------------------ | -------------------------------------------------------------- |
| <img src="docs/screenshots/feasible_rate.png" width="600px"> | <img src="docs/screenshots/reach_best_rate.png" width="600px"> |
| Table DATA |
| ----------------------------------------------------- |
| <img src="docs/screenshots/table.png" width="1225px"> |
## Getting started
### Installation
Amplify benchmark is provided as a Python (>=3.8) library. It can be installed with pip as follows:
```bash
$ pip install amplify-bench
```
### Run example job set
After installation, the `amplify-bench` command is enabled.
```bash
$ amplify-bench --help
Usage: amplify-bench [OPTIONS] COMMAND [ARGS]...
Options:
--help Show this message and exit.
Commands:
download Download all supported instance files in the specified...
run QUBO Benchmark
stats Generate QUBO benchmark stats data.
```
To run a benchmark, you have to create a benchmark definition ("job set") file. The [`example/benchmark.yml`](example/benchmark.yml) file contains a sample job set file.
[`example/benchmark.yml`](example/benchmark.yml)
```yaml
jobs:
- problem:
class: Tsp
instance: eil51
client:
- FixstarsClient
- token: INPUT_API_TOKEN
parameters:
timeout: 3000
num_samples: 2
```
The benchmark job set file contains a list of benchmark jobs in YAML or JSON file format. The jobs consist of the number of runs, a list of problems to solve, and parameter values passed to the [Client class](https://amplify.fixstars.com/docs/client.html) to run. For this job set, it consists of the following benchmark jobs:
* target problem:
* `TSPLIB`: `eil51` instance
* number of runs: 2
* [`FixstarsClient`](https://amplify.fixstars.com/docs/client.html#fixstars)
* `token`: INPUT_API_TOKEN
* `parameter.timeout`: 3000
Now to start the benchmark using this job set file, run the `amplify-bench` command with the `run` subcommand with the path to the job set file.
> **Note**
> Replace `INPUT_API_TOKEN` with your API token. If you do not have an API token, go to [Amplify WEB site](https://amplify.fixstars.com/) and [create an account](https://amplify.fixstars.com/ja/register).
```bash
$ amplify-bench run benchmark.yml
input_json: benchmark.yml
label: 20230803_223440
output: None
parallel: 1
aws_profile: None
dry_run: False
cli_benchmark_impl() 20230803_223440
2023-08-03 22:34:41,308 [pid:94470] [INFO]: 542.49 ms in amplify_bench.cli.parser.parse_input_data
2023-08-03 22:34:41,309 [pid:94470] [INFO]: make model of eil51
2023-08-03 22:34:41,519 [pid:94470] [INFO]: 209.08 ms in amplify_bench.problem.tsp.make_tsp_model
total jobs: 2
success jobs: 2
error jobs: 0
Jobs not yet started: 0
```
When execution completes a JSON file is output as the result of the execution in the same directory. The file name is appended with the execution time by default. The output file path can be changed with the `--output <path>` option.
### Open the result with Amplify Benchmark Viewer
The results can be visualized using the [Amplify Benchmark Viewer](https://github.com/fixstars/amplify-benchmark-viewer). To analyze for the viewer, give the `stats` subcommand with the path to a directory or a result file to the `amplify-bench` command.
```bash
$ amplify-bench stats preset_20230803_223440.json
```
By default, a `report/data.json` file is created in the current directory. The path directory of the output file can be changed with the `--output` option.
Then, drag and drop the created `report/data.json` file into the Amplify Benchmark Viewer [GitHub pages](https://fixstars.github.io/amplify-benchmark-viewer/) to visualize the results.
> **Note**
> The data is analyzed only on the browser and is not stored on the external server.
| Drag and Drop `data.json` file | Show the evaluated problem list | The detailed evaluation for each problem |
| ----------------------------------------------------- | ----------------------------------------------------------- | ------------------------------------------------------ |
| <img src="docs/screenshots/upload.png" width="600px"> | <img src="docs/screenshots/problem_list.png" width="600px"> | <img src="docs/screenshots/problem.png" width="600px"> |
## Advanced usage
### job set file in details
#### `PresetObject`
A job set file consisted of JSON objects with the following keys. The schema of the job set file is described in [`amplify_bench/cli/schemas`](amplify_bench/cli/schemas).
| key | type | description |
| ----------- | ------------------ | -------------------------------------------------- |
| `jobs` | `array[JobObject]` | list of benchmark jobs |
| `variables` | `object` | definitions of variables used in `jobs` (Optional) |
| `imports` | `array[string]` | User-defined problem file path (Optional) |
Strings in the file that begin with `$` are treated as variable names. Variables are first expanded by the environment variables at runtime, then the variable definitions given in the `variables` key are referenced in `jobs`. This is useful, for example, to specify a setting that is commonly used in multiple jobs
```yaml
variables:
CLIENT:
- FixstarsClient
- parameters:
timeout: 3000
jobs:
- problem:
class: Tsp
instance: eil51
client: $CLIENT
num_samples: 2
- problem:
class: Tsp
instance: burma14
client: $CLIENT
num_samples: 1
```
`imports` specifies the list of paths to the user-defined problem file. Enter the path as a relative path from a job set file, a relative path from the current directory, or an absolute path. For details on user-defined question files, see [Create your own benchmark problems](#create-your-own-benchmark-problems).
#### `JobObject`
| key | type | description |
| ------------- | --------------- | ----------------------------------------------- |
| `num_samples` | `int` | the number of runs |
| `client` | `array` | client name and parameters |
| `problem` | `ProblemObject` | the problem definition |
| `matrix` | `object[array]` | the definitions of variable patterns (Optional) |
If `num_samples` is an integer greater than 1, it will be run multiple times with the same settings. The `matrix` key is given with the patterns of variables explained later.
The `client` key is given an array of length 2. The first element of the array is the name of the client class, and the second element is an object with the property values of the client. For example, the following client configuration in the Amplify SDK
```python
from amplify.client import FixstarsClient
client = FixstarsClient()
client.token = "INPUT_API_TOKEN"
client.parameters.timeout = 1000
```
should be specified as follows in the job set file.
```yaml
jobs:
- client:
- FixstarsClient
- token: INPUT_API_TOKEN
parameters:
timeout: 1000
```
> **Note**
> See the [documentation](https://amplify.fixstars.com/ja/docs/client.html) for the available properties for each `Client` class.
#### `ProblemObject`
The `problem` key has an object consisting of the following keys:
| key | type | description |
| ------------ | -------- | ------------------------------------------- |
| `class` | `string` | The name of the problem class |
| `instance` | `string` | Instance name |
| `parameters` | `object` | Constructor parameters of the problem class |
The `class` is the name of the problem class contained in [`amplify_bench/problem`](https://github.com/fixstars/amplify-benchmark/tree/main/amplify_bench/problem). The predefined problem classes are `Tsp` (TSPLIB), `Qap` (QAPLIB), `Cvrp` (CVRPLIB), `MaxCut` (GSET), `Sudoku` and `Qplib` (QPLIB). The `instance` is the name of the instance in the problem set corresponding to each problem class. See [`amplify_bench/problem/data`](amplify_bench/problem/data) for details. Problem classes may have formulation parameters that can be passed to the constructor, which can be specified by `parameter` key.
#### Using a matrix for your jobs
Multiple jobs can be automatically generated for all combinations of multiple variable patterns given in a single job definition. For example, to run for all combinations of multiple instances and multiple runtimes, the following job set file can be used.
```yaml
variables:
NUM_SAMPLES: 100
FIXSTARS:
- FixstarsClient
- token: INPUT_API_TOKEN
parameters:
timeout: $TIMEOUT
jobs:
- problem:
class: Qap
instance: $INSTANCE
client: $FIXSTARS
num_samples: $NUM_SAMPLES
matrix:
INSTANCE:
- esc32a
- sko56
TIMEOUT:
- 10000
- 30000
```
The `matrix` is an array of values with the variable names as keys. In this case, jobs with `timeout` of `10000` and `30000` will be created for `esc32a` and `sko56` respectively. Note that you can refer to the variables you pass to `matrix` in the variables defined in `variables`.
> **Note**
> Variables are referenced recursively, but an infinite loop will fail.
### Create your own benchmark problems
User-defined formulations can be added as benchmark problems that are recognized in Amplify Benchmark.
The following example runs a benchmark against the `MyTsp` class defined in the `mytsp.py` file. Setting a list of Python file paths to the `imports` key will load additional probem classes defined in the files. A file path must be specified as an absolute path or relative to the job set file or the current directory.
```yaml
imports:
- mytsp.py
jobs:
- problem:
class: MyTsp
instance: random8
```
> **Note**
> User-defined class names should not duplicate the built-in problem classes.
The problem class must extend the `Problem` class and implement the constructor (`__init__`) and the methods `make_model` and `evaluate`. The `make_model` method formulates the problem in Amplify SDK and the `evaluate` method evaluates the formulated model with the solution as input.
The following code snippet is an example of a `MyTsp` problem class.
`mytsp.py`
```python
class MyTsp(Problem):
def __init__(
self,
instance: str,
constraint_weight: float = 1.0,
seed: int = 0,
path: Optional[str] = None,
):
super().__init__()
self._instance: str = instance
self._problem_parameters["constraint_weight"] = constraint_weight
if instance.startswith("random"):
self._problem_parameters["seed"] = seed
self._symbols = None
ncity, distances, locations, best_known = self.__load(self._instance, seed, path)
self._ncity = ncity
self._distances = distances
self._locations = locations # not used
self._best_known = best_known
def make_model(self):
symbols, model = make_tsp_model(self._ncity, self._distances, self._problem_parameters["constraint_weight"])
self._symbols = symbols
self._model = model
def evaluate(self, solution: SolverSolution) -> Dict[str, Union[None, float, str]]:
value: Optional[float] = None
path: str = ""
if solution.is_feasible:
spins = solution.values
variables = np.array(self._symbols.decode(spins)) # type: ignore
index = np.where(variables == 1)[1]
index_str = [str(idx) for idx in index]
value = calc_tour_dist(list(index), self._distances)
path = " ".join(index_str)
else:
pass
return {"label": "total distances", "value": value, "path": path}
...
```
#### Constructor
```python
def __init__(self, instance: str, **kwargs) -> None
```
The constructor must accept at least an `instance: str` argument. Otherwise, if you add `constraint_weight: float` to the constructor arguments for example, the `parameters` key in `ProblemObject` can have a `constraint_weight`.
#### `make_model` method
```python
def make_model(self) -> None
```
The `make_model` method is responsible for formulating and storing an instance of the `amplify.BinaryQuadraticModel` class in `self._model`.
#### `evaluate` method
```python
def evaluate(self, solution: amplify.SolverSolution) -> Dict[str, Union[None, float, str]]
```
The `evaluate` method takes and evaluates a `amplify.SolverSolution`. The return value can be any key and value as `Dict[str, Union[None, float, str]]`. The return value is output to the `objective_value` key in the JSON file of the benchmark result.
## Contributing
Amplify Benchmark is an open source project. Contributions are welcome, including bug reports, feature additions, documentation improvements, etc.
## Developed by
The Amplify benchmark project ties together:
* [Fixstars Amplify SDK](https://amplify.fixstars.com/sdk)
* [Fixstars Amplify Annealing Engine](https://amplify.fixstars.com/engine)
* [Fixstars Amplify Benchmark Viewer](https://github.com/fixstars/amplify-benchmark-viewer)
<div align="left">
<a href="https://www.fixstars.com/" target="_blank" rel="nofollow noreferrer">
<img
height="85"
src="docs/logos/fixstars-logo.svg"
alt="Fixstars logo"
>
</a>
<a href="https://amplify.fixstars.com/" target="_blank" rel="nofollow noreferrer">
<img
height="86"
src="docs/logos/amplify-logo.svg"
alt="Fixstars Amplify logo"
>
</a>
</div>
Raw data
{
"_id": null,
"home_page": "https://github.com/fixstars/amplify-benchmark",
"name": "amplify-bench",
"maintainer": "Yoshiki Matsuda, Hiroki Kawahara, Kosuke Suzuki",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "quantum,quantum annealing,annealer,annealing machine,QUBO,quadratic unconstrained binary optimization,d-wave,digital annealer,sdk,sbm,simulated bifurcation,simulated annealing,Ising machine,Ising model,fixstars",
"author": "Fixstars Corporation, Fixstars Amplify Corporation",
"author_email": "\"Yoshiki Matsuda\" <y_matsuda@fixstars.com>",
"download_url": "",
"platform": null,
"description": "# Fixstars Amplify Benchmark\n\n[](https://www.fixstars.com/)\n[](https://amplify.fixstars.com/)\n[](README-ja.md)\n\n## What is Amplify Benchmark?\n\nFixstars Amplify Benchmark is a framework for benchmarking the performances of solvers for quadratic unconstrained binary optimization problems (QUBO). It provides a command line interface to perform benchmarking and a definition of the benchmark problem.\n\nThe [Fixstars Amplify SDK](https://amplify.fixstars.com/sdk) is used as the backend, allowing benchmarks to be run with many solvers such as quantum annealing machines, Ising machines, and mathematical optimization solvers. Benchmarks are run based on a job set file that defines the target problems, the number of runs, and the solvers to be used, making it easy to automate the process from benchmark execution to analyzing results.\n\nThe results of this library run can be loaded into the [Amplify Benchmark Viewer](https://github.com/fixstars/amplify-benchmark-viewer) to visualize the results in a web browser. A demo of the benchmark results for Amplify AE is [here](https://amplify.fixstars.com/benchmark/).\n\n### Features\n\n* Easy to run\n* Parallel execution\n* Automatic evaluation and analysis\n* Benchmark result viewer is provided\n* Customizable solver and problem parameters\n* Formulations for several benchmark sets are pre-defined\n* User-define problems can be added\n\nPre-defined benchmark sets:\n\n* Traveling Salesperson Problem: [TSPLIB](http://comopt.ifi.uni-heidelberg.de/software/TSPLIB95/)\n* Quadratic Assignment Problem: [QAPLIB](https://coral.ise.lehigh.edu/data-sets/qaplib/)\n* Max-CUT Problem: [Gset](https://web.stanford.edu/~yyye/yyye/Gset/)\n* Capacitated Vehicle Routing Problem: [CVRPLIB](http://vrp.galgos.inf.puc-rio.br/index.php/en/)\n* Quadratic Problem: [QPLIB](https://qplib.zib.de/)\n* Sudoku (logic-based combinatorial number-placement puzzle)\n\nSupported solvers powered by Amplify SDK:\n\n* [Fixstars Amplify AE](https://amplify.fixstars.com/)\n* [D-Wave Advantage](https://www.dwavesys.com/solutions-and-products/systems/)\n* [Fujitsu Digital Annealer 3/4](https://www.fujitsu.com/global/services/business-services/digital-annealer/)\n* [Toshiba SQBM+](https://www.global.toshiba/ww/products-solutions/ai-iot/sbm.html)\n* [Gurobi Optimizer](https://www.gurobi.com/solutions/gurobi-optimizer/)\n* (See [supported solvers of Amplify SDK](https://amplify.fixstars.com/docs/client.html))\n\n## Gallery\n\n| Objective value for execution time | Time To Solution (TTS) |\n| :----------------------------------------------------------: | --------------------------------------------------------------- |\n| <img src=\"docs/screenshots/target_energy.png\" width=\"600px\"> | <img src=\"docs/screenshots/time_to_solution.png\" width=\"600px\"> |\n\n| Probability of obtaining a feasible solution | Probability of obtaining the best solution Rate |\n| ------------------------------------------------------------ | -------------------------------------------------------------- |\n| <img src=\"docs/screenshots/feasible_rate.png\" width=\"600px\"> | <img src=\"docs/screenshots/reach_best_rate.png\" width=\"600px\"> |\n\n| Table DATA |\n| ----------------------------------------------------- |\n| <img src=\"docs/screenshots/table.png\" width=\"1225px\"> |\n\n## Getting started\n\n### Installation\n\nAmplify benchmark is provided as a Python (>=3.8) library. It can be installed with pip as follows:\n\n```bash\n$ pip install amplify-bench\n```\n\n### Run example job set\n\nAfter installation, the `amplify-bench` command is enabled.\n\n```bash\n$ amplify-bench --help\nUsage: amplify-bench [OPTIONS] COMMAND [ARGS]...\n\nOptions:\n --help Show this message and exit.\n\nCommands:\n download Download all supported instance files in the specified...\n run QUBO Benchmark\n stats Generate QUBO benchmark stats data.\n```\n\nTo run a benchmark, you have to create a benchmark definition (\"job set\") file. The [`example/benchmark.yml`](example/benchmark.yml) file contains a sample job set file.\n\n[`example/benchmark.yml`](example/benchmark.yml)\n\n```yaml\njobs:\n - problem:\n class: Tsp\n instance: eil51\n client:\n - FixstarsClient\n - token: INPUT_API_TOKEN\n parameters:\n timeout: 3000\n num_samples: 2\n```\n\nThe benchmark job set file contains a list of benchmark jobs in YAML or JSON file format. The jobs consist of the number of runs, a list of problems to solve, and parameter values passed to the [Client class](https://amplify.fixstars.com/docs/client.html) to run. For this job set, it consists of the following benchmark jobs:\n\n* target problem:\n * `TSPLIB`: `eil51` instance\n* number of runs: 2\n* [`FixstarsClient`](https://amplify.fixstars.com/docs/client.html#fixstars)\n * `token`: INPUT_API_TOKEN\n * `parameter.timeout`: 3000\n\nNow to start the benchmark using this job set file, run the `amplify-bench` command with the `run` subcommand with the path to the job set file.\n\n> **Note**\n> Replace `INPUT_API_TOKEN` with your API token. If you do not have an API token, go to [Amplify WEB site](https://amplify.fixstars.com/) and [create an account](https://amplify.fixstars.com/ja/register).\n\n```bash\n$ amplify-bench run benchmark.yml\ninput_json: benchmark.yml\nlabel: 20230803_223440\noutput: None\nparallel: 1\naws_profile: None\ndry_run: False\ncli_benchmark_impl() 20230803_223440\n2023-08-03 22:34:41,308 [pid:94470] [INFO]: 542.49 ms in amplify_bench.cli.parser.parse_input_data\n2023-08-03 22:34:41,309 [pid:94470] [INFO]: make model of eil51\n2023-08-03 22:34:41,519 [pid:94470] [INFO]: 209.08 ms in amplify_bench.problem.tsp.make_tsp_model\ntotal jobs: 2\nsuccess jobs: 2\nerror jobs: 0\nJobs not yet started: 0\n```\n\nWhen execution completes a JSON file is output as the result of the execution in the same directory. The file name is appended with the execution time by default. The output file path can be changed with the `--output <path>` option.\n\n### Open the result with Amplify Benchmark Viewer\n\nThe results can be visualized using the [Amplify Benchmark Viewer](https://github.com/fixstars/amplify-benchmark-viewer). To analyze for the viewer, give the `stats` subcommand with the path to a directory or a result file to the `amplify-bench` command.\n\n```bash\n$ amplify-bench stats preset_20230803_223440.json\n```\n\nBy default, a `report/data.json` file is created in the current directory. The path directory of the output file can be changed with the `--output` option.\n\nThen, drag and drop the created `report/data.json` file into the Amplify Benchmark Viewer [GitHub pages](https://fixstars.github.io/amplify-benchmark-viewer/) to visualize the results.\n\n> **Note**\n> The data is analyzed only on the browser and is not stored on the external server.\n\n| Drag and Drop `data.json` file | Show the evaluated problem list | The detailed evaluation for each problem |\n| ----------------------------------------------------- | ----------------------------------------------------------- | ------------------------------------------------------ |\n| <img src=\"docs/screenshots/upload.png\" width=\"600px\"> | <img src=\"docs/screenshots/problem_list.png\" width=\"600px\"> | <img src=\"docs/screenshots/problem.png\" width=\"600px\"> |\n\n## Advanced usage\n\n### job set file in details\n\n#### `PresetObject`\n\nA job set file consisted of JSON objects with the following keys. The schema of the job set file is described in [`amplify_bench/cli/schemas`](amplify_bench/cli/schemas).\n\n| key | type | description |\n| ----------- | ------------------ | -------------------------------------------------- |\n| `jobs` | `array[JobObject]` | list of benchmark jobs |\n| `variables` | `object` | definitions of variables used in `jobs` (Optional) |\n| `imports` | `array[string]` | User-defined problem file path (Optional) |\n\nStrings in the file that begin with `$` are treated as variable names. Variables are first expanded by the environment variables at runtime, then the variable definitions given in the `variables` key are referenced in `jobs`. This is useful, for example, to specify a setting that is commonly used in multiple jobs\n\n```yaml\nvariables:\n CLIENT:\n - FixstarsClient\n - parameters:\n timeout: 3000\njobs:\n - problem:\n class: Tsp\n instance: eil51\n client: $CLIENT\n num_samples: 2\n - problem:\n class: Tsp\n instance: burma14\n client: $CLIENT\n num_samples: 1\n```\n\n`imports` specifies the list of paths to the user-defined problem file. Enter the path as a relative path from a job set file, a relative path from the current directory, or an absolute path. For details on user-defined question files, see [Create your own benchmark problems](#create-your-own-benchmark-problems).\n\n#### `JobObject`\n\n| key | type | description |\n| ------------- | --------------- | ----------------------------------------------- |\n| `num_samples` | `int` | the number of runs |\n| `client` | `array` | client name and parameters |\n| `problem` | `ProblemObject` | the problem definition |\n| `matrix` | `object[array]` | the definitions of variable patterns (Optional) |\n\nIf `num_samples` is an integer greater than 1, it will be run multiple times with the same settings. The `matrix` key is given with the patterns of variables explained later.\n\nThe `client` key is given an array of length 2. The first element of the array is the name of the client class, and the second element is an object with the property values of the client. For example, the following client configuration in the Amplify SDK\n\n```python\nfrom amplify.client import FixstarsClient\n\nclient = FixstarsClient()\nclient.token = \"INPUT_API_TOKEN\"\nclient.parameters.timeout = 1000\n```\n\nshould be specified as follows in the job set file.\n\n```yaml\njobs:\n - client:\n - FixstarsClient\n - token: INPUT_API_TOKEN\n parameters:\n timeout: 1000\n```\n\n> **Note**\n> See the [documentation](https://amplify.fixstars.com/ja/docs/client.html) for the available properties for each `Client` class.\n\n#### `ProblemObject`\n\nThe `problem` key has an object consisting of the following keys:\n\n| key | type | description |\n| ------------ | -------- | ------------------------------------------- |\n| `class` | `string` | The name of the problem class |\n| `instance` | `string` | Instance name |\n| `parameters` | `object` | Constructor parameters of the problem class |\n\nThe `class` is the name of the problem class contained in [`amplify_bench/problem`](https://github.com/fixstars/amplify-benchmark/tree/main/amplify_bench/problem). The predefined problem classes are `Tsp` (TSPLIB), `Qap` (QAPLIB), `Cvrp` (CVRPLIB), `MaxCut` (GSET), `Sudoku` and `Qplib` (QPLIB). The `instance` is the name of the instance in the problem set corresponding to each problem class. See [`amplify_bench/problem/data`](amplify_bench/problem/data) for details. Problem classes may have formulation parameters that can be passed to the constructor, which can be specified by `parameter` key.\n\n#### Using a matrix for your jobs\n\nMultiple jobs can be automatically generated for all combinations of multiple variable patterns given in a single job definition. For example, to run for all combinations of multiple instances and multiple runtimes, the following job set file can be used.\n\n```yaml\nvariables:\n NUM_SAMPLES: 100\n FIXSTARS:\n - FixstarsClient\n - token: INPUT_API_TOKEN\n parameters:\n timeout: $TIMEOUT\njobs:\n - problem:\n class: Qap\n instance: $INSTANCE\n client: $FIXSTARS\n num_samples: $NUM_SAMPLES\n matrix:\n INSTANCE:\n - esc32a\n - sko56\n TIMEOUT:\n - 10000\n - 30000\n```\n\nThe `matrix` is an array of values with the variable names as keys. In this case, jobs with `timeout` of `10000` and `30000` will be created for `esc32a` and `sko56` respectively. Note that you can refer to the variables you pass to `matrix` in the variables defined in `variables`.\n\n> **Note**\n> Variables are referenced recursively, but an infinite loop will fail.\n\n### Create your own benchmark problems\n\nUser-defined formulations can be added as benchmark problems that are recognized in Amplify Benchmark.\n\nThe following example runs a benchmark against the `MyTsp` class defined in the `mytsp.py` file. Setting a list of Python file paths to the `imports` key will load additional probem classes defined in the files. A file path must be specified as an absolute path or relative to the job set file or the current directory.\n\n```yaml\nimports:\n - mytsp.py\njobs:\n - problem:\n class: MyTsp\n instance: random8\n```\n\n> **Note**\n> User-defined class names should not duplicate the built-in problem classes.\n\nThe problem class must extend the `Problem` class and implement the constructor (`__init__`) and the methods `make_model` and `evaluate`. The `make_model` method formulates the problem in Amplify SDK and the `evaluate` method evaluates the formulated model with the solution as input.\n\nThe following code snippet is an example of a `MyTsp` problem class.\n\n`mytsp.py`\n\n```python\nclass MyTsp(Problem):\n def __init__(\n self,\n instance: str,\n constraint_weight: float = 1.0,\n seed: int = 0,\n path: Optional[str] = None,\n ):\n super().__init__()\n self._instance: str = instance\n self._problem_parameters[\"constraint_weight\"] = constraint_weight\n if instance.startswith(\"random\"):\n self._problem_parameters[\"seed\"] = seed\n self._symbols = None\n\n ncity, distances, locations, best_known = self.__load(self._instance, seed, path)\n self._ncity = ncity\n self._distances = distances\n self._locations = locations # not used\n self._best_known = best_known\n\n def make_model(self):\n symbols, model = make_tsp_model(self._ncity, self._distances, self._problem_parameters[\"constraint_weight\"])\n self._symbols = symbols\n self._model = model\n\n def evaluate(self, solution: SolverSolution) -> Dict[str, Union[None, float, str]]:\n value: Optional[float] = None\n path: str = \"\"\n\n if solution.is_feasible:\n spins = solution.values\n variables = np.array(self._symbols.decode(spins)) # type: ignore\n index = np.where(variables == 1)[1]\n index_str = [str(idx) for idx in index]\n value = calc_tour_dist(list(index), self._distances)\n path = \" \".join(index_str)\n else:\n pass\n\n return {\"label\": \"total distances\", \"value\": value, \"path\": path}\n...\n```\n\n#### Constructor\n\n```python\ndef __init__(self, instance: str, **kwargs) -> None\n```\n\nThe constructor must accept at least an `instance: str` argument. Otherwise, if you add `constraint_weight: float` to the constructor arguments for example, the `parameters` key in `ProblemObject` can have a `constraint_weight`.\n\n#### `make_model` method\n\n```python\ndef make_model(self) -> None\n```\n\nThe `make_model` method is responsible for formulating and storing an instance of the `amplify.BinaryQuadraticModel` class in `self._model`.\n\n#### `evaluate` method\n\n```python\ndef evaluate(self, solution: amplify.SolverSolution) -> Dict[str, Union[None, float, str]]\n```\n\nThe `evaluate` method takes and evaluates a `amplify.SolverSolution`. The return value can be any key and value as `Dict[str, Union[None, float, str]]`. The return value is output to the `objective_value` key in the JSON file of the benchmark result.\n\n## Contributing\n\nAmplify Benchmark is an open source project. Contributions are welcome, including bug reports, feature additions, documentation improvements, etc.\n\n## Developed by\n\nThe Amplify benchmark project ties together:\n\n* [Fixstars Amplify SDK](https://amplify.fixstars.com/sdk)\n* [Fixstars Amplify Annealing Engine](https://amplify.fixstars.com/engine)\n* [Fixstars Amplify Benchmark Viewer](https://github.com/fixstars/amplify-benchmark-viewer)\n\n<div align=\"left\">\n <a href=\"https://www.fixstars.com/\" target=\"_blank\" rel=\"nofollow noreferrer\">\n <img\n height=\"85\"\n src=\"docs/logos/fixstars-logo.svg\"\n alt=\"Fixstars logo\"\n >\n </a>\n <a href=\"https://amplify.fixstars.com/\" target=\"_blank\" rel=\"nofollow noreferrer\">\n <img\n height=\"86\"\n src=\"docs/logos/amplify-logo.svg\"\n alt=\"Fixstars Amplify logo\"\n >\n </a>\n</div>\n",
"bugtrack_url": null,
"license": "",
"summary": "Benchmark framework for Quantum and Classical Annealing Machines",
"version": "0.2.0",
"project_urls": {
"Homepage": "https://github.com/fixstars/amplify-benchmark"
},
"split_keywords": [
"quantum",
"quantum annealing",
"annealer",
"annealing machine",
"qubo",
"quadratic unconstrained binary optimization",
"d-wave",
"digital annealer",
"sdk",
"sbm",
"simulated bifurcation",
"simulated annealing",
"ising machine",
"ising model",
"fixstars"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ad6f957ff3a736288abc47cc5f201fcf6d5cbbfd1d19f1167651e5d6c4cdd048",
"md5": "48a959ed7dfa2c0c316cd8521f626983",
"sha256": "00fc8b825e5c1034a9794b711ed94cd29e12157decfff8bace993567aaaf6f27"
},
"downloads": -1,
"filename": "amplify_bench-0.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "48a959ed7dfa2c0c316cd8521f626983",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 68537,
"upload_time": "2023-10-08T05:08:46",
"upload_time_iso_8601": "2023-10-08T05:08:46.031152Z",
"url": "https://files.pythonhosted.org/packages/ad/6f/957ff3a736288abc47cc5f201fcf6d5cbbfd1d19f1167651e5d6c4cdd048/amplify_bench-0.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-10-08 05:08:46",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "fixstars",
"github_project": "amplify-benchmark",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "amplify-bench"
}
JSON
