# QP solvers benchmark
[](https://github.com/qpsolvers/qpsolvers_benchmark/actions)
[](https://pypi.org/project/qpsolvers_benchmark/)
[](https://github.com/qpsolvers/qpsolvers_benchmark/tree/master/CONTRIBUTING.md)
Benchmark for quadratic programming (QP) solvers available in Python.
The goal of this benchmark is to help users compare and select QP solvers. Its methodology is open to [discussions](https://github.com/qpsolvers/qpsolvers_benchmark/discussions). The benchmark ships standard and community [test sets](#test-sets), as well as a ``qpsolvers_benchmark`` command-line tool to run test sets directly. The main output of the benchmark are standardized reports evaluating all [metrics](#metrics) across all QP solvers available on the test machine. This repository also distributes [results](#results) from running the benchmark on a reference computer.
New test sets are welcome! The benchmark is designed so that each test-set comes in a standalone directory. Feel free to create a new one and [contribute it](CONTRIBUTING.md) here so that we grow the collection over time.
## Test sets
The benchmark comes with standard and community test sets to represent different use cases for QP solvers:
| Test set | Problems | Brief description |
| ------------------------ | -------- | ----------------- |
| **Maros-Meszaros** | 138 | Standard, designed to be difficult. |
| **Maros-Meszaros dense** | 62 | Subset of Maros-Meszaros restricted to smaller dense problems. |
| **GitHub free-for-all** | 12 | Community-built, new problems [are welcome](https://github.com/qpsolvers/qpsolvers_benchmark/issues/new?template=new_problem.md)! |
## Solvers
| Solver | Keyword | Algorithm | Matrices | License |
| ------ | ------- | --------- | -------- | ------- |
| [Clarabel](https://github.com/oxfordcontrol/Clarabel.rs) | ``clarabel`` | Interior point | Sparse | Apache-2.0 |
| [CVXOPT](http://cvxopt.org/) | ``cvxopt`` | Interior point | Dense | GPL-3.0 |
| [DAQP](https://github.com/darnstrom/daqp) | ``daqp`` | Active set | Dense | MIT |
| [ECOS](https://web.stanford.edu/~boyd/papers/ecos.html) | ``ecos`` | Interior point | Sparse | GPL-3.0 |
| [Gurobi](https://www.gurobi.com/) | ``gurobi`` | Interior point | Sparse | Commercial |
| [HiGHS](https://highs.dev/) | ``highs`` | Active set | Sparse | MIT |
| [HPIPM](https://github.com/giaf/hpipm) | ``hpipm`` | Interior point | Dense | BSD-2-Clause |
| [MOSEK](https://mosek.com/) | ``mosek`` | Interior point | Sparse | Commercial |
| NPPro | ``nppro`` | Active set | Dense | Commercial |
| [OSQP](https://osqp.org/) | ``osqp`` | Douglas–Rachford | Sparse | Apache-2.0 |
| [ProxQP](https://github.com/Simple-Robotics/proxsuite) | ``proxqp`` | Augmented Lagrangian | Dense & Sparse | BSD-2-Clause |
| [qpOASES](https://github.com/coin-or/qpOASES) | ``qpoases`` | Active set | Dense | LGPL-2.1 |
| [qpSWIFT](https://qpswift.github.io/) | ``qpswift`` | Interior point | Sparse | GPL-3.0 |
| [quadprog](https://pypi.python.org/pypi/quadprog/) | ``quadprog`` | Goldfarb-Idnani | Dense | GPL-2.0 |
| [SCS](https://www.cvxgrp.org/scs/) | ``scs`` | Douglas–Rachford | Sparse | MIT |
## Metrics
We evaluate QP solvers based on the following metrics:
- **Success rate:** percentage of problems a solver is able to solve on a given test set.
- **Computation time:** time a solver takes to solve a given problem.
- **Optimality conditions:** we evaluate all three [optimality conditions](https://scaron.info/blog/optimality-conditions-and-numerical-tolerances-in-qp-solvers.html):
- **Primal residual:** maximum error on equality and inequality constraints at the returned solution.
- **Dual residual:** maximum error on the dual feasibility condition at the returned solution.
- **Duality gap:** value of the duality gap at the returned solution.
- **Cost error:** difference between the solution cost and the known optimal cost.
### Shifted geometric mean
Each metric (computation time, primal and dual residuals, duality gap) produces a different ranking of solvers for each problem. To aggregate those rankings into a single metric over the whole test set, we use the *shifted geometric mean* (shm), which is a standard to aggregate computation times in [benchmarks for optimization software](#other-benchmarks). This mean has the advantage of being compromised by neither large outliers (as opposed to the arithmetic mean) nor by small outliers (in contrast to the geometric geometric mean). Check out the [references](#references) below for further details.
Here are some intuitive interpretations:
- A solver with a shifted-geometric-mean runtime of $Y$ is $Y$ times slower than the best solver over the test set.
- A solver with a shifted-geometric-mean primal residual $R$ is $R$ times less accurate on equality and inequality constraints than the best solver over the test set.
## Results
The outcome from running a test set is a standardized report comparing [solvers](#solvers) against the different [metrics](#metrics). Here are the results obtained on a reference computer:
| Test set | Results | CPU info |
| -------- | ------- | -------- |
| **GitHub free-for-all** | [Full report](github_ffa/results/github_ffa.md) | Intel(R) Core(TM) i7-6500U CPU @ 2.50GHz |
| **Maros-Meszaros** | [Full report](maros_meszaros/results/maros_meszaros.md) | Intel(R) Core(TM) i7-6500U CPU @ 2.50GHz |
| **Maros-Meszaros dense** | [Full report](maros_meszaros/results/maros_meszaros_dense.md) | Intel(R) Core(TM) i7-6500U CPU @ 2.50GHz |
You can check out results from a variety of machines, and share the reports produced by running the benchmark on your own machine, in the [Results category](https://github.com/qpsolvers/qpsolvers_benchmark/discussions/categories/results) of the discussions forum.
## Limitations
Here are some known areas of improvement for this benchmark:
- *Cold start only:* we don't evaluate warm-start performance for now.
Check out the [issue tracker](https://github.com/qpsolvers/qpsolvers_benchmark/issues) for ongoing works and future improvements.
## Installation
You can install the benchmark and its dependencies in an isolated environment using ``conda``:
```console
conda create -f environment.yaml
conda activate qpsolvers_benchmark
```
Alternatively, you can install the benchmark on your system using ``pip``:
```console
pip install qpsolvers_benchmark
```
By default, the benchmark will run all supported solvers it finds.
## Running the benchmark
Once the benchmark is installed, you will be able to run the ``qpsolvers_benchmark`` command. Provide it with the script corresponding to the [test set](#test-sets) you want to run, followed by a benchmark command such as "run". For instance, let's run the "dense" subset of the Maros-Meszaros test set:
```console
qpsolvers_benchmark maros_meszaros/maros_meszaros_dense.py run
```
You can also run a specific solver, problem or set of solver settings:
```console
qpsolvers_benchmark maros_meszaros/maros_meszaros_dense.py run --solver proxqp --settings default
```
Check out ``qpsolvers_benchmark --help`` for a list of available commands and arguments.
## Plots
The command line ships a ``plot`` command to compare solver performances over a test set for a specific metric. For instance, run:
```console
qpsolvers_benchmark maros_meszaros/maros_meszaros_dense.py plot runtime high_accuracy
```
To generate the following plot:
## Contributing
Contributions to improving this benchmark are welcome. You can for instance propose new problems, or share the runtimes you obtain on your machine. Check out the [contribution guidelines](CONTRIBUTING.md) for details.
## See also
### References
- [How not to lie with statistics: the correct way to summarize benchmark results](https://www.cse.unsw.edu.au/~cs9242/18/papers/Fleming_Wallace_86.pdf): why geometric means should always be used to summarize normalized results.
- [Optimality conditions and numerical tolerances in QP solvers](https://scaron.info/blog/optimality-conditions-and-numerical-tolerances-in-qp-solvers.html): note written while figuring out the ``high_accuracy`` settings of this benchmark.
### Other benchmarks
- [Benchmarks for optimization software](http://plato.asu.edu/bench.html) by Hans Mittelmann, which includes reports on the Maros-Meszaros test set.
- [jrl-qp/benchmarks](https://github.com/jrl-umi3218/jrl-qp/tree/master/benchmarks): benchmark of QP solvers available in C++.
- [osqp\_benchmark](https://github.com/osqp/osqp_benchmarks): benchmark examples for the OSQP solver.
- [proxqp\_benchmark](https://github.com/Simple-Robotics/proxqp_benchmark): benchmark examples for the ProxQP solver.
Raw data
{
"_id": null,
"home_page": null,
"name": "qpsolvers_benchmark",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": "St\u00e9phane Caron <stephane.caron@normalesup.org>",
"keywords": "quadratic programming,benchmark,solvers,numerical optimization",
"author": null,
"author_email": "St\u00e9phane Caron <stephane.caron@normalesup.org>",
"download_url": "https://files.pythonhosted.org/packages/35/cc/1a59bcf0f8ebe85013a106aede7383cc1e559df93a870e5c333c076e961e/qpsolvers_benchmark-1.1.0.tar.gz",
"platform": null,
"description": "# QP solvers benchmark\n\n[](https://github.com/qpsolvers/qpsolvers_benchmark/actions)\n[](https://pypi.org/project/qpsolvers_benchmark/)\n[](https://github.com/qpsolvers/qpsolvers_benchmark/tree/master/CONTRIBUTING.md)\n\nBenchmark for quadratic programming (QP) solvers available in Python.\n\nThe goal of this benchmark is to help users compare and select QP solvers. Its methodology is open to [discussions](https://github.com/qpsolvers/qpsolvers_benchmark/discussions). The benchmark ships standard and community [test sets](#test-sets), as well as a ``qpsolvers_benchmark`` command-line tool to run test sets directly. The main output of the benchmark are standardized reports evaluating all [metrics](#metrics) across all QP solvers available on the test machine. This repository also distributes [results](#results) from running the benchmark on a reference computer.\n\nNew test sets are welcome! The benchmark is designed so that each test-set comes in a standalone directory. Feel free to create a new one and [contribute it](CONTRIBUTING.md) here so that we grow the collection over time.\n\n## Test sets\n\nThe benchmark comes with standard and community test sets to represent different use cases for QP solvers:\n\n| Test set | Problems | Brief description |\n| ------------------------ | -------- | ----------------- |\n| **Maros-Meszaros** | 138 | Standard, designed to be difficult. |\n| **Maros-Meszaros dense** | 62 | Subset of Maros-Meszaros restricted to smaller dense problems. |\n| **GitHub free-for-all** | 12 | Community-built, new problems [are welcome](https://github.com/qpsolvers/qpsolvers_benchmark/issues/new?template=new_problem.md)! |\n\n## Solvers\n\n| Solver | Keyword | Algorithm | Matrices | License |\n| ------ | ------- | --------- | -------- | ------- |\n| [Clarabel](https://github.com/oxfordcontrol/Clarabel.rs) | ``clarabel`` | Interior point | Sparse | Apache-2.0 |\n| [CVXOPT](http://cvxopt.org/) | ``cvxopt`` | Interior point | Dense | GPL-3.0 |\n| [DAQP](https://github.com/darnstrom/daqp) | ``daqp`` | Active set | Dense | MIT |\n| [ECOS](https://web.stanford.edu/~boyd/papers/ecos.html) | ``ecos`` | Interior point | Sparse | GPL-3.0 |\n| [Gurobi](https://www.gurobi.com/) | ``gurobi`` | Interior point | Sparse | Commercial |\n| [HiGHS](https://highs.dev/) | ``highs`` | Active set | Sparse | MIT |\n| [HPIPM](https://github.com/giaf/hpipm) | ``hpipm`` | Interior point | Dense | BSD-2-Clause |\n| [MOSEK](https://mosek.com/) | ``mosek`` | Interior point | Sparse | Commercial |\n| NPPro | ``nppro`` | Active set | Dense | Commercial |\n| [OSQP](https://osqp.org/) | ``osqp`` | Douglas\u2013Rachford | Sparse | Apache-2.0 |\n| [ProxQP](https://github.com/Simple-Robotics/proxsuite) | ``proxqp`` | Augmented Lagrangian | Dense & Sparse | BSD-2-Clause |\n| [qpOASES](https://github.com/coin-or/qpOASES) | ``qpoases`` | Active set | Dense | LGPL-2.1 |\n| [qpSWIFT](https://qpswift.github.io/) | ``qpswift`` | Interior point | Sparse | GPL-3.0 |\n| [quadprog](https://pypi.python.org/pypi/quadprog/) | ``quadprog`` | Goldfarb-Idnani | Dense | GPL-2.0 |\n| [SCS](https://www.cvxgrp.org/scs/) | ``scs`` | Douglas\u2013Rachford | Sparse | MIT |\n\n## Metrics\n\nWe evaluate QP solvers based on the following metrics:\n\n- **Success rate:** percentage of problems a solver is able to solve on a given test set.\n- **Computation time:** time a solver takes to solve a given problem.\n- **Optimality conditions:** we evaluate all three [optimality conditions](https://scaron.info/blog/optimality-conditions-and-numerical-tolerances-in-qp-solvers.html):\n - **Primal residual:** maximum error on equality and inequality constraints at the returned solution.\n - **Dual residual:** maximum error on the dual feasibility condition at the returned solution.\n - **Duality gap:** value of the duality gap at the returned solution.\n- **Cost error:** difference between the solution cost and the known optimal cost.\n\n### Shifted geometric mean\n\nEach metric (computation time, primal and dual residuals, duality gap) produces a different ranking of solvers for each problem. To aggregate those rankings into a single metric over the whole test set, we use the *shifted geometric mean* (shm), which is a standard to aggregate computation times in [benchmarks for optimization software](#other-benchmarks). This mean has the advantage of being compromised by neither large outliers (as opposed to the arithmetic mean) nor by small outliers (in contrast to the geometric geometric mean). Check out the [references](#references) below for further details.\n\nHere are some intuitive interpretations:\n\n- A solver with a shifted-geometric-mean runtime of $Y$ is $Y$ times slower than the best solver over the test set.\n- A solver with a shifted-geometric-mean primal residual $R$ is $R$ times less accurate on equality and inequality constraints than the best solver over the test set.\n\n## Results\n\nThe outcome from running a test set is a standardized report comparing [solvers](#solvers) against the different [metrics](#metrics). Here are the results obtained on a reference computer:\n\n| Test set | Results | CPU info |\n| -------- | ------- | -------- |\n| **GitHub free-for-all** | [Full report](github_ffa/results/github_ffa.md) | Intel(R) Core(TM) i7-6500U CPU @ 2.50GHz |\n| **Maros-Meszaros** | [Full report](maros_meszaros/results/maros_meszaros.md) | Intel(R) Core(TM) i7-6500U CPU @ 2.50GHz |\n| **Maros-Meszaros dense** | [Full report](maros_meszaros/results/maros_meszaros_dense.md) | Intel(R) Core(TM) i7-6500U CPU @ 2.50GHz |\n\nYou can check out results from a variety of machines, and share the reports produced by running the benchmark on your own machine, in the [Results category](https://github.com/qpsolvers/qpsolvers_benchmark/discussions/categories/results) of the discussions forum.\n\n## Limitations\n\nHere are some known areas of improvement for this benchmark:\n\n- *Cold start only:* we don't evaluate warm-start performance for now.\n\nCheck out the [issue tracker](https://github.com/qpsolvers/qpsolvers_benchmark/issues) for ongoing works and future improvements.\n\n## Installation\n\nYou can install the benchmark and its dependencies in an isolated environment using ``conda``:\n\n```console\nconda create -f environment.yaml\nconda activate qpsolvers_benchmark\n```\n\nAlternatively, you can install the benchmark on your system using ``pip``:\n\n```console\npip install qpsolvers_benchmark\n```\n\nBy default, the benchmark will run all supported solvers it finds.\n\n## Running the benchmark\n\nOnce the benchmark is installed, you will be able to run the ``qpsolvers_benchmark`` command. Provide it with the script corresponding to the [test set](#test-sets) you want to run, followed by a benchmark command such as \"run\". For instance, let's run the \"dense\" subset of the Maros-Meszaros test set:\n\n```console\nqpsolvers_benchmark maros_meszaros/maros_meszaros_dense.py run\n```\n\nYou can also run a specific solver, problem or set of solver settings:\n\n```console\nqpsolvers_benchmark maros_meszaros/maros_meszaros_dense.py run --solver proxqp --settings default\n```\n\nCheck out ``qpsolvers_benchmark --help`` for a list of available commands and arguments.\n\n## Plots\n\nThe command line ships a ``plot`` command to compare solver performances over a test set for a specific metric. For instance, run:\n\n```console\nqpsolvers_benchmark maros_meszaros/maros_meszaros_dense.py plot runtime high_accuracy\n```\n\nTo generate the following plot:\n\n\n\n## Contributing\n\nContributions to improving this benchmark are welcome. You can for instance propose new problems, or share the runtimes you obtain on your machine. Check out the [contribution guidelines](CONTRIBUTING.md) for details.\n\n## See also\n\n### References\n\n- [How not to lie with statistics: the correct way to summarize benchmark results](https://www.cse.unsw.edu.au/~cs9242/18/papers/Fleming_Wallace_86.pdf): why geometric means should always be used to summarize normalized results.\n- [Optimality conditions and numerical tolerances in QP solvers](https://scaron.info/blog/optimality-conditions-and-numerical-tolerances-in-qp-solvers.html): note written while figuring out the ``high_accuracy`` settings of this benchmark.\n\n### Other benchmarks\n\n- [Benchmarks for optimization software](http://plato.asu.edu/bench.html) by Hans Mittelmann, which includes reports on the Maros-Meszaros test set.\n- [jrl-qp/benchmarks](https://github.com/jrl-umi3218/jrl-qp/tree/master/benchmarks): benchmark of QP solvers available in C++.\n- [osqp\\_benchmark](https://github.com/osqp/osqp_benchmarks): benchmark examples for the OSQP solver.\n- [proxqp\\_benchmark](https://github.com/Simple-Robotics/proxqp_benchmark): benchmark examples for the ProxQP solver.\n",
"bugtrack_url": null,
"license": null,
"summary": "Benchmark for quadratic programming solvers available in Python.",
"version": "1.1.0",
"project_urls": {
"Changelog": "https://github.com/qpsolvers/qpsolvers_benchmark/blob/master/CHANGELOG.md",
"Source": "https://github.com/qpsolvers/qpsolvers_benchmark",
"Tracker": "https://github.com/qpsolvers/qpsolvers_benchmark/issues"
},
"split_keywords": [
"quadratic programming",
"benchmark",
"solvers",
"numerical optimization"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "b33663a119a88414472b215d534eadec7cbe60f9fcf9dacdf28a2d5fd825b1c8",
"md5": "639aa9cb4961455135093f6c3c7a158c",
"sha256": "ce0c718990ea75331d104afcae1e9d9256de8d9d36aa9d35c002a6f4c1fa52cc"
},
"downloads": -1,
"filename": "qpsolvers_benchmark-1.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "639aa9cb4961455135093f6c3c7a158c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 38062,
"upload_time": "2023-09-08T16:00:07",
"upload_time_iso_8601": "2023-09-08T16:00:07.201769Z",
"url": "https://files.pythonhosted.org/packages/b3/36/63a119a88414472b215d534eadec7cbe60f9fcf9dacdf28a2d5fd825b1c8/qpsolvers_benchmark-1.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "35cc1a59bcf0f8ebe85013a106aede7383cc1e559df93a870e5c333c076e961e",
"md5": "a27ab64ee30b28f849f0777f047dc178",
"sha256": "5532a05ae7e4070318988abc4eec64fa5c2ef87e0031d040dc1ed1c9b2f638c8"
},
"downloads": -1,
"filename": "qpsolvers_benchmark-1.1.0.tar.gz",
"has_sig": false,
"md5_digest": "a27ab64ee30b28f849f0777f047dc178",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 32292,
"upload_time": "2023-09-08T16:00:10",
"upload_time_iso_8601": "2023-09-08T16:00:10.441050Z",
"url": "https://files.pythonhosted.org/packages/35/cc/1a59bcf0f8ebe85013a106aede7383cc1e559df93a870e5c333c076e961e/qpsolvers_benchmark-1.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-09-08 16:00:10",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "qpsolvers",
"github_project": "qpsolvers_benchmark",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "qpsolvers_benchmark"
}
JSON
