Visualizations of Mittelmann benchmarks

Interactive plots showing pairwise time differences for every instance and every solver

Interactive charts comparing the results of Hans Mittelmann’s benchmarks. Each solver can be selected to show pairwise running time factors for every other solver in the respective benchmark. These plots should make browsing the results easier. The score (scaled shifted geometric mean) is recomputed using the reported solving times. We also calculate a “virtual best” or “ensemble” solver that picks the best performance over all solvers for every single problem instance. This might reveal how much potential the individual solvers still have. Please let me know if you have a question or if there is an error.

Benchmark of Simplex LP solvers (29 Sep 2022)

Choose base solver for comparison:

solver score (as reported) solved of 59
⭐ virtual best 0.67 100%
🥇 MindOpt-0.21.0 1.00 (1.00) 100%
🥈 COPT-5.0.4 1.03 (1.03) 102%
🥉 Gurobi-9.5.0 1.71 (1.71) 100%
📊 Optverse-0.2.6 2.64 (2.64) 90%
📊 CLP-1.17.7 10.29 (10.30) 73%
📊 HiGHS-1.2.1 14.44 (14.40) 80%
📊 MATLAB-R2022b 16.18 (16.20) 75%
📊 MOSEK-10.0.18 16.65 (16.60) 78%
📊 Google-GLOP 27.78 (27.80) 54%
📊 SOPLEX-6.0.0 44.80 (44.80) 58%
📊 GLPK-5.0 84.21 (84.20) 47%
previous benchmarks 🔽

Benchmark of Barrier LP solvers (29 Sep 2022)

Choose base solver for comparison:

solver score (as reported) solved of 51
⭐ virtual best 0.64 100%
🥇 COPT-5.0.0 1.00 (1.00) 100%
🥈 Gurobi-9.5.1 1.31 (1.31) 98%
🥉 MindOpt-0.18.2 2.39 (2.39) 96%
📊 MOSEK-10.0.18 4.00 (4.00) 92%
📊 PDLP$ 12.09 (12.10) 86%
📊 KNITRO-13.0.0 14.47 (14.40) 76%
📊 HiGHS-1.2.2 20.35 (18.60) 82%
📊 MATLAB-R2022b 37.90 (34.30) 69%
📊 Tulip-0.9.4 52.37 (52.40) 63%
📊 CLP-1.17.7 69.14 (69.10) 69%
previous benchmarks 🔽

Large Network-LP Benchmark (commercial vs free) (24 Aug 2022)

Choose base solver for comparison:

solver score (as reported) solved of 25
⭐ virtual best 0.92 100%
🥇 MindOpt-0.18.6 1.00 (1.00) 100%
🥈 OptVerse-0.2.6 1.19 (1.19) 100%
🥉 Gurobi-9.5.1 1.32 (1.32) 100%
📊 COPT-5.0.0 1.68 (1.68) 100%
📊 Clp-1.17.7 4.71 (4.71) 100%
📊 HiGHS-1.1.1 10.15 (10.20) 80%
📊 MATLAB-R2020b 17.17 (17.20) 80%
📊 MOSEK-10.0.16 20.31 (20.30) 92%
📊 QSopt-1.01 27.55 (27.50) 68%
📊 SOPLEX-6.0.0 52.38 (52.40) 64%
previous benchmarks 🔽

The MIPLIB2017 Benchmark Instances - 8 threads (30 Jun 2022)

Choose base solver for comparison:

solver score (as reported) solved of 240
⭐ virtual best 0.81 96%
🥇 Gurobi-9.5.0 1.00 (1.00) 93%
🥈 COPT-5.0.0 2.34 (2.34) 81%
🥉 SCIPC/spx-8.0.0 7.95 (7.95) 63%
📊 HiGHS-1.2.2 9.15 (9.15) 62%
📊 SCIP/spx-8.0.0 9.72 (9.72) 57%
📊 CBC-2.10.5 14.53 (14.50) 45%
previous benchmarks 🔽

MILP cases that are slightly pathological (25 Jul 2022)

Choose base solver for comparison:

solver score (as reported) solved of 45
⭐ virtual best 0.80 98%
🥇 GUROBI-9.5.0 1.00 (1.00) 98%
🥈 COPT-5.0.0 3.99 (3.99) 76%
🥉 HiGHS-1.2.2 17.28 (17.30) 56%
📊 SCIPC-8.0.0 17.33 (17.30) 51%
📊 SCIP-8.0.0 22.21 (22.20) 42%
📊 CBC-2.10.7 34.68 (34.70) 11%
📊 GLPK-5.0 35.15 (35.20) 13%
📊 MATLAB-2020a 42.36 (42.40) 7%
previous benchmarks 🔽

Infeasibility Detection for MILP Problems (24 Jul 2022)

Choose base solver for comparison:

solver score (as reported) solved of 32
⭐ virtual best 0.66 94%
🥇 GUROBI-9.5.0 1.00 (1.00) 91%
🥈 COPT-5.0.0 1.39 (1.39) 91%
🥉 SCIPC-8.0.0 4.73 (4.68) 81%
📊 SCIP-8.0.0 6.06 (6.06) 78%
📊 HiGHS-1.2.2 8.12 (8.12) 75%
📊 CBC-2.10.5 14.80 (14.80) 62%
📊 MATLAB-2020b 23.00 (23.00) 47%
previous benchmarks 🔽

Several SDP-codes on sparse and other SDP problems (13 Sep 2022)

Choose base solver for comparison:

solver score (as reported) solved of 76
⭐ virtual best 0.36 99%
🥇 COPT-5.0 1.00 (1.00) 97%
🥈 MOSEK-10.0.18 1.64 (1.64) 95%
🥉 SDPT3-4.0 2.13 (2.17) 91%
📊 CSDP-6.2.0 2.16 (2.19) 92%
📊 HDSDP-0.9.2 3.50 (3.57) 92%
📊 SDPA-7.4.2 4.29 (4.40) 80%
📊 SeDuMi-1.3.5 11.69 (12.20) 82%
previous benchmarks 🔽

Large Second Order Cone Benchmark (19 Sep 2022)

Choose base solver for comparison:

solver score (as reported) solved of 18
⭐ virtual best 0.77 100%
🥇 MOSEK-10.0.18 1.00 (1.00) 100%
🥈 Gurobi-9.5.0 1.14 (1.14) 100%
🥉 COPT-5.0.0 1.16 (1.16) 100%
📊 KNITRO-13.0.0 10.06 (10.10) 83%
📊 ECOS-2.0.4 83.02 (83.00) 33%
previous benchmarks 🔽

Mixed-integer SOCP Benchmark (31 Aug 2022)

Choose base solver for comparison:

solver score (as reported) solved of 47
⭐ virtual best 1.00 100%
🥇 Gurobi-9.5.1 1.00 (1.00) 100%
🥈 MOSEK-10.0.18 11.84 (11.80) 72%
🥉 SCIP-8.0.0 20.43 (20.40) 66%
previous benchmarks 🔽

Binary Non-Convex QPLIB Benchmark (5 Sep 2022)

Choose base solver for comparison:

solver score (as reported) solved of 96
⭐ virtual best 0.67 100%
🥇 Gurobi-9.5.2 1.00 (1.00) 93%
🥈 OCTERACT-4.5.0 1.65 (1.65) 96%
🥉 Baron-22.9.1 6.65 (6.65) 57%
📊 SCIP-8.0.0 33.75 (33.80) 36%
📊 ANTIGONE-1.1 68.43 (68.40) 17%
📊 COUENNE-0.5 81.43 (81.40) 6%
previous benchmarks 🔽

Discrete Non-Convex QPLIB Benchmark (non-binary) (10 Sep 2022)

Choose base solver for comparison:

solver score (as reported) solved of 99
⭐ virtual best 0.37 88%
🥇 Gurobi-9.5.0 1.00 (1.00) 83%
🥈 SCIP-8.0.0 9.33 (20.90) 37%
🥉 Baron-22.9.1 14.83 (33.30) 31%
📊 ANTIGONE-1.1 15.82 (35.60) 29%
📊 MINOTAUR-0.3.0 19.12 (43.10) 15%
📊 COUENNE-0.5 29.53 (66.80) 8%
previous benchmarks 🔽

Continuous Non-Convex QPLIB Benchmark (10 Sep 2022)

Choose base solver for comparison:

solver score (as reported) solved of 68
⭐ virtual best 0.19 94%
🥇 GUROBI-9.5.0 1.00 (1.00) 62%
🥈 OCTERACT-4.5.0 1.00 (1.25) 72%
🥉 ANTIGONE-1.1 3.93 (4.90) 41%
📊 Baron-22.9.1 4.62 (5.76) 28%
📊 MINOTAUR-0.3.0 5.57 (6.95) 22%
📊 SCIP-8.0.0 7.89 (9.85) 19%
📊 COUENNE-0.5 8.98 (11.20) 12%
previous benchmarks 🔽

Convex Continuous QPLIB Benchmark (29 Aug 2022)

Choose base solver for comparison:

solver score (as reported) solved of 32
⭐ virtual best 0.87 100%
🥇 COPT-5.0.0 1.00 (1.00) 100%
🥈 MOSEK-10.0.18 1.35 (1.35) 100%
🥉 KNITRO-13.0.0 1.55 (1.55) 100%
📊 Gurobi-9.5.0 2.03 (2.03) 94%
📊 IPOPT-3.14.5 7.40 (7.40) 91%
previous benchmarks 🔽

Convex Discrete QPLIB Benchmark (25 Sep 2022)

Choose base solver for comparison:

solver score (as reported) solved of 31
⭐ virtual best 0.44 94%
🥇 GUROBI-9.5.2 1.00 (1.00) 77%
🥈 OCTACT 3.38 (3.38) 74%
🥉 Shot-1.1 5.38 (5.38) 48%
📊 MOSEK-10.0.18 5.73 (5.73) 61%
📊 KNITRO-13.1.0 8.23 (8.23) 52%
📊 Baron-22.9.1 8.69 (8.69) 58%
📊 SCIP-8.0.0 22.69 (22.70) 39%
📊 MNTAUR 26.00 (26.00) 45%
📊 Bonmin-1.8.7 33.57 (33.60) 23%
previous benchmarks 🔽

Mixed Integer Nonlinear Programming Benchmark (MINLPLIB) (20 Sep 2022)

Choose base solver for comparison:

solver score (as reported) solved of 87
⭐ virtual best 0.22 95%
🥇 OCTERACT 1.00 (0.85) 77%
🥈 SCIP 1.07 (0.91) 74%
🥉 BARON 1.17 (1.00) 79%
📊 ANTIGONE 4.03 (3.43) 61%
📊 LINDO 5.30 (4.51) 43%
📊 COUENNE 9.12 (7.77) 28%
previous benchmarks 🔽

MPEC Benchmark (Math. Progr. w. Equilibrium Constraints) (12 Apr 2022)

Choose base solver for comparison:

solver score (as reported) solved of 29
⭐ virtual best 1.00 83%
🥇 KNITRO-13.0 1.00 (1.00) 83%
🥈 filter-MPEC 7.82 (8.90) 62%
🥉 LOQO-7.03 16.69 (19.50) 21%