Visualizations of Mittelmann benchmarks

Interactive plots showing pairwise time differences for every instance and every solver

Interactive charts comparing the results of Hans Mittelmann’s benchmarks. Each solver can be selected to show pairwise running time factors for every other solver in the respective benchmark. These plots should make browsing the results easier. The score (scaled shifted geometric mean) is recomputed using the reported solving times. We also calculate a “virtual best” or “ensemble” solver that picks the best performance over all solvers for every single problem instance. This might reveal how much potential the individual solvers still have. Please let me know if you have a question or if there is an error.

LPfeas Benchmark (find PD feasible point) (8 Feb 2024)

Choose base solver for comparison:

solver score (as reported) solved of 65
⭐ virtual best 0.74 100%
🥇 COPT-7.1.0 1.00 (1.00) 100%
🥈 Gurobi-11.0.0 1.29 (1.29) 97%
🥉 MindOpt-1.0.0 1.81 (1.81) 97%
📊 MOSEK-10.1.9 2.62 (2.62) 98%
📊 XOPT-0.0.3 5.45 (5.45) 91%
📊 ORTOOLS-9.7 17.09 (17.10) 75%
📊 KNITRO-14.0.0 19.34 (19.30) 74%
📊 HiGHS-1.6.0 19.35 (19.30) 80%
📊 MATLAB-R2023a 27.07 (27.10) 77%
📊 Tulip-0.9.4 66.89 (66.90) 55%
previous benchmarks 🔽

LPopt Benchmark (find optimal basic solution) (8 Feb 2024)

Choose base solver for comparison:

solver score (as reported) solved of 65
⭐ virtual best 0.79 100%
🥇 COPT-7.1.0 1.00 (1.00) 100%
🥈 Gurobi-11.0.0 1.58 (1.58) 97%
🥉 Optverse-0.7.1 1.66 (1.66) 98%
📊 MindOpt-1.0.0 1.89 (1.89) 97%
📊 MOSEK-10.1.9 5.61 (5.61) 80%
📊 XOPT-0.0.3 6.79 (6.79) 78%
📊 HiGHS-1.6.0 17.44 (17.40) 78%
📊 CLP-1.17.7 26.13 (26.10) 62%
📊 MATLAB-R2022b 39.83 (39.80) 65%
📊 Google-GLOP 56.97 (57.00) 51%
📊 SOPLEX-6.0.0 87.20 (87.20) 49%
previous benchmarks 🔽

Large Network-LP Benchmark (commercial vs free) (6 Feb 2024)

Choose base solver for comparison:

solver score (as reported) solved of 25
⭐ virtual best 0.87 100%
🥇 OptVerse-0.7.0 1.00 (1.00) 100%
🥈 MindOpt-1.0.0 1.18 (1.18) 100%
🥉 Gurobi-11.0.0 1.69 (1.69) 100%
📊 COPT-7.1.0 2.16 (2.16) 100%
📊 Clp-1.17.7 6.07 (6.07) 100%
📊 HiGHS-1.6.0 12.65 (12.60) 80%
📊 MATLAB-R2022b 21.28 (21.30) 80%
📊 MOSEK-10.1.9 26.86 (26.90) 88%
📊 QSopt-1.01 35.49 (35.50) 68%
📊 SOPLEX-6.0.0 67.48 (67.40) 64%
previous benchmarks 🔽

The MIPLIB2017 Benchmark Instances - 8 threads (7 Feb 2024)

Choose base solver for comparison:

solver score (as reported) solved of 240
⭐ virtual best 0.66 98%
🥇 Gurobi-11.0.0 1.00 (1.00) 95%
🥈 COPT-7.1.0 1.44 (1.44) 92%
🥉 OptVerse-0.7.0 2.88 (2.88) 84%
📊 MindOpt-1.0.0 4.18 (4.18) 82%
📊 XOPT-0.0.3 7.09 (7.09) 75%
📊 HiGHS-1.6.0 9.98 (9.98) 66%
📊 SCIPC/spx-8.1.0 10.70 (10.70) 62%
📊 SCIP/spx-8.1.0 12.46 (12.50) 55%
📊 CBC-2.10.5 18.42 (18.40) 45%
📊 MATLAB-2023a 37.65 (37.60) 30%
previous benchmarks 🔽

MILP cases that are slightly pathological (5 Feb 2024)

Choose base solver for comparison:

solver score (as reported) solved of 45
⭐ virtual best 0.55 98%
🥇 Gurobi 1.00 (1.00) 96%
🥈 COPT 1.98 (1.98) 87%
🥉 optverse 3.89 (3.89) 80%
📊 MindOpt 16.84 (16.80) 44%
📊 XOPT 18.12 (18.10) 47%
📊 HiGHS 19.42 (19.40) 53%
📊 SCIPC 21.76 (21.80) 56%
📊 SCIP 29.52 (29.50) 49%
📊 CBC 45.02 (45.00) 11%
📊 GLPK 45.63 (45.60) 13%
📊 MATLAB 60.86 (60.90) 4%
previous benchmarks 🔽

Infeasibility Detection for MILP Problems (6 Feb 2024)

Choose base solver for comparison:

solver score (as reported) solved of 32
⭐ virtual best 0.98 94%
🥇 GUROBI-11.0.0 1.00 (1.00) 94%
🥈 COPT-7.1.0 1.37 (1.37) 94%
🥉 OptVerse-0.7.0 3.00 (3.00) 88%
📊 XOPT-0.0.3 7.34 (7.34) 78%
📊 MindOpt-1.0.0 7.81 (7.81) 84%
📊 HiGHS-1.6.0 8.18 (8.18) 81%
📊 SCIPC-8.1.0 8.77 (8.77) 81%
📊 SCIP-8.1.0 9.26 (9.26) 78%
📊 CBC-2.10.5 22.51 (22.50) 62%
📊 MATLAB-2023a 44.55 (44.50) 50%
previous benchmarks 🔽

Several SDP-codes on sparse and other SDP problems (8 Feb 2024)

Choose base solver for comparison:

solver score (as reported) solved of 75
⭐ virtual best 0.61 100%
🥇 COPT-7.1.0 1.00 (1.00) 100%
🥈 MindOpt-1.0.0 1.45 (1.45) 100%
🥉 MOSEK-10.1.23 3.67 (3.67) 97%
📊 SDPT3-4.0 5.18 (5.18) 92%
📊 CSDP-6.2.0 5.25 (5.25) 93%
📊 HDSDP-1.0.0 7.91 (7.91) 93%
📊 SDPA-7.4.2 10.52 (10.50) 81%
📊 SeDuMi-1.3.5 29.06 (29.10) 83%
previous benchmarks 🔽

Large Second Order Cone Benchmark (7 Feb 2024)

Choose base solver for comparison:

solver score (as reported) solved of 18
⭐ virtual best 0.80 100%
🥇 COPT-7.1.0 1.00 (1.00) 100%
🥈 MOSEK-10.1.9 1.06 (1.06) 100%
🥉 Gurobi-11.0.0 1.11 (1.11) 100%
📊 KNITRO-14.0.0 9.24 (9.24) 94%
📊 ECOS-2.0.4 97.22 (97.20) 61%
previous benchmarks 🔽

Mixed-integer SOCP Benchmark (9 Feb 2024)

Choose base solver for comparison:

solver score (as reported) solved of 47
⭐ virtual best 0.97 100%
🥇 Gurobi-11.0.0 1.00 (1.00) 100%
🥈 COPT-7.1.0 2.20 (2.16) 98%
🥉 MOSEK-10.1.9 10.68 (9.00) 77%
📊 SCIP-8.1.0 26.84 (21.10) 66%
previous benchmarks 🔽

Binary Non-Convex QPLIB Benchmark (11 Feb 2024)

Choose base solver for comparison:

solver score (as reported) solved of 97
⭐ virtual best 0.70 100%
🥇 Gurobi-11.0.0 1.00 (1.00) 97%
🥈 SHOT-1.1 1.48 (1.48) 88%
🥉 OCTERACT-4.7.1 1.69 (1.69) 96%
📊 Baron-24.1.30 8.44 (8.44) 67%
📊 SCIP-8.1.0 38.80 (38.80) 37%
📊 ANTIGONE-1.1 81.19 (81.20) 16%
📊 COUENNE-0.5 96.45 (96.40) 6%
previous benchmarks 🔽

Nonconvex QUBO-QPLIB Benchmark (10 Feb 2024)

Choose base solver for comparison:

solver score (as reported) solved of 23
⭐ virtual best 0.87 70%
🥇 QuBowl 1.00 (1.00) 65%
🥈 Gurobi-11.0.0 1.30 (1.30) 57%
🥉 OCTERACT-4.7.1 1.74 (1.74) 52%
📊 SHOT-1.1 1.76 (1.76) 52%
📊 Baron-24.1.30 1.82 (1.82) 52%
📊 McSparse-2.0 2.36 (2.36) 52%
📊 Biqbin 5.40 (5.40) 39%
📊 SCIP-8.1 5.46 (5.46) 35%
previous benchmarks 🔽

Discrete Non-Convex QPLIB Benchmark (non-binary) (22 Feb 2024)

Choose base solver for comparison:

solver score (as reported) solved of 112
⭐ virtual best 0.30 97%
🥇 Gurobi-11.0.0 1.00 (1.00) 83%
🥈 SHOT-1.1 2.23 (2.23) 72%
🥉 OCTERACT-4.7.1 5.47 (5.47) 71%
📊 Baron-24.1.30 6.19 (6.19) 59%
📊 SCIP-8.1.0 24.00 (24.00) 37%
📊 ANTIGONE-1.1 43.22 (43.20) 26%
📊 MINOTAUR-0.3.0 51.09 (51.10) 13%
previous benchmarks 🔽

Continuous Non-Convex QPLIB Benchmark (22 Feb 2024)

Choose base solver for comparison:

solver score (as reported) solved of 67
⭐ virtual best 0.21 96%
🥇 GUROBI-11.0.0 1.00 (1.00) 66%
🥈 OCTERACT-4.7.1 1.55 (1.55) 69%
🥉 Baron-24.1.30 2.60 (2.60) 49%
📊 ANTIGONE-1.1 5.73 (5.73) 42%
📊 MINOTAUR-0.3.0 9.99 (9.99) 19%
📊 SCIP-8.1.0 10.46 (10.50) 21%
📊 COUENNE-0.5 13.25 (13.30) 12%
📊 RAPOSa-4.0.2 16.78 (16.80) 6%
previous benchmarks 🔽

Convex Continuous QPLIB Benchmark (ext) (7 Feb 2024)

Choose base solver for comparison:

solver score (as reported) solved of 42
⭐ virtual best 0.73 100%
🥇 COPT-7.1.0 1.00 (1.00) 100%
🥈 OptVerse-0.7.0 1.28 (1.28) 98%
🥉 KNITRO-14.0.0 1.49 (1.49) 98%
📊 Gurobi-11.0.0 1.59 (1.59) 98%
📊 MOSEK-10.1.21 1.60 (1.60) 98%
📊 MindOpt-1.0.0 5.74 (5.43) 81%
📊 IPOPT-3.14.5 8.95 (8.95) 83%
previous benchmarks 🔽

Convex Discrete QPLIB Benchmark (11 Feb 2024)

Choose base solver for comparison:

solver score (as reported) solved of 31
⭐ virtual best 0.68 87%
🥇 GUROBI-11.0.0 1.00 (1.00) 81%
🥈 COPT-7.1.0 1.00 (1.00) 77%
🥉 Shot-1.1 1.18 (1.18) 81%
📊 OCTERACT-4.7.1 2.44 (2.44) 77%
📊 Baron-24.1.30 3.45 (3.45) 65%
📊 MOSEK-10.1.23 10.05 (10.00) 58%
📊 KNITRO-14.0.0 13.63 (13.60) 52%
📊 SCIP-8.1.0 26.71 (26.70) 42%
📊 MNTAUR 42.32 (42.30) 45%
📊 Bonmin-1.8.7 54.64 (54.60) 23%
previous benchmarks 🔽

Mixed Integer Nonlinear Programming Benchmark (MINLPLIB) (7 Feb 2024)

Choose base solver for comparison:

solver score (as reported) solved of 87
⭐ virtual best 0.41 100%
🥇 OCTERACT 1.00 (1.00) 100%
🥈 BARON 2.42 (2.40) 89%
🥉 SHOT 7.17 (6.20) 62%
📊 SCIP 9.87 (9.90) 75%
📊 LINDO 32.85 (32.80) 48%
📊 ANTIGONE 39.32 (39.30) 61%
📊 COUENNE 89.78 (89.80) 28%
previous benchmarks 🔽

MPEC Benchmark (Math. Progr. w. Equilibrium Constraints) (9 Feb 2024)

Choose base solver for comparison:

solver score (as reported) solved of 29
⭐ virtual best 1.00 93%
🥇 KNITRO-14.0 1.00 (1.00) 93%
🥈 filter-MPEC 16.90 (16.90) 62%
🥉 LOQO-7.03 37.10 (37.10) 21%
previous benchmarks 🔽