Visualizations of Mittelmann benchmarks

Interactive plots showing pairwise time differences for every instance and every solver

Interactive charts comparing the results of Hans Mittelmann’s benchmarks. Each solver can be selected to show pairwise running time factors for every other solver in the respective benchmark. These plots should make browsing the results easier. The score (scaled shifted geometric mean) is recomputed using the reported solving times. Please let me know if you have a question or if there is an error.

Benchmark of Simplex LP solvers (26 May 2022)

Choose base solver for comparison:

solver score (as reported) solved of 56
πŸ₯‡ COPT-4.0.4 1.00 (1.00) 100%
πŸ₯ˆ MindOpt-0.18.2 1.16 (1.16) 98%
πŸ₯‰ Gurobi-9.5.0 1.43 (1.43) 100%
πŸ“Š Optverse-0.2.1 4.23 (4.23) 70%
πŸ“Š CLP-1.17.7 9.52 (9.52) 77%
πŸ“Š HiGHS-1.2.1 14.21 (14.20) 79%
πŸ“Š MOSEK-9.3.18 16.80 (16.80) 75%
πŸ“Š Google-GLOP 27.15 (27.20) 57%
πŸ“Š MATLAB-R2020b 28.22 (28.20) 66%
πŸ“Š SOPLEX-6.0.0 44.93 (44.90) 61%
πŸ“Š GLPK-5.0 87.37 (87.40) 50%
previous benchmarks πŸ”½

Benchmark of Barrier LP solvers (25 May 2022)

Choose base solver for comparison:

solver score (as reported) solved of 49
πŸ₯‡ COPT-4.0.4 1.00 (1.00) 100%
πŸ₯ˆ Gurobi-9.5.1 1.32 (1.32) 98%
πŸ₯‰ MindOpt-0.18.2 2.33 (2.33) 98%
πŸ“Š MOSEK-9.3.20 5.11 (5.11) 94%
πŸ“Š PDLP$ 14.20 (14.20) 86%
πŸ“Š KNITRO-13.0.0 15.31 (15.30) 80%
πŸ“Š HiGHS-1.2.2 21.87 (19.90) 86%
πŸ“Š MATLAB-R2020b 49.58 (40.50) 73%
πŸ“Š Tulip-0.9.3 55.38 (55.40) 67%
πŸ“Š CLP-1.17.7 78.26 (78.30) 71%
previous benchmarks πŸ”½

Large Network-LP Benchmark (commercial vs free) (17 Apr 2022)

Choose base solver for comparison:

solver score (as reported) solved of 25
πŸ₯‡ Gurobi-9.5.1 1.00 (1.00) 100%
πŸ₯ˆ MindOpt-0.18.1 1.07 (1.07) 100%
πŸ₯‰ COPT-4.0.4 1.26 (1.26) 100%
πŸ“Š Clp-1.17.7 3.56 (3.56) 100%
πŸ“Š HiGHS-1.1.1 7.66 (7.66) 80%
πŸ“Š MATLAB-R2020b 12.97 (13.00) 80%
πŸ“Š MOSEK-9.3.6 13.71 (13.70) 84%
πŸ“Š QSopt-1.01 20.80 (20.80) 68%
πŸ“Š SOPLEX-6.0.0 39.55 (39.60) 64%
previous benchmarks πŸ”½

The MIPLIB2017 Benchmark Instances - 8 threads (8 May 2022)

Choose base solver for comparison:

solver score (as reported) solved of 240
πŸ₯‡ Gurobi-9.5.0 1.00 (1.00) 93%
πŸ₯ˆ COPT-4.0.1 3.53 (3.53) 77%
πŸ₯‰ SCIPC/cpx-8.0.0 8.39 (8.39) 62%
πŸ“Š FSCIP/spx-7.0.0 8.69 (8.69) 61%
πŸ“Š HiGHS-1.2.2 9.15 (9.15) 62%
πŸ“Š SCIP/spx-8.0.0 10.15 (10.20) 55%
πŸ“Š CBC-2.10.5 14.53 (14.50) 45%
previous benchmarks πŸ”½

MILP cases that are slightly pathological (27 Apr 2022)

Choose base solver for comparison:

solver score (as reported) solved of 45
πŸ₯‡ GUROBI-9.5.0 1.00 (1.00) 98%
πŸ₯ˆ COPT-4.0.0 6.72 (6.72) 64%
πŸ₯‰ FSCIP-7.0.0 13.75 (13.80) 53%
πŸ“Š SCIPC-8.0.0 16.86 (16.90) 56%
πŸ“Š HiGHS-1.2.2 17.28 (17.30) 56%
πŸ“Š SCIP-8.0.0 23.02 (23.00) 38%
πŸ“Š CBC-2.10.7 34.68 (34.70) 11%
πŸ“Š GLPK-5.0 35.15 (35.20) 13%
πŸ“Š MATLAB-2020a 42.36 (42.40) 7%
previous benchmarks πŸ”½

Infeasibility Detection for MILP Problems (25 Apr 2022)

Choose base solver for comparison:

solver score (as reported) solved of 32
πŸ₯‡ GUROBI-9.5.0 1.00 (1.00) 91%
πŸ₯ˆ COPT-4.0.0 2.76 (2.76) 84%
πŸ₯‰ SCIPC-8.0.0 4.68 (4.68) 81%
πŸ“Š SCIP-8.0.0 6.06 (6.06) 78%
πŸ“Š FSCIP-7.0.0 7.37 (7.37) 75%
πŸ“Š HiGHS-1.2.2 8.12 (8.12) 75%
πŸ“Š CBC-2.10.5 14.80 (14.80) 62%
πŸ“Š MATLAB-2020b 23.00 (23.00) 47%
previous benchmarks πŸ”½

Large Second Order Cone Benchmark (6 May 2022)

Choose base solver for comparison:

solver score (as reported) solved of 18
πŸ₯‡ MOSEK-9.3.20 1.00 (1.00) 100%
πŸ₯ˆ Gurobi-9.5.0 1.07 (1.07) 100%
πŸ₯‰ COPT-4.0.0 1.21 (1.21) 100%
πŸ“Š KNITRO-13.0.0 9.47 (9.47) 83%
πŸ“Š ECOS-2.0.4 78.11 (78.10) 33%
previous benchmarks πŸ”½

Mixed-integer SOCP Benchmark (5 May 2022)

Choose base solver for comparison:

solver score (as reported) solved of 47
πŸ₯‡ Gurobi-9.5.1 1.00 (1.00) 100%
πŸ₯ˆ MOSEK-9.3.20 13.25 (13.20) 68%
πŸ₯‰ SCIP-8.0.0 20.43 (20.40) 66%
previous benchmarks πŸ”½

Binary Non-Convex QPLIB Benchmark (6 May 2022)

Choose base solver for comparison:

solver score (as reported) solved of 91
πŸ₯‡ Gurobi-9.5.0 1.00 (1.00) 95%
πŸ₯ˆ OCTERACT-4.3.0 2.15 (2.15) 95%
πŸ₯‰ Baron-22.1.8 13.41 (13.40) 45%
πŸ“Š FSCIP-7.0.0 35.46 (35.50) 37%
πŸ“Š SCIP-8.0.0 35.83 (35.80) 33%
πŸ“Š ANTIGONE-1.1 49.30 (49.30) 25%
πŸ“Š MINOTAUR-0.2.1 64.69 (64.70) 8%
previous benchmarks πŸ”½

Discrete Non-Convex QPLIB Benchmark (non-binary) (30 Apr 2022)

Choose base solver for comparison:

solver score (as reported) solved of 86
πŸ₯‡ Gurobi-9.5.0 1.00 (1.00) 85%
πŸ₯ˆ OCTERACT-4.3.0 7.97 (7.97) 52%
πŸ₯‰ SCIP-8.0.0 23.34 (23.30) 43%
πŸ“Š ANTIGONE-1.1 42.87 (42.90) 34%
πŸ“Š Baron-22.3.21 44.49 (44.50) 31%
πŸ“Š COUENNE-0.5 87.93 (87.90) 9%
πŸ“Š MINOTAUR-0.2.1 99.61 (99.60) 5%
previous benchmarks πŸ”½

Continuous Non-Convex QPLIB Benchmark (3 May 2022)

Choose base solver for comparison:

solver score (as reported) solved of 65
πŸ₯‡ GUROBI-9.5.0 1.00 (1.00) 65%
πŸ₯ˆ OCTERACT-4.3.0 1.94 (1.94) 52%
πŸ₯‰ ANTIGONE-1.1 4.76 (4.76) 43%
πŸ“Š Baron-22.1.8 6.55 (6.55) 26%
πŸ“Š SCIP-8.0.0 9.89 (9.89) 20%
πŸ“Š COUENNE-0.5 11.31 (11.30) 12%
πŸ“Š MINOTAUR-0.2.1 11.84 (11.80) 9%
previous benchmarks πŸ”½

Convex Discrete QPLIB Benchmark (25 Apr 2022)

Choose base solver for comparison:

solver score (as reported) solved of 31
πŸ₯‡ GUROBI-9.5.0 1.00 (1.00) 74%
πŸ₯ˆ OCTERACT-4.3.0 3.36 (3.36) 81%
πŸ₯‰ Baron-22.1.8 7.22 (7.22) 55%
πŸ“Š MOSEK-9.3.11 11.72 (11.70) 42%
πŸ“Š SCIP-8.0.0 13.06 (13.10) 42%
πŸ“Š Bonmin-1.8.7 17.78 (17.80) 32%
πŸ“Š KNITRO-12.3.0 18.13 (18.10) 29%
πŸ“Š MINOTAUR-0.2.1 29.48 (29.50) 35%
πŸ“Š ANTIGONE-1.1 46.86 (46.90) 6%
πŸ“Š Shot-1.0 54.86 (54.90) 3%
previous benchmarks πŸ”½

Convex Continuous QPLIB Benchmark (11 Apr 2022)

Choose base solver for comparison:

solver score (as reported) solved of 32
πŸ₯‡ COPT-4.0.0 1.00 (1.00) 100%
πŸ₯ˆ KNITRO-13.0.0 1.54 (1.54) 100%
πŸ₯‰ MOSEK-9.3.12 1.98 (1.98) 97%
πŸ“Š Gurobi-9.5.0 2.01 (2.01) 94%
πŸ“Š IPOPT-3.14.5 7.31 (7.31) 91%
previous benchmarks πŸ”½