MIP Solver Benchmarking and Tuning

Solver selection and tuning decisions are routinely made on folklore — vendor slides, five-year-old blog posts, one lucky test instance. This page is the craft of doing it honestly: what the public benchmarks do and don't tell you (especially after the 2024 vendor withdrawals), why MIP performance is intrinsically noisy, how to run a benchmark on your own instances that survives scrutiny, and the parameter tuning that actually moves solve times.

The public benchmarks and their new limits

Mittelmann's benchmarks (Arizona State) were the field's independent scoreboard for two decades — standardized instance sets, shifted geometric means of solve times, all major solvers. Their authority is precisely why the 2024 exits matter: Gurobi withdrew in August 2024, MindOpt in December — so the public MIP tables no longer include the presumptive leader, and vendor-vs-vendor claims have retreated into marketing material. What remains public and useful: the open-source solvers and COPT/Xpress/CPLEX entries (COPT notably topped the LP table), and MIPLIB 2017, the community's curated instance library, whose "benchmark" subset defines difficulty tiers every solver paper reports against.

Reading any solver benchmark, check three things: the metric (shifted geometric mean of times, with timeouts penalized — arithmetic means are dominated by timeouts; per-instance win counts tell a different story than means), the time limit (rankings reorder between 1-hour and 10-second budgets — the short-budget ranking is the one interactive applications should care about), and instance provenance (public libraries over-represent hard academic structures and under-represent your industry's easy-but-huge models).

Performance variability: the noise floor

MIP solve times are chaotically sensitive: permuting rows/columns, changing the random seed, or moving between machines routinely shifts solve time 2–10x on the same instance — documented behavior, not defect. Branch-and-bound amplifies tiny tie-breaking differences into different trees. Consequences for practice: never conclude anything from one run — benchmark with multiple seeds (Gurobi/SCIP expose seed parameters for exactly this) and compare distributions; treat any single-run "solver A is 30% faster" as noise; and expect production variance day-to-day on the same model family — the reason time limits and gap targets, not point estimates, define the operational contract.

Benchmarking on your instances: the method

1. Collect 20–50 representative instances spanning your real size/difficulty mix — export LP/MPS files from production runs, including the ugly ones.
2. Fix the contract: your production time limit and MIP gap, identical hardware, same thread count per solver.
3. Run ≥3 seeds per instance per solver; record time-to-target-gap, achieved gap at timeout, and time-to-first-feasible (for interactive uses, often the metric that matters most).
4. Aggregate with shifted geometric means and performance profiles (fraction of instances solved within factor τ of the best) — both standard, both resistant to outlier domination.
5. Get trial licenses — every commercial vendor grants full-featured evaluations; a benchmark without the paid contenders answers only half the question. (What the licenses cost and buy.)

Parameter tuning: where the free speed is

Defaults are compromise settings across all problem classes; your class deserves better. The levers, in observed order of payoff:

• Emphasis/focus settings — one knob declaring your intent: prove optimality vs find good solutions fast (MIPFocus in Gurobi, emphasis settings in SCIP/CPLEX). The single highest-leverage parameter.
• Presolve aggressiveness — usually helps; occasionally destroys structure your model needs. Toggling presolve off is also a diagnostic: a model that gets faster without presolve has a formulation story worth investigating.
• Cuts and heuristics intensity — more cuts tighten bounds but cost node throughput; more heuristics find incumbents sooner. Tune toward whichever side of the gap (bound vs incumbent) is lagging in your logs — the solver log tells you which.
• Threads — MIP parallelism saturates fast; 8–16 threads is typically the knee, and more can hurt via memory bandwidth. Never benchmark solver A at 32 threads against B at 4.
• Automated tuners — Gurobi's tune, CPLEX's tuning tool: give them representative instances and hours, harvest the parameter file. Worth one overnight run on any recurring model family.

The ceiling on tuning is real: parameters buy tens of percent; reformulation buys orders of magnitude. When the log shows a weak LP relaxation (large root gap), no parameter fixes it — the model does. Tune after tightening, not instead of it.

See Also

• Open-Source Optimization Solvers · Commercial MIP Solvers — the contenders
• MILP Formulation Patterns — the bigger lever than any parameter
• Running Optimization Models in Production — the operational contract benchmarks inform