Ijoc Methods

Method & Experimental Design (ijoc-methods)

When to trigger

• The algorithm is settled but the experimental protocol (instances, baselines, tuning, hardware, metrics) is not designed
• You are about to run experiments and want to avoid a protocol a referee will reject after the fact
• A referee challenges the fairness of a comparison (asymmetric tuning, mismatched time limits, weak baseline)
• You must choose what to measure so the computational claim is actually supported

Designing an IJOC-grade computational experiment

At IJOC the experiment is the evidence, so it is held to a high methodological standard. Design it before running it, around five pillars. Getting these right up front is cheaper than re-running after an R&R.

1. Instances. Use public, standard benchmark sets wherever they exist (e.g., MIPLIB-class libraries, TSPLIB-style, established generators) so results are comparable and not cherry-picked. If you must generate instances, document the generator, the parameter ranges, and the seeds, and deposit them. Report the size distribution; do not test only on the sizes where you win.
2. Baselines. Compare against the strongest available method, not a strawman: the leading published algorithm and, for exact methods, a current commercial/open solver (CPLEX/Gurobi/SCIP/HiGHS) at default and tuned. A win over a weak baseline is not a win.
3. Fair tuning. Tune your method and the baselines with the same budget on a disjoint tuning set, then report on a held-out test set. Disclose the tuning procedure; never tune on the test instances. Asymmetric tuning is the single most common fatal flaw.
4. Hardware & time. Report CPU/GPU, cores used, memory, language/solver versions, and the time limit. Compare at equal wall-clock or equal hardware; if you must compare across machines, normalize and say how. State whether runs are single- or multi-threaded.
5. Metrics & statistics. Pick metrics that match the claim — solve time, optimality gap, primal/dual bound at time limit, solution quality, number solved, generalization error, estimator variance. Run multiple seeds for stochastic methods and report dispersion, not just means. Plan the statistical test (Wilcoxon signed-rank, performance profiles) now.

Method choice should follow the structure, not fashion

Choose the method because the problem structure warrants it: decomposition when the model is block-angular; column generation when columns are exponential but priceable; a learned heuristic when many similar instances are solved repeatedly; variance reduction when the simulation estimand is rare-event-like. "We used deep learning because it is popular" invites the reviewer to ask what it buys over a tuned classical baseline — so include that baseline.

Common protocol traps by archetype

The fair-comparison standard bites differently across archetypes, and each has a signature trap:

• Exact methods: comparing your tuned cuts against a solver with cuts/presolve turned off — disclose every solver parameter you changed, and justify it.
• Heuristics: reporting the best of many runs as "the result." Report a fixed budget and the distribution; the comparison is run-to-run, not best-to-best.
• ML-for-OR: training and testing on instances from the same generator seed, so "generalization" is memorization. Hold out an out-of-distribution instance family and report it.
• Simulation: unequal replication counts across methods, or comparing variance without equalizing CPU effort. Fix the budget, then compare variance.