Ln 45 Benchmark Comparator

3. Execute and Capture Evidence

• Run candidates in a balanced or randomized order that avoids systematic warm-cache or temporal advantage.
• Capture start and end state, command, exit status, timing, resource metrics, logs, outputs, diffs, tests, and candidate-specific artifacts for every run.
• Verify activation before grading; mark unproven or fallback runs invalid rather than assigning them to the intended candidate.
• Grade correctness against the predefined oracle before examining performance and cost metrics.
• Grade task completeness separately from correctness and efficiency: verify every required outcome and prohibited side effect, rather than treating a smaller diff, lower token count, or successful subset as completion.
• Blind manual or qualitative graders to candidate identity and randomize presentation order; record disagreements instead of resolving them toward a preferred candidate.
• Record timeout, crash, malformed output, partial completion, tool error, and environmental failure as distinct failure classes.
• Follow the predeclared repetition or sequential stopping rule; count and preserve all attempts, including failed and invalid runs. Do not keep retrying until the desired number of successes appears.
• Pause when environmental drift, rate limits, external outages, background load, or runner defects make additional runs incomparable.
• After a harness fix, invalidate affected comparisons and rerun both candidates for those scenarios under the corrected harness; retain unaffected evidence and label prior invalid results.
• Treat setup, activation, parser, and environmental failures separately from task incorrectness, then state whether setup reliability is part of the actual product decision.

4. Analyze Validity and Results

• Exclude only runs that meet a predefined invalidation rule and record the reason, evidence, and whether exclusion changes the conclusion.
• Report per-scenario correctness, failures, latency, resource use, tokens or usage, tool calls, and other costs before aggregating.
• Use median, percentile, confidence interval, or another statistic appropriate to the sample and distribution; show spread and sample size.
• Keep metrics with different units or workloads separate and avoid a single composite score unless its weighting was defined before execution.
• Check whether differences exceed measurement noise and whether one scenario dominates the aggregate.
• Analyze setup cost, steady-state cost, maintenance complexity, portability, failure behavior, and operational burden separately from runtime metrics.
• Label synthetic fixtures, estimated tokens, character-based proxies, modeled cost, and manual judgments so they cannot be mistaken for observed telemetry.
• Request an independent blind review when qualitative output quality materially affects the decision and automated correctness is insufficient.

5. Decide, Preserve, and Clean Up

• Identify which decision the comparison can support, its valid workload/environment range, and conditions requiring another experiment; do not extrapolate outside observed evidence.
• Use WIN only when the candidate satisfies correctness and the predefined decision rule with sufficient valid evidence.
• Use TIE only when evidence supports the predefined negligible-difference margin or balanced tradeoff; failure to detect a difference with insufficient evidence is INCONCLUSIVE.
• Use INCONCLUSIVE when sample size, activation, oracle, environmental control, or conflicting scenarios prevent a reliable choice.
• Preserve reproducible commands, configuration, scenario definitions, expectations, raw results, normalized results, and analysis needed for independent verification.
• Remove only run-owned ledger entries: verify absolute paths remain inside approved temporary roots, stop exact recorded process IDs, preserve dirty or pre-existing worktrees, never delete credentials, and verify source and external baseline state.
• Report invalid runs, exclusions, confounders, sensitivity to assumptions, and how the conclusion could be falsified.
• Report residual decision risks that remain after the comparison, including unsupported workloads, unmeasured costs, unstable environments, and assumptions that could reverse the verdict.
• Verify that decision guidance follows scenario-level evidence and the frozen rule, with cleanup and limitations accounted for.

Self-Check

• Reconcile before returning. Check item-level evidence, requirement coverage, contradictions, scope, verdict, and applicable cleanup. Correct the report or authorized artifacts. Reuse valid evidence; do not automatically rescan the repository or rerun successful commands. Repeat checks only for relevant changes, failures, or unresolved evidence. Disclose remaining gaps.

Output Contract

Report in the user's language, in this order; label all five fields and state each fact once. Use controlled plain language: one fact per sentence, usually under 20 words, active voice, and one term per concept, with no synonyms for verdicts, IDs, or states. Small results may use one line per field; omit empty tables and do not copy linked artifacts:

1. Result: The exact skill-specific verdict token first, then the supported outcome.
2. Scope: Reviewed/changed scope, exclusions, baseline, and material assumptions.
3. Evidence: Skill-specific fields below; distinguish facts, inferences, and unverified claims. Link artifacts; use tables when useful.
4. Verification: Checks/results, unavailable evidence, and applicable cleanup/external state.
5. Completion: Checklist: X/Y complete; Incomplete: None or each UNPROVEN item's reason, outcome impact, and exact next action; residual risks and required decisions.

Skill-specific evidence: Frozen decision, candidates, scenarios, independent oracle, fixed variables, configuration, metrics, repetitions, exclusions, and decision rule. Prove activation and harness validity; report invalid runs, confounders, exclusions, and cleanup. Compare scenario-level completeness, correctness, failures, primary metric, spread/sample size, and other costs before aggregating. Preserve setup/maintenance tradeoffs, sensitivity, falsification conditions, raw results, configurations, scenario artifacts, and hashes when available.