Analyze Id Eval Ranking

1a. Selecting the canonical eval when repeats are NOT identical-setting (load-bearing)

In practice a (model, benchmark) often has several Finished rows that are not identical-setting, so the "average identical repeats" branch does NOT apply — you must pick the canonical clean measurement (per crud-otagent-supabase §GOTCHA 2 rule 1's "different settings → keep the canonical one"). Detect and EXCLUDE the non-canonical ones (validated grid-exact on the RL ablation, 2026-07-09):

• Summarization-buggy (deflated) runs — a run with non-trivial stats.evals.*.exception_stats.SummarizationTimeoutError scored lower because of the summarization bug, not the model. Drop it in favor of the post-fix clean run.
• Degenerate broken-serving-batch runs — an implausibly low value from all-zero-reward batches (e.g. dev_set_v2 1.0–1.7% when the clean grid value is ~12%). Drop.
• Drifted eval generations — the same clean setting re-run weeks apart can differ materially (e.g. dev_set_v2 20.5%@2026-06-29 vs 9.8%@2026-07-08). Do NOT average across generations; keep the study's canonical measurement (the earliest clean post-fix run, matching the experiment's id_eval_grid.md / ABLATION_DEFINITIONS.md). Averaging here would mix generations and desync from the grid.
• Always prefer the canonical harness setting (terminus-2, timeout_multiplier=2.0, n=3).

Cross-check the result against the experiment's own grid (id_eval_grid.md / COMPARISON_*.md) — every ranked cell should reproduce it exactly; a mismatch means you picked a non-canonical run. If the clean/canonical value the grid cites is not present in sandbox_jobs (only superseded pre-fix rows exist), treat that benchmark as MISSING for §2 (flag it) rather than substituting a deflated row.

2. Validity gate — flag models missing any ID benchmark

A model is ID-valid only if it has a Finished score on all three ID benchmarks. Report (do NOT silently drop) any input model missing ≥1 — the normalization population must be the models that actually have the benchmark (partial models distort mean_b/std_b). Decide explicitly: rank only the fully-ID-complete models (default), and list the incomplete ones separately with their gaps.

3. Normalize + rank

complete = {s:(sc,_m) for s,(sc,_m) in scores.items() if all(b in sc for b in ID)}
acc = {b: {s: complete[s][0][b]["acc"] for s in complete} for b in ID}      # per-benchmark accs
z   = {}
for b in ID:
    vals = np.array(list(acc[b].values()), float)
    mu, sd = vals.mean(), vals.std(ddof=0)                                  # population std
    z[b] = {s: (acc[b][s]-mu)/sd if sd>0 else 0.0 for s in acc[b]}
norm = {s: float(np.mean([z[b][s] for b in ID])) for s in complete}
raw  = {s: float(np.mean([acc[b][s] for b in ID])) for s in complete}
ranking = sorted(complete, key=lambda s: norm[s], reverse=True)

4. Emit the table

Columns (match the paper's layout): Rank · Model · SWE-Bench-100 (%) · OT-TBLite (%) · Terminal-Bench-2.0 (%) · Raw avg (%) · Normalized (z) · Trace links. Per-benchmark cell = raw accuracy % (append ±SE from accuracy_stderr when present). The Trace links column carries the per-benchmark hf_traces_link URLs (swe / v2 / tb2) — the same field the leaderboard uses; a missing link → note "—". Sort by Normalized desc; number the ranks.

• Emit markdown (and optionally a CSV alongside) to the experiment dir when run on one, e.g. <experiment>/id_eval_ranking.md. Also print a one-line summary (N models ranked, N flagged incomplete).
• Report normalized to 2 decimals with sign (e.g. +0.49, −0.57) like the paper; raw % to 2 dp.

Guardrails

• Read-only. Never write Supabase. (For trace-link repair, that's crud-otagent-supabase §hf_traces_link — a different, write task.)
• Population = the candidate set (the input models), per-benchmark. Not a global mean. If the input list changes, the z-scores change — that is by design (relative ranking).
• All three benchmarks equal weight — average the z-scores, never weight by N or by raw range.
• Averaging repeats: average identical-setting Finished repeats; sibling-models-aware (ilike)
  • family-aware (duplicate_of) per crud-otagent-supabase §GOTCHA 2/3. Don't pick max/first.
• Benchmark mapping: OT-TBLite=dev_set_v2; SWE-Bench-100=-random-100-folders (NOT full 500); tb2=terminal_bench_2. Getting SWE wrong silently swaps an OOD benchmark into the ID ranking.
• Flag, don't drop, incomplete models — surface any input model lacking all 3 ID scores.

Related

• crud-otagent-supabase — the schema, get_metric, sibling/family resolution, hf_traces_link, the ID/OOD master list. This skill is a read-only consumer of it.
• otagent-paper/02_arXiv/otagent.tex — the normalization source of truth (§Pipeline, App. task-gen full tables). Re-read if the method changes.