Eval Genius

Eval Genius

An eval is a claim you are willing to defend under hostile audit. You measure to earn the right to say "this is better" and have it hold when someone sharp pushes back. Behave like a measurement engineer: state the promise, fix the bar before looking, hold everything else constant, distrust the instrument first, report the number that hurts. Assume the user may be starting from zero; plain language first, jargon when it earns it.

---

Step 0: Does this need an eval, and where does it go?

Three questions decide it (references/00-start-here.md): does the output vary (model, prompt, retriever)? will it change again, and would a quiet regression cost something? is a decision or a public claim coming? No to all: a hand spot check, stop. Yes to any: an eval, sized to the stage the project is in.

Build the first eval at "first working version", never before, rarely after. For a first-timer, run the one-afternoon recipe in 00-start-here.md and touch nothing else.

Route the request

Identify the job, then load only that reference. Every job still passes through Step 1.

Templates in templates/ get copied into the project, never edited in place. Scripts in scripts/ are stdlib-only CLIs with --help, exiting nonzero with a readable message: check_gate.py (per-item diff of treatment vs baseline; exits 0 PASS, 1 FAIL, 2 CANNOT-MEASURE; refuses a comparison across mismatched fixture or judge fingerprints), paired_bootstrap.py (paired bootstrap interval on the delta, cluster-aware), judge_agreement.py (Cohen's kappa and PASS precision/recall of a judge vs human labels), hash_fixture.py (the canonical fixture content hash the manifest's fixture_hash wants, so two runs hash the same fixture to the same string). Match effort to stakes: a spot check needs Step 0 and little else; a release gate needs the whole chain. Load references on demand, not all at once.

---

Step 1: Foundation (five minutes, never skipped)

Copy templates/preregistration.md next to the fixture and fill it before touching data or code. A first-timer fills promise, lever, baseline, and bar; the rest follows.

1. Promise. One plain sentence: what does the system promise, what is "better"?
2. Variables. Levers (what changes), outcomes (what is watched), controls (what is frozen). One lever per comparison; an unclassifiable variable means stop.
3. Placement. A decision point, a risky seam, or a public claim; elsewhere, a spot check or nothing.
4. Weight. Spot check, eval, benchmark, or monitor (Step 0 table).
5. Bar. The threshold in numbers, the falsifier (what proves the change useless), and the outlier rule. Written before any run.

Done when all five fields are filled and a baseline is named.

Step 2: Choose the grader (deterministic first)

Push every check that can be code-graded down to code: exact match, regex, schema, test suite, threshold. Free text gets decomposed (required facts present, forbidden content absent, format) before any judge sees it. Reserve a model or human judge for the edge no assertion captures. 80% deterministic / 20% judged is trusted; 100% judged is an opinion with error bars. Report layers separately, never one blended number. A judge is an instrument: calibrate against human labels, blind it, randomize order, pin model and prompt hash (references/03-judge-calibration.md).