Eval Harness First

Eval Harness First

The Phase 0 gate for the whole plugin: finetuning-method-selection and every downstream skill assume this harness exists before a training config gets written. The harness is not a run-end side artifact — it is the data-curation engine. The same labeled traces that build the goldens feed training data, minus an explicit holdout.

Input: production/agent traces if they exist, or a task spec if they don't, plus labelers willing to grade ≥100 examples. Output format: the eval/ directory below — goldens, graders, drift suite, and the base-model baseline that later phases gate on.

The Gate

No eval harness, no fine-tune. Skip to a training config and there is nothing to measure against, nothing to catch regressions, and no labeled data to train on. The flywheel:

1. Collect traces — production/agent spans, or synthetic tasks if none exist yet.
2. Error analysis — open coding on ≥100 traces, axial coding into 4–8 failure buckets.
3. One grader per bucket — deterministic first; calibrated LLM-judge only for genuinely subjective criteria.
4. Prioritize by frequency × severity × value.
5. The labeled traces feed dataset curation, minus an explicit holdout. Every eval/goldens.jsonl ID stays excluded from training data by ID.
6. Train.
7. Re-run the same harness on the checkpoint — not a different, looser one.
8. Drift detection feeds back to step 2 — new production failure modes re-open error analysis.

Steps 2–4 build the harness; steps 5–8 are why it must exist first — it is both the training data source and the checkpoint's exit gate.

Building Goldens

• From traces, when they exist: run error analysis — open coding on ≥100 real traces (read them, tag failures in your own words, no fixed taxonomy yet), then axial coding to collapse those tags into 4–8 named failure buckets. Fewer than 4 means the coding pass was too shallow; more than 8 means buckets need merging. Exception: single-failure-surface tasks (e.g. strict-schema extraction) may land at 1–2 buckets with per-field sub-metrics inside one grader — don't invent artificial splits with no evidence behind them.
• Synthetic, when traces don't exist yet: dimension-based generation — enumerate the axes that matter (task type, difficulty, edge case, persona) and sample the cross-product; free- generated prompts cluster around whatever's easiest to write.
• Goldens are versioned like code — commit eval/goldens.jsonl, diff it in review, tag it per release. It doubles as the CI regression suite.

Graders

One grader per failure bucket from error analysis — not one for the whole eval set. A single blended score hides which bucket regressed.

• Deterministic first. Regex, schema validation, or execution checks are cheaper, reproducible, and need no calibration.
• LLM-judge only for genuinely subjective criteria — tone, faithfulness, "which response is better" — where no deterministic check can express it.
• Binary pass/fail over Likert. A 1–5 or 1–10 scale is noisier to calibrate and harder to apply consistently; collapse to pass/fail.
• Drift-suite MMLU-style scoring: prefer logprob over generate-and-extract — a tight token budget makes generate-and-extract parse-brittle for models that preamble, conflating format compliance with the knowledge being measured. Templates for all four grader shapes and this scoring note: references/grader-templates.md.