Skill Eval

Procedure

1. Fix the decision and bounds. Name the subject package/version or qualified memory update, relevant cases, allowed runtime and existing aggregate time, trial and cost limits. Do not infer billing enforcement from token counters. Smoke runs, infrastructure retries, interrupted attempts and inner review consume the same declared envelope; a new configuration or context does not renew it. Do not launch live work without caller authorization and bounds.
2. Choose the smallest relevant measurement. Use behavioral probes for acts, coding tasks for engineering outcomes, and separate later sessions for memory. There is no universal two-effort requirement. Keep the deployed model and effort unless the caller's decision concerns effort. Retain easy regression and cost controls; do not weaken the producer to manufacture separation.
3. Freeze and calibrate. Fix task, acceptance, package, model/runtime, environment and grader identities before trials. Executable oracles must accept the intended solution and reject plausible incorrect/no-op solutions. Include genuinely correct and incomplete cases when evaluating judgment. Exposed incidents are development cases, never unseen holdouts by renaming. Broken or leaked cases invalidate affected comparisons; preserve their historical disposition when versioning a correction.
4. Run within the selected consumer's bounds. Coding trials expose the actual selected package and required resources. A worktree or a prompt prohibition is not runtime isolation. Exclude operator home, production tracker, session history, sibling output and solutions; capture launched configuration and final artifacts outside the worker. Report an incompatible adapter as such; do not build a replacement platform to rescue a result.
5. Read all attempts. Use native runner results and existing accounting; collection must not require another model call or handwritten evaluation. Wrong identity, changed acceptance, contamination or ambiguous pairing cannot establish comparison proof even when a deterministic check passed.
6. Compare only supported facts. Pair by task and repetition; preserve repetitions within task clusters. Endpoint reward, worker done claim, in-workflow validator PASS and independent acceptance are different facts. Missing review, usage, billing, phase or feasibility evidence stays unknown. A worker following an instruction establishes adherence, not reduced rework or causal benefit. A passing case far from a failed boundary does not prove the boundary is repaired. Coding and memory comparisons follow coding and memory readout.
7. Recommend once and stop. A concrete reproduced defect with clean controls can support a provisional narrow repair; general improvement needs held-out comparison. Do not automatically publish a lesson.

Raw trials and new proof go to caller-selected protected external non-Git storage; only public, sanitized fixtures cleared for that destination belong in Git (ADR-0016).

Output

Decision: retain | revise | remove | insufficient evidence; scope <skill, version, cases>
Question: <maintenance decision and the measurement chosen>
Setup: <runner, model, effort, grader; what differs between arms>
Attempts: <per arm: assigned, completed, crashed or infra, interrupted, reruns>
Outcomes: <paired by case and repetition; whether the skill loaded in each treatment run>
Uncertainty: <interval and method, or "directional, n=<count>">
Cost: <measured time and cost per arm, or unknown>
Not proven: <confounds, missing coverage, what this measurement cannot show>

For behavioral authoring, also supply the existing probe package (probe.json, question.md, discriminator.sh, fixtures/, and a prelude only in injected-prelude mode) and its replay result. No new per-run worksheet is required.

Done when the requested measurement has reached its accepted stop, the relevant replay/oracle checks discriminate, missing coverage is explicit, and one recommendation answers the named maintenance decision. Insufficient evidence, an adverse result or an incompatible runtime can complete this evaluation; none counts as demonstrated skill benefit.

References

• Behavioral runner and conventions: scripts/probe-skill.sh, evals/skill-probes/README.md, seeding.
• Behavioral verdicts and non-verdict incidents: LEDGER.md, RUNBOOK.md.
• Existing coverage and headroom gates: check-skill-probe-coverage.sh, check-skill-probe-headroom.sh.
• Evidence and overclaim limits: ADR-0011, ADR-0016 and RPI traversal.