Skill Autobench

skill-autobench — write the eval from lived usage

Convention: see conventions/brain-first.md — mining starts in the brain. Search the conversation archive before touching raw transcript files, and never declare "no history" without having queried the brain first.

Convention: see conventions/model-routing.md — mining and synthesis run on the cheap tier by default. The full multi-model judging pass is an explicit opt-in (see Contract).

The self-improving loop has three legs: an eval, a variant generator (SkillOpt), and a replay + judge harness (gbrain eval cross-modal). The generator and the judge ship with gbrain. The persistently missing leg is the eval author — someone has to WRITE the eval, and a spec-derived benchmark only tests what the skill promised, not what users actually asked for or what actually went wrong. This skill writes the eval from reality instead of imagination.

Pipeline

1. MINE — extract real invocation windows

Substrates, in priority order:

1. Brain conversation archive — pages under conversations/, populated by the conversation-archive skill (hard dependency for this substrate: if it hasn't ingested your history yet, run it first). Search for the target skill's name, trigger phrases, and output shapes:

   gbrain search "<skill-name>"
   gbrain query "when did I use <skill-name> and what did I ask for"

2. Per-harness session transcripts, as available — use what the harness exposes; do not assume a layout. gbrain transcripts recent --full reads the configured local transcript corpus (local-only by design). Claude Code keeps per-project session JSONL under ~/.claude/projects/; other harnesses have their own session stores. Absent stores are simply skipped.

From each hit, extract an invocation window: the user ask before the invocation, the invocation turn itself, and the 2 turns after — because that is where corrections live. A user correction after an invocation is the gold signal: it is a real, observed failure mode, and it becomes a hard_fail plus a replayable case.

FAIL-CLOSED: if no substrate yields a single real invocation of the target skill, emit an honest no-history report (substrates checked, queries run, windows scanned, zero matches) and stop. Do NOT invent "typical" invocations. For a skill with no history, the right tool is gbrain skillopt <name> --bootstrap-from-skill (spec-derived, and honest about it) — see Dedup.

2. SYNTH — turn windows into a proposed eval

From the mined windows plus the current SKILL.md, produce:

• A proposed eval_contract: goal, dimensions, hard_fails. Dimensions come from observed asks; hard_fails encode observed corrections.
• 4-8 replayable cases, each shaped {input, expected_behavior, failure_mode_to_catch} — realistic input, a checkable expected behavior, and the named failure mode the case exists to catch.
• Spec-vs-usage gaps: "the spec says X, users consistently ask Y."

HONESTY LABELS are mandatory. Every dimension and every case is labeled:

• HISTORY-IMPLIED — a real mined window backs it; cite which one.
• SPEC-DERIVED — inferred from SKILL.md only; no usage evidence.

Never conflate the two. If history is thin or off-target, say so prominently at the top of the staged file ("GROUNDING WARNING: only N windows found, none exercised the core path") instead of padding with fabricated evidence.

Privacy scrub before staging: staged evals live in the skill repo and are distributable. Mined windows contain real names, companies, and deals — rewrite every case onto placeholder slugs (alice-example, acme-example) before writing the file. A history-grounded case keeps its shape and failure mode, never its real entities.

3. STAGE — human gate, always

Write skills/<name>/eval/autobench-<date>.md with frontmatter status: PENDING-HUMAN-APPROVAL.

This skill NEVER rewrites SKILL.md — not the eval_contract, not the body, not the triggers. Merging the staged eval is the human's decision. (This is a workflow contract the agent must honor, not a mechanically-enforced gate.)

The loop (after approval)

1. Human reviews, edits, and approves the staged eval; the approved eval_contract is merged into the skill's frontmatter explicitly.

2. Convert approved cases into skills/<name>/skillopt-benchmark.jsonl lines and run gbrain skillopt <name> — this is the SkillOpt surface extension: a history-grounded benchmark replacing the spec-derived bootstrap.

3. Judge outputs through the native gate:

   gbrain eval cross-modal --task "<what the output was meant to achieve>" --output <path>

4. Re-run autobench after more usage accumulates; diff against the prior staged baseline.

Panel integrity — trust no aggregate

Multi-model judging is only as good as the panel being real. The silent failure class: a model-id normalizer strips provider prefixes and every "different model" call lands on one host, so a "3-frontier consensus" is one model's opinion in a trench coat. Before trusting any multi-model verdict, assert over the result object:

1. Each named model returned a non-empty response. An empty or failed slot is the first tell of a collapse.
2. Responses came from DISTINCT provider endpoints. If three "different models" all report the same provider, the panel collapsed to one host.
3. No two "different models" returned byte-identical output. If two differently-named models return the same bytes, they are the same model. This catches a collapse even when provider metadata is missing or faked.

gbrain eval cross-modal already exits 2 (INCONCLUSIVE) when fewer than 2/3 models return parseable scores; the byte-identical duplicate check and the distinct-endpoint check are the independent backstops this skill layers on top. Run them over the receipt JSON (written to the receipt dir) before treating a PASS/FAIL as authoritative. The integrity check is pure assertion logic over an existing result — it never calls a model itself: no network, no cost.