Improve Skill Quality

Improve Skill Quality

Turn a failing or unconvincing evaluation into a targeted fix. The single most common mistake in this repo is rewriting skill prose in response to a verdict whose real cause was the eval, the fixtures, or the harness. Classify first, then fix.

When to Use

• An evaluation verdict is a regression, underpowered, or "no credible improvement".
• A skill wins in the isolated arm but not in the plugin arm, or is reported "not activated".
• /evaluate reports "Evaluation ran but produced no results".
• A skill scores well but costs too much (tokens, turns, wall time, plugin menu budget).
• Deciding whether to strengthen or retire a persistently weak skill.

When Not to Use

• Creating a new skill from scratch — use create-skill.
• Creating a new eval.yaml from scratch — use create-skill-test.
• Changing the harness itself (eng/skill-validator, eng/vally-adapter, evaluation*.yml).

Inputs

Workflow

Step 1: Get the evidence before forming a hypothesis

Read InvestigatingResults.md for how to download artifacts and read results.json. Extract, per failing stimulus:

• authoritative stimulus-vote W/T/L and separate repeated-run W/T/L
• activation status in the isolated and plugin arms, separately
• the judge's verbatim reason on each losing trial
• whether any trial errored, timed out, or produced empty output

Do not change skill content until you can quote a losing trial and the judge's reason for it. For the other cause classes the evidence is different: harness failures are diagnosed from the job log and the spec, and power problems from the trial record — neither has a losing trial to quote, and demanding one is what sends people rewriting prose instead.

Step 2: Classify the failure

Work down this table and stop at the first row that matches. Rows are ordered by how often the symptom has been misdiagnosed as a skill-content problem — the fixture row is first because a fixture failure also presents as a setup or reliability failure and gets misfiled as one.

A verdict is only a measured result when the comparison was conclusive: adapt.mjs requires zero errored trials, zero unmatched trajectories, and an agreeing summary before it will report a pass or a regression. Confirm that before reading a record as a power problem.

Step 3: Rule out harness and reliability causes

See references/eval-triage.md for the full catalogue. The recurring ones:

• The repository gate rejects the deprecated top-level config: alias, and Vally rejects a spec that declares both config: and defaults:. Replace the alias with one defaults: block.
• An errored trial is not automatically a fixture problem — judge-side auth and session.idle failures look identical from the verdict and need harness fixes, not SDK pins.
• expect_tools: [bash] on an advisory question forces a restore or build and turns an answer into a timeout with no quality gain.
• Genuine code-generation stimuli need roughly 360s; a timeout yields empty output, which fails every grader and hides the real quality signal.
• Unmatched trajectories, an errored trial, or a summary that disagrees make the comparison inconclusive: the remaining matched trials are biased, so the record is not a measured null and must not be read as a power or content problem.
• A generic YAML parser is not the production loader. If Vally rejects a skill eval, or the native SDK lane rejects an agent eval's executable scenario fields, classify it as harness / spec-load before changing content. The native agent parser ignores golden references, so validate those separately with check_eval_quality.py and deterministic golden-workspace replay.
• A pass with one worker or an enlarged local timeout is not normal execution evidence. Reproduce with the repository's normal concurrency and declared suite budget; a failure there is a reliability defect.

Step 4: Verify the fixtures before touching the skill

Run python eng/eval-quality/check_eval_quality.py — it blocks 22 defect classes that can cost a real result here. Then confirm by hand:

• every fixture behaves as its stimulus assumes — a fixture meant to be healthy builds, and one meant to be broken fails for the exact reason the stimulus is about and no other;
• every referenced fixture is in the git index (git ls-files), not merely on disk — .gitignore has silently swallowed committed coverage fixtures;
• a fixture never states the same fact in two places that disagree — a Cobertura report whose declared line-rate, summary totals and <line> elements differ is the canonical case — or the two arms legitimately read different truths.
• every preservation or scope assertion covers the complete in-scope file set, not one representative file.
• the golden trajectory and patch pass the deterministic graders, and a realistic broken mutation fails the grader that is meant to protect the behavior.

Step 5: Check whether the eval could ever have passed

The gate has two independent bars, and confusing them is the usual misdiagnosis:

1. Distinct stimuli ≥ 5. Below that the verdict is reported underpowered — never a pass, never a regression.
2. The sign test must reach p ≤ 0.05 over the discordant (non-tie) stimulus votes. Ties are not discarded silently; they hold the discordant count down.

So at exactly 5 stimuli a single tie is fatal — it leaves 4 discordant. At 6 stimuli one tie is survivable (5W/1T/0L); at 7, up to two are (5W/2T/0L). A loss is not.

So a positive record with a failing verdict is a power problem, not a content problem. Fix it by adding discriminating stimuli. Raising runs measures reliability for the same task and cannot clear the floor.