Skills Validate

Validate the skill named in the invocation arguments by having a fresh agent do real work with it, then report where that agent got stuck. A defect here is a defect in the skill text, never in the agent.

Flags: --pr <number> targets a PR instead of detecting one, --plan-only runs the zero-cost planning protocol from AGENTS.md instead of live generation, --task "..." supplies the use case instead of writing one, --no-post stops before publishing anything, --keep keeps the run directory.

Live runs spend Scenario credits. Keep every generation the smallest one that still proves the point.

1. Review the objective

Read the skill's SKILL.md and every file it links; node scripts/lib/skills.mjs --dir <name> prints its folder, nested under skills/dcc/ or skills/game-engines/ for an expert tool. If no skill has that name, list the close ones and stop.

State, in your own words: the objective (one sentence), the triggering conditions the description claims, and the three to six non-obvious facts the skill exists to teach, the ones an agent would otherwise guess wrong (upload flow, jobs_wait re-calls, runs_as wiring, dry runs, launch semantics). Those facts are the traps the run has to spring.

2. Write a concrete use case

One realistic task, in the words a user would actually use, that forces at least three of those traps and cannot be satisfied by generic MCP intuition (for an expert tool, generic knowledge of the application; its budget counts application runs, not generations). Give it explicit success criteria (which artifacts must exist, and what has to be true of them) and a hard budget (how many generations, which of them may be dry_run). Print the task and the criteria before spending anything. With --task, use the supplied task and still write the criteria.

3. Run the mechanical checks first

They are free and they catch the cheap failures: pnpm spec (spec validation), plus pnpm test when the skill ships a script. Record the results and continue either way; the report carries both layers.

4. Spin up the fresh agent

Build a run directory outside the repository and install the skill under test into it, from the working tree, so the run tests the version under review rather than the published one:

SKILL="<name>"
DIR=$(node scripts/lib/skills.mjs --dir "$SKILL")
RUN=$(mktemp -d "${TMPDIR:-/tmp}/skill-validate-$SKILL-XXXXXX")
mkdir -p "$RUN/.claude/skills" "$RUN/assets"
cp -R skills/scenario "$RUN/.claude/skills/"
case "$DIR" in
  skills/*/*/*) cp -R "${DIR%/*}"/scenario-* "$RUN/.claude/skills/" ;; # expert tool: its whole family
  *) cp -R "$DIR" "$RUN/.claude/skills/" ;;
esac
awk 'f; /^---$/ { if (++c == 2) f = 1 }' .claude/agents/skill-tester.md >"$RUN/contract.md"

Copy scenario alongside every other skill: real installs ship both. An expert tool brings its whole family, since the specialists import the lead skill's scripts/, and a live run needs the application installed: ask which machine and application version the run may drive, and ask for a team and project only when the task hands off to a Scenario skill. Otherwise, ask which team and project the run should use before spending anything: every generation and upload lands in that scope, and the tester must never pick one (teams_list and projects_list enumerate the choices). Write the task from step 2 to $RUN/task.md, including the budget, the success criteria, the run directory path, and the team and project.

Then pick the strongest isolation available. If the Claude Code CLI is installed, run cd "$RUN" && claude mcp list; otherwise use the subagent route below.

• Separate process (preferred, and a genuinely empty context) when that lists a connected Scenario server:

  cd "$RUN" && claude -p --setting-sources user,project \
    --append-system-prompt "$(cat "$RUN/contract.md")" \
    --allowedTools "mcp__<server> Read Write Bash" \
    --output-format json "$(cat "$RUN/task.md")" | tee "$RUN/result.json"

  Use the server name claude mcp list printed, normalized the way tool prefixes are: dots and spaces become underscores, so claude.ai Scenario is mcp__claude_ai_Scenario. The child loads the skills from $RUN/.claude/skills/ and never sees this repository, though --setting-sources user,project (which a user-scope connector needs) also carries the user's plugins and hooks into it; watch the transcript for injected noise.

• Subagent when the CLI is unavailable or the list has no connected Scenario server. Spawn a fresh agent with the contract, the task, and the run directory path (Claude Code can use its skill-tester agent). Isolation is weaker: it can still see repository context, which is why the contract tells it to ignore it.

Say in the report which mode ran. With --plan-only, the same setup applies but the task asks for a numbered tool-call plan with exact tool names and argument shapes, executing nothing.