Skill Conductor

Skill Conductor

Full lifecycle management for agent skills: draft → test → review → improve → repeat.

One skill to rule them all — from architecture to packaging. The core loop is always the same: write something, test it, see what fails, fix it, test again.

Runtime requirements (pre-flight)

Before any mode that touches scripts (CREATE, IMPROVE, VALIDATE, OPTIMIZE, PACKAGE), run the pre-flight block → references/runtime-setup.md (checks uv, sets UV_BIN/SKILL_CONDUCTOR_DIR, verifies LLM access). If uv is absent, stop and tell the user.

How to communicate

Read context cues. If the user is a skill author iterating on their own work, be direct and technical. If they're new to skills, explain the why behind each step — not just what to do, but why it matters. Default to conversational, not robotic.

• Explain trade-offs when there's a real choice to make
• Use concrete examples over abstract rules
• When something fails, explain the root cause, not just the fix
• Imperative voice in instructions: "Extract the data", not "You should extract"

Modes

Detect mode from context. If ambiguous, ask.

Mode 1: CREATE

Step 1: Capture Intent

Before writing anything, extract 2–3 concrete scenarios.

Ask:

• "What specific task should this skill handle?"
• "What would a user say to trigger it?"
• "What should NOT trigger it?"

Don't move on until you have a clear picture of what the skill does, for whom, and when. This prevents the most common failure: a skill that does something but triggers for the wrong things.

Step 2: Baseline (TDD RED)

Before writing the skill, verify the agent fails without it:

1. Take one scenario from Step 1
2. Run it in a clean session without the skill
3. Document what went wrong — what the agent guessed, what it missed

If the agent already handles it perfectly, the skill is unnecessary. This sounds obvious, but it's the most skipped step and the most valuable one.

Step 3: Architecture

Choose a primary pattern from references/patterns.md (can combine):

Choose degrees of freedom — this determines how much control vs. flexibility the skill gives the agent:

Freedom test: ask "if the agent makes a mistake here, what is the consequence?" High consequence → low freedom (an exact script it must not modify). Low consequence → high freedom (prose, let it judge). Calibrate per step, not per skill — one skill can hold both.

Golden rule: read references/sop-practices.md before authoring or reviewing ANY skill. It holds the canonical 10 authoring principles (universal): pre-flight, no-process-in-description, MOC (SKILL.md = map, not prose), fresh-practitioner author, TWI "why", blind-agent test, inline checklists, one-term-per-concept, cut-the-fat (env/keys OUT of SKILL.md), match-the-form-to-the-failure. For procedural skills (business process with branching: request, quote, onboarding, escalation) the same file also has the deep SOP methodology — format selection, 7-step process, procedural checklist.

Step 4: Scaffold

uv run scripts/init_skill.py <skill-name> --path <output-dir> [--resources scripts,references,assets]

Or create manually:

skill-name/
├── SKILL.md          # required — the brain
├── scripts/          # deterministic operations (executed, not loaded)
├── references/       # detailed docs (loaded on demand)
└── assets/           # templates, images for output (never loaded)

Step 5: Write SKILL.md

Frontmatter

---
name: kebab-case-name
description: >
  [What it does]. Use when [4-5 phrasing variations users actually say] — even
  if they don't explicitly say "[canonical term]". Do NOT use for [negatives].
---

The description is the single most important line — it decides whether the skill triggers at all. The full formula, the pushy clause and worked GOOD/BAD examples live in references/sop-practices.md Principle #2. Read it before writing one.

• name: lowercase, digits, hyphens only. No consecutive hyphens. Matches folder name. Max 64 chars
• description: max 1024 chars. No angle brackets. No process/workflow steps
• Don't put workflow in the description — tested: when the description lists process steps, the agent follows it and skips the body entirely

Body structure

# Skill Name

## Overview

What this enables. 1-2 sentences. Core principle.

## [Main sections]

Step-by-step with numbered sequences.
Concrete templates over prose.
Imperative voice throughout.

## Common Mistakes

What goes wrong + how to fix.

## Troubleshooting (if applicable)

Error: [message] → Cause: [why] → Fix: [how]

Writing rules

• One term per concept. Pick "template" and stick with it — not template/boilerplate/scaffold (Principle 8)
• SKILL.md = map, not prose. Body is a table-of-contents pointing to references; detail lives there (Principle 3)
• No secrets/env in SKILL.md. No keys, passwords, tokens, env values, or user-absolute paths (/home/<user>, /Users/<user>) — reference them, never inline (Principle 9a)
• Progressive disclosure. SKILL.md = brain (<500 lines). References = details. One level deep
• Token budget. Frequently loaded: <200 words. Standard: <500 lines. Heavy: move to references/
• No junk files. No README, CHANGELOG inside the skill
• Scripts: bundle when same code rewritten repeatedly, or operation is fragile. Must return descriptive stdout/stderr on failure
• Imperative voice. Use "Extract the data", not "you should extract" or capitalized "MUST/NEVER" — explanation > rule (see references/sop-practices.md Principle 5, TWI)

Step 6: Test Cases & Eval Loop

This is the critical step — most failures hide here. Treat it as three sub-phases.

Before the full loop, micro-test the wording of anything you just wrote (5+ fresh-context reps, always with a no-guidance control) → references/pressure-testing.md. For a discipline skill — one that makes the agent follow a rule it's tempted to break — a pressure scenario from that file is mandatory, not optional.

6a. Pre-flight (before spawning anything)

• evals/evals.json exists with 3–5 prompts (see references/schemas.md)
• Workspace dir created: <skill-name>-workspace/iteration-1/
• Each eval has a descriptive name (not just eval-0) and eval_metadata.json
• Anthropic key for executor subagents is set
• uv and eval-viewer/generate_review.py are reachable from current working dir

If any item fails — fix before proceeding. A missing workspace dir mid-run loses outputs.

6b. Run loop (do all in one turn)

6c. Post-run checklist

• All timing.json files written (one per run)
• Each run has a grading.json with fields text, passed, evidence (not name/met)
• benchmark.json aggregated: uv run scripts/aggregate_benchmark.py <workspace>/iteration-N --skill-name <name>
• Analyst pass done — see agents/analyzer.md for what to look for (non-discriminating assertions, high-variance evals, time/token tradeoffs)
• Eval viewer launched: uv run eval-viewer/generate_review.py <workspace> --skill-name <name> --benchmark <path>
  • In headless mode: --static <output.html> and send file to user
  • For iteration 2+: add --previous-workspace <previous-iteration-path>
• User saw the viewer before I started editing the skill

The last bullet is the trap. If you skip user review and "improve" based on your own reading of outputs, you optimize against your taste, not the user's.

Step 7: Verify & Refactor

1. Does the skill trigger automatically for the right queries?
2. Does the agent follow body instructions (not just description)?
3. Does the output meet use case requirements?
4. Does it NOT trigger on unrelated queries?

If any fail → iterate. Find how the agent rationalizes around the skill, plug loopholes, re-verify.

Mode 2: IMPROVE

Step 1: Diagnose

Read the existing SKILL.md completely. Identify the problem class:

Improvement mindset

1. Generalize from feedback. You're iterating on a few examples, but the skill will be used on thousands of prompts. Don't overfit — avoid fiddly patches or oppressive MUSTs for one test case. Try different metaphors or patterns instead
2. Keep the prompt lean. Read transcripts, not just outputs. If the skill makes the model waste time on unproductive steps, remove those instructions and see what happens
3. Explain the why. LLMs have good theory of mind. Instead of ALWAYS/NEVER in caps, explain the reasoning — it's more powerful and robust. If you're writing rigid rules, reframe as explanations
4. Look for repeated work. If all test runs independently write the same helper script, bundle it in scripts/. Saves every future invocation from reinventing the wheel
5. Apply the authoring canon. Read references/sop-practices.md — the 10 canonical principles (universal) map directly to skill failure modes: process leaking into description, SKILL.md bloated instead of a map, env/keys inlined, silent improvisation from missing "why", missed edge cases, agents skipping end-of-doc checklists, a rule whose form doesn't match its failure. For process skills (ticket, quote, escalation) also apply the deep SOP methodology in the same file

Step 2: Eval Iteration Loop

The improvement cycle mirrors CREATE Step 6, but focused on the broken behavior. Micro-test each candidate wording before it enters the loop, and re-run the pressure scenarios if the skill enforces a rule → references/pressure-testing.md.

1. Run the failing case with current skill → document failure
2. Apply fix using writing rules from CREATE Step 5
3. Run eval again → grade with agents/grader.md
4. Launch viewer: uv run eval-viewer/generate_review.py <workspace>
   • Headless/Cowork: use --static <output.html> instead of live server
5. Review, provide feedback, iterate

Step 3: Gated Self-Update Loop

Drive iteration off failing BinEval questions, not taste — and accept edits only against evidence the editor never saw. Full rules: references/bineval-method.md § Gated self-update loop.

1. Freeze the split once per session: uv run scripts/split_evals.py evals/evals.json --holdout 0.4 --write <workspace>/split.json — deterministic, stratified by the optional per-eval category. Never re-split after seeing results
2. Run ALL evals on the current version and grade (see Mode 3 Stage 3 + references/bineval-method.md) → collect failing[]
3. Analyze failures on TRAIN cases only: spawn agents/analyzer.md with train transcripts + gradings to produce generalized, deduped lessons. Held-out grading stays unopened until the gate
4. Apply at most 3 atomic edits (add/delete/replace one rule, paragraph, or table row; one edit = one lesson, labeled). No wholesale rewrites — small diffs keep cause and effect attributable at the gate
5. Re-run ALL evals. Gate — accept iff: (a) no held-out assertion flips pass→fail vs the parent (a flip counts only once it reproduces in 2 consecutive runs — single runs are noisy); (b) train pass-rate strictly improves; (c) no NEW failing critical question. Held-out improvement is welcome but not required — with 5–8 held-out cases, demanding it measures luck
6. Record per-assertion transitions (improved / regressed / persistent-fail / stable-success) → transitions block in benchmark.json
7. Terminate when train failing[] (or its critical subset) is empty, or after 3 iterations. Keep the best ACCEPTED version by held-out pass-rate, then train pass-rate