Skill Doctor

skill-doctor

Grade the user's agent setup by scoring recent local agent conversations, then propose concrete skill edits and render one shareable report page.

The report can cover conversations in the current repository, conversations in selected projects, or all local conversations. It can evaluate project skills alone or project and global skills together.

Everything runs locally. Never upload transcripts, session files, or any excerpt of them anywhere. The only shareable artifact is the report the user chooses to post.

Let SKILL_ROOT be the directory containing this SKILL.md.

Step 0: Start the run

Verify the executing harness

Read $SKILL_ROOT/references/supported-harnesses.md and identify the harness executing this skill from the runtime context. If it is unsupported or cannot be identified confidently, follow the reference's stop behavior. Do not create a report directory or read conversation history.

Ask which conversations to grade

First check whether the current directory is inside a git repository:

git rev-parse --show-toplevel

Use the harness's user-question tool when available.

When a current repository is available, ask “Which conversations should I grade?” with:

1. Conversations in this repository — recommended.
2. All conversations.
3. Choose projects to analyze.

When there is no current repository, ask the same question with:

1. All conversations — recommended.
2. Choose projects to analyze.

If the user chooses projects, ask for one or more project paths. Expand and validate every path as a git repository before continuing. The run produces one combined report across those projects.

Ask which skills to evaluate

Then ask “Which skills should I evaluate?” with:

1. Project skills + global skills — recommended.
2. Project skills only.

For an all-conversations run, “Project skills” means skills from local git repositories inferred from the conversations' working directories. After these answers, proceed immediately.

Never write artifacts into the user's repo. Create one fresh, collision-free scratch directory per run and use it as REPORT_DIR for every artifact:

REPORT_DIR="$(mktemp -d "${TMPDIR:-/tmp}/skill-doctor-XXXXXXXX")"

Step 1: Collect

Build the collector arguments from the startup answers:

• Current repository: --repo "$REPO".
• Selected projects: repeat --repo PATH for every project.
• All conversations: --all-conversations.
• Project and global skills: add --include-global-skills.
• Project skills only: do not add --include-global-skills.

python3 "$SKILL_ROOT/scripts/collect_sessions.py" \
  --out "$REPORT_DIR" \
  <conversation-scope arguments> \
  <skill-scope arguments>

By default --harness auto scans every locally available supported source. Read $SKILL_ROOT/references/supported-harnesses.md for source identifiers, storage details, skill locations, and source-specific override flags.

Useful flags:

• --harness VALUE — which local session sources to scan; use the reference's collector IDs.
• --repo PATH — include a project; repeatable.
• --all-conversations — do not filter conversations by project.
• --include-global-skills — also grade global skills.
• --days N — lookback window (default 45).
• --max-sessions N — cap on sampled sessions (default 12).
• --skills-dir PATH — nonstandard skill locations.
• --include-subagents — include child or sidechain sessions.

Read $REPORT_DIR/inventory.json. If sessions_sampled is 0, tell the user there is nothing recent to score in the selected conversation scope (suggest raising --days or choosing different projects) and stop. If skills_found is 0, continue — the report becomes a case for creating skills, and skill_coverage is 0.

Step 2: Score each sampled transcript

Scoring is based on efficiency, code quality, procedure compliance, and verbosity for the sessions sampled. Process datasets of 50 transcripts or fewer in a single batch. For datasets with more than 50 transcripts, use parallel batches (20 transcripts per batch recommended). Score batches in the current local agent process, or delegate only to local child agents that keep transcript contents on the user's machine. Pass the following rubrics as context:

• $SKILL_ROOT/scorers/efficiency.md
• $SKILL_ROOT/scorers/code-quality.md
• $SKILL_ROOT/scorers/procedure-compliance.md
• $SKILL_ROOT/scorers/verbosity.md

Instructions: For each transcript in $REPORT_DIR/transcripts/, read it and judge it against all four rubrics. For each scorer record: label, numeric score (from the rubric's label table), and a 1–3 sentence reason citing specifics from the transcript. Apply the code-quality scorer only where the transcript shows code changes; otherwise record insufficient_evidence and exclude that result from the code-quality average and failed-conversation filter.