Skill Auditor

Skill Auditor — Quality Grading System (A–F)

Audit any Hermes skill file and assign a quality grade based on clarity, completeness, tool guidance, and shareability. Returns specific fix suggestions ranked by impact.

When to Use

• User asks you to review, audit, or grade a skill
• You're about to create a new skill and want to validate the draft
• A skill is behaving unreliably (agent skips it, calls wrong tools, misses steps)
• User shares a skill file or path for feedback
• You're preparing skills for sharing with others

Don't use for: general Hermes troubleshooting, model selection, config review — those have their own skills.

Grading Criteria

Each skill is scored across five dimensions. Points are deducted from 100. The final grade maps to a letter:

• Grade A (90–100) — Production-ready. Solid frontmatter, exact commands, real pitfalls, verification steps, consistent structure. Will fire reliably and execute correctly across model sizes.
• Grade B (80–89) — Minor gaps. Missing one dimension but still reliable.
• Grade C (70–79) — Functional but vague in places. Needs clarification on 1–2 key areas, especially with smaller models.
• Grade D (60–69) — Error-prone patterns detected. Incomplete steps or missing critical sections. Will fail silently on model switches.
• Grade F (<60) — Broken discovery or execution. Either the description is too vague to fire, or the steps are too incomplete to follow.

Dimension 1: Frontmatter & Description (max 25 points)

The description is your skill's only chance to be discovered. Hermes sees the one-line description from available_skills before deciding whether to load SKILL.md at all. If that description is vague, the skill never fires — nothing inside SKILL.md matters if this step fails.

Full marks (25): Valid YAML frontmatter with name and description fields. Description starts with "Use when..." and covers the trigger class (not a single task). Specific enough that Hermes can distinguish it from similar skills. ≤ 1024 chars.

• Example A: Use when debugging Python: test failures, uncaught exceptions, silent bugs. Covers root cause analysis, not just error messages.
• Example B: Use when debugging code issues and test failures. (too broad — could overlap with other skills)

Partial marks (10–24): Has frontmatter but description is too generic or overlaps with another skill's scope.

• Problem: "Debug stuff", "debugging", no "Use when..." pattern, description that could match 3+ skills

Zero marks (<10): Missing frontmatter entirely, or missing name/description fields.

Penalties:

• Missing frontmatter: -5 pts
• Missing description field: -3 pts
• Description too generic (no "Use when" pattern): -2 pts
• Description overlaps with another skill's scope: -1 pt

Dimension 2: Exact Commands (max 25 points)

Every step should include the actual command, tool call, or file path — not a description of what to do. Use the actual registered tool names from the Hermes toolset (skill_view, skill_manage, skills_list, write_file, search_files, terminal, read_file, patch) — generic phrasing misleads readers on different setups.

Full marks (25): Every numbered step has an exact command or explicit file path with verification condition.

• Example: search_files(pattern='config', target='files', path='.') instead of "search for the config"
• Example: pytest tests/test_module.py::test_name -v instead of "run the failing test"

Partial marks (10–24): Most steps have commands but some are vague.

• Problem: "run the script", "use the appropriate tool", "handle errors appropriately"

Zero marks (<10): Steps describe actions without showing how to execute them.

Dimension 3: Pitfalls Section (max 20 points)

Lists real-world failure scenarios that have actually been observed, with recovery actions.

Full marks (20): 2–3 specific pitfalls with concrete recovery steps.

• Example: "PDF is password-protected → use --password X flag"
• Example: "LM Studio returns HTTP 400 for gpt-4o → fall back to qwen3.5"

Partial marks (8–19): Has pitfalls but they're theoretical or only 1 item.

• Problem: "something might go wrong", "the tool may fail" — no specificity, no recovery

Zero marks (<8): No pitfalls section, or purely generic warnings.

Dimension 4: Verification Steps (max 15 points)

Explicit checks after execution to confirm success before proceeding.

Full marks (15): At least one concrete verification step per major action.

• Example: "Verify exit code is 0", "Check file exists at ~/.hermes/skills/<name>/SKILL.md"
• Example: "Confirm email was sent by checking Sent folder"

Partial marks (6–14): Has some verification but incomplete or vague.

• Problem: "make sure it worked" — no way to actually verify

Zero marks (<6): No verification steps at all.

Dimension 5: Structure & Conventions (max 15 points)

Consistent structure makes skills scannable and maintainable. Follows the peer-matched pattern from Hermes core skills.

Full marks (15): Has ## Overview section (what and why), ## When to Use with bulleted triggers AND counter-triggers ("Don't use for:"), topic-specific body sections, file size 8-15k chars (peer average ~12k). Uses references/*.md for large supporting content instead of bloating SKILL.md.

Partial marks (6–14): Missing one structural element or inconsistent with peers in the same category.

• Problem: No counter-triggers, no Overview, file > 20k chars without splitting to references

Zero marks (<6): No structure at all — just a wall of text with no sections.

Penalties:

• Missing Overview section: -2 pts
• Missing When to Use section: -2 pts
• No counter-triggers ("Don't use for"): -1 pt
• File > 20k chars without splitting to references: -2 pts
• Inconsistent with peer skills in same category: -1 pt

How Skills Actually Work in Hermes

When auditing, keep the three-phase mechanism in mind:

1. Discovery phase — Hermes scans the available_skills block (the one-line description from each skill's frontmatter). If your description is vague, the router never loads the skill. Nothing inside SKILL.md matters if this step fails.
2. Loading phase — The full SKILL.md loads into context. Now structure, commands, and clarity matter.
3. Execution phase — The model follows the skill. Vague steps, missing commands, and absent verification cause silent failures, especially on smaller models.

Dimension 1 checks discovery. Dimensions 2-5 check loading and execution.

Audit Workflow

Step 0: Prefer class-level umbrellas

If the topic is broad (skill maintenance, config hygiene, verification, authoring), update or extend the umbrella skill for the class rather than creating a one-off session artifact. If the change includes a reusable check or workaround, move the detailed recipe into references/ and keep the SKILL.md body as the durable overview.