Verifying Skill Accuracy

Verifying Skill Accuracy

Skills in this repo are drafted with LLM assistance. The methodology in them is usually sound; the specifics are not trustworthy until checked. The measured rate from the first verification pass was 31 factual errors across 11 of 12 skills — about 2.6 per skill, with a single skill clean on both passes. Assume that rate applies to anything unstamped.

This skill is the procedure for driving that rate down, and for knowing when you are allowed to say a skill has been verified.

When to Use

• Auditing an existing SKILL.md against primary sources
• Before adding or renewing a verified: frontmatter stamp
• After drafting new skill content, on the specifics you just wrote
• When a reader reports that a command, ID, or field name does not work
• Reviewing any technical content where the author was a model

When NOT to Use

• Writing a new skill from scratch — use authoring-security-skills, then verify with this skill as the final gate
• Structural or style problems (frontmatter, sections, line count) — that is python3 scripts/validate.py --strict, which checks form only
• Deciding whether a technique is a good idea — that is judgment, not fact

The Core Distinction: Form vs Truth

The repo's CI is a closed loop. validate.py, sync_attack.py, and run_evals.py check frontmatter shape, cross-reference integrity, ATT&CK-index consistency, and routing against self-authored eval cases. Every one of those can pass on a skill whose commands do not exist.

Green CI means the skill is well-formed. It says nothing about whether it is true. Never cite a passing test run as evidence of accuracy.

Sweep by Class Across the Whole Collection

Once you identify an error class, check it everywhere before moving on — not just in the skill that surfaced it. Errors of a class cluster, and a per-skill pass leaves the collection internally inconsistent.

Two cases from this repo make the point. The legacy Sigma date format was fixed in writing-sigma-rules, then turned up again in engineering-detections — in a skill that had already been stamped, because the per-skill inventory did not include a class fixed elsewhere. And attacking-entra-id was still calling retired AzureAD PowerShell cmdlets while investigating-m365-entra already used the modern Graph equivalents: the collection contradicted itself, and only a cross-cutting grep showed it.

So: fix the instance, then immediately grep every skill for the pattern, and say in the commit which classes you swept. Classes worth sweeping in a security collection — tool renames and archived projects, CVE IDs, cloud metadata endpoints, removed API versions, default ports, spec version claims, and any behaviour gated on a platform version.

Triage: Class Errors Before Instance Errors

Do not start by checking facts one at a time. First ask: is there a single systematic claim whose failure invalidates everything below it?

In analyzing-memory-images, the skill invoked Volatility as volatility3. The real entry point is vol. That one error made all 40 commands in the file uncopyable — far more damage than the two nonexistent plugins found afterwards, and fixable with one substitution.

Check in this order:

1. Tool identity — the binary/entry-point name, and the major version the syntax belongs to. Volatility 2 and 3 share almost no command surface.
2. Naming scheme — how the tool names its plugins, modules, or subcommands. Getting the scheme right validates or invalidates dozens of lines at once.
3. Path and file conventions — where artifacts actually live on the version in question.
4. Individual claims — only now, one by one.

A class error is cheap to fix and expensive to miss. An instance error is the reverse.

Source Hierarchy

Use the highest tier available, and record which tier you used.

Two rules on top of the table:

• Match the version. Docs for 2.x do not establish 3.x behaviour. Prefer latest or stable doc branches, and note when the skill targets an older pinned release.
• Ask when it last changed, not just whether it is true. Vendor defaults move: Microsoft's Audit (Standard) retention went 90 → 180 days in October 2023, and material written before that is now wrong without being obviously wrong.

Never Assert These From Memory

Each of these has produced a real error in this repo:

• CLI entry points and subcommand names
• Plugin, module, and package names
• Version numbers, and any "the default is X" claim
• Event IDs and their exact semantics (4778 is session reconnect, not connect)
• Log field and column names
• Retention windows and licence gates
• CVE IDs, CVSS vectors, and affected-version ranges
• API paths, parameter names, required scopes
• Legal precedent and case outcomes — a draft in this repo cited Mango Markets and Platypus as convictions proving exploitation is prosecuted as theft. Both went the other way: the Platypus pair were acquitted in Paris, and Eisenberg's Mango convictions were vacated in May 2025. Memory produced a confident claim that was backwards.

Reading Sources: Fetch Markdown, Not HTML

Pull documentation through defuddle.md, which strips a page to its main content and returns Markdown with YAML frontmatter. Prefix any URL:

curl -sL "https://defuddle.md/learn.microsoft.com/en-us/purview/audit-log-retention-policies"

The scheme is optional in the path — defuddle.md/example.com/x and defuddle.md/https://example.com/x both work.

Two reasons, and the second matters more than the first:

1. It cuts tokens. Measured against the pages used in this repo's passes: 78% smaller on a prose doc page (Microsoft Learn), 33% on a link-heavy API index. Prose collapses hard; link tables less so.
2. It returns the full text, deterministically. You get the whole page as Markdown you can grep, instead of a model's summary of the page. That is what makes it safe to draw a negative conclusion — see below.

Use it for documentation, specs, vendor KB, and articles. Do not use it for:

• JSON or API responses — readability extraction mangles structured data. Fetch those raw.
• HTTP status probes — you need the status of the real host, not of a proxy that may return 200 for its own error page.
• Anything internal, client-owned, or target-owned. The URL leaves your machine and goes to a third party. Never route an engagement URL, an internal hostname, or a client's estate through an external extraction service. Public vendor documentation only.
• Authenticated or JS-rendered pages — it fetches as an anonymous client.

A blocked fetch is not a negative result. Some hosts refuse the extractor and return an error blob instead of the page — freedesktop.org answers with {"error":"Failed to fetch: 418 I'm a teapot"}. Treat that as "I did not read the page", never as "the page does not say this". It is the truncated-negative trap wearing a different hat, and the fix is the same: re-fetch directly before concluding anything.

Programmatic Existence Probes

Where a project publishes one doc page per module, existence is a status code, not a judgment call. This is the highest-confidence, lowest-effort check available, and it batches.

# One plugin: 200 = exists, 404 = does not
curl -s -o /dev/null -w "%{http_code}\n" \
  "https://volatility3.readthedocs.io/en/latest/volatility3.plugins.windows.idt.html"

# Every plugin the skill references, in one pass
plugins=$(grep -oE '\b(windows|linux|mac)\.[a-z_]+' SKILL.md | sort -u)
for p in $plugins; do
  code=$(curl -s -o /dev/null -w "%{http_code}" \
    "https://volatility3.readthedocs.io/en/latest/volatility3.plugins.$p.html")
  [ "$code" = "200" ] || echo "MISSING $code  $p"
done

That pass settled 32 plugin references in seconds and found the two that do not exist. Adapt the URL pattern per project — most doc generators (Sphinx, mkdocs, pkg.go.dev, docs.rs, npm, PyPI) expose a per-symbol or per-package URL you can probe the same way.

When a project has no per-symbol docs, fall back to an exact-string search of the upstream source — a raw file fetch and a grep for the literal identifier. Both of these are deterministic. Prefer them over asking any model, including yourself.