Full Audit

Full Audit — Exhaustive Area Review (v1.4)

Dominant Variable

Accuracy of the coverage claim — the word "exhaustive" ships with a method label or it doesn't ship at all. The moment an unreviewed area gets reported as reviewed, this skill has failed its own purpose.

Trigger

• /full-audit [area] · "audit everything" · "exhaustive audit" · "full audit" · "double-check everything"
• Default mode on any of the above: AUDIT_ONLY. These are analysis requests, not execution requests — see Execution Modes below. Advancing to PROPOSE or APPLY_APPROVED in the same invocation requires the user to say so explicitly (e.g. "audit everything and apply the fixes", "전수감사하고 바로 고쳐줘").

Discard If

• Single file / single question needs checking → use a regular code review instead
• The goal is a harness-maturity score against a fixed checklist → use a fixed-checklist scoring tool instead
• The goal is a single docs-vs-code drift check → too narrow a scope for this
• An identical-scope full audit finished within the last 7 days and nothing has changed → just diff against the existing coverage map instead

---

Key Assumptions

1. Target area is agreed in Phase 0 — if not, don't start without an area table.
2. Deterministic sweeping (scripts/grep) is available for the target — if not, skip Phase 1 and never claim "exhaustive" from Phase 2 (LLM review) alone.
3. A prior coverage map can shrink the scope via diff — if not, do a full re-scan.

Execution Modes: AUDIT_ONLY / PROPOSE / APPLY_APPROVED

This skill keeps analysis and execution in separate, explicitly-named modes. A trigger phrase like "audit everything" or "분석해줘" selects a mode — it does not, by itself, authorize any file write.

One-way per pass: AUDIT_ONLY never auto-advances into PROPOSE or APPLY_APPROVED within the same invocation. Advancing needs a new, explicit user statement. This closes the gap where "audit everything" silently walked all the way to Phase 4 fix-bucket execution — including a Protect-Hooks-guarded file changing — without the user ever approving execution, not just analysis.

Phase 0: Agree Scope + Diff Against Prior Map

1. Declare the execution mode for this run in the first line of the response (e.g. "Mode: AUDIT_ONLY — read-only, no files will change"). Default AUDIT_ONLY unless the request explicitly names PROPOSE/APPLY_APPROVED or explicitly approves specific items up front.
2. Declare the target areas as a table (e.g. codebase / docs / global skills / memory / DB / settings). For code and docs areas, build the file list as the union of git ls-files and git ls-files --others --exclude-standard - a tracked-only list drops uncommitted files, which are the ones most in need of review.
3. If a prior coverage map exists, read it and queue its remaining gaps first.
4. Areas the user explicitly excludes go on the map as "intentionally excluded" — never silently dropped.

Phase 1: Deterministic Sweep

Whatever a machine can count, a script counts — never eyeball it:

• Counts (test count, DB rows, file count) / version stamps (single source of truth in N places) / path and reference existence (dead links)
• Parsing (YAML frontmatter, JSON settings) / stale-number greps (old numbers still lingering) / expiry (TTL, aging)
• No inline throwaway scripts — write a script to a file, run it, then delete it (guards against quoting/escaping mistakes)
• Reuse existing checkers first (test suites, project-specific validation scripts, linters)

Cross-index contract sweep (mandatory sub-step — this is the layer most exhaustive audits skip): The layer most easily missed in "exhaustive" audits is "does the index/routing doc match the real files?" — careful reading of individual files alone will never catch this. Sweep deterministically:

• Index vs. reality reconciliation: names listed in an inventory/index file vs. the actual directory/file listing — check both directions (ghost entries with no backing file, and real files missing from the index)
• Routing vs. reality reconciliation: names a routing table points to vs. whether those targets actually exist (dead routes to archived/renamed targets)
• Declared-dependency sweep: for each unit's declared dependencies (other files/skills/agents it depends on), do all of them actually exist? (including malformed declarations, e.g. a flag where a name was expected)
• Frontmatter parsing integrity: duplicate YAML keys in frontmatter (the later one silently wins — a safety profile could flip silently)
• Rationale: in comparable audits, most of the gap came not from "reading more carefully" but from "did we actually sweep these specific things deterministically" — a model-independent, reproducible methodology improvement.

Coverage caps intervention value (borrowed from arXiv 2608.04618): before starting Phase 2 content review, count how many items this audit could actually affect (e.g., "N files this rule change would apply to"). That count caps the maximum value of the intervention before you've even seen the results — if only 3 items are in scope, no amount of review sophistication can produce more improvement than those 3 items allow. Computing this cap up front prevents over-investing deep-review time in low-cap areas, and gives a concrete basis for shifting review effort toward higher-cap areas instead.

Canary mixing (a control group for telling "clean" apart from "the reviewer missed it"): when reviewing a code area, stage one known-clean file and one file with a planted defect from this skill's bundled example pool (scripts/canaries/{clean,seeded}/) using python scripts/canary_mix.py select --pool both --n-clean 1 --n-seeded 1 --stage-dir <scratch>/<run>/bundle --out <scratch>/<run>/canary-manifest.json, then mix the staged files into one or two real review bundles. Don't tell the reviewer dispatch that a control exists or which file it is, and don't count canaries in the Phase 1 denominator. The bundled pool ships with only 2 clean / 2 seeded example files — extend or replace it with pairs representative of the actual codebase before treating a recall number as meaningful. Docs/rules areas have no canary pool by default; skip this step there and note "canary not applied (no pool for this area)" on the map.

Phase 2: Content Review + Rule Dry-Run (Three-Layer Principle)

Structural checks (Phase 1) alone do NOT justify calling something "exhaustive" — exhaustive = structure + content + rule dry-run, three layers. Rules and guards can't be confirmed as actually working just by reading their documentation — only running them against mock input fills in the third layer. The three layers are non-substitutable: structural checks can come back clean while the content is wrong, and the content can be correct while a rule still fails to fire at runtime.

• Fan out parallel review agents (unlimited breadth for coverage, read-only — never give reviewers edit access)
• Neutral framing: phrase the review dispatch's goal as "judge whether this bundle satisfies policy P — with equal rigor whether it does or doesn't," not "find problems in this code." An instruction to find problems pulls something out of a clean file too.
• Single-reviewer batch size cap (borrowed from arXiv 2609.09696 — in a controlled experiment on 150 academic papers, a review recall that held at 50–60% for 1–10 papers collapsed to 2.8% at 48 papers in a single batch): the line above is about how many parallel agents you fan out; this is a separate axis — how many files/documents one reviewer call scans at once. Even when nominal context capacity looks sufficient, the more items you put in one call, the more likely "no findings" stops being an honest negative and becomes a confident fabrication. For large scopes, lean toward splitting bundles small (the exact cap is domain-dependent, so no number is fixed here).
• Force citations: reviewers must attach a grep/ls output as proof when they claim something is missing — "I can't find it" from memory alone is invalid
• Anti-false-positive 4-bucket (enforce in the review dispatch's output-format instructions): classify every finding as CONFIRMED / FALSE-POSITIVE (reviewed and dismissed, with a refuting citation) / UNCERTAIN (needs inference — keep it, don't discard, to avoid false negatives) / NIT, each with a reasoning note. CONFIRMED at Critical/High needs 2+ of {condition, impact, reproduction} or it gets downgraded to Medium. FALSE-POSITIVE needs the discarded hypothesis + a refuting citation (command output or a line quote) — "no issue" in one line is not acceptable. An empty false-positive list is not a penalty (state "none dismissed" explicitly — this prevents over-suppression).
• Kill-test (enforce in the dispatch instructions): before reporting each finding, run one command that tries to refute it, include the output, and add one line: (refutation check: output refutes the finding / output is unrelated and insufficient). If refuted, it moves to the false-positive bucket. "It's missing" claims must be backed by an exhaustive grep across the whole denominator (explicit regex and scope, grep -rn <pattern> <root> — substring matching alone doesn't count). Passing a mock test alone does not count as a refutation (remove the code and re-test instead). The conclusion needs one objective anchor: a rule/linter, an actual execution result, a direct two-point comparison within the reviewed content, or a grep-derived denominator — "it looks like" with no anchor is invalid (inference-requiring cases aren't invalid, they go to UNCERTAIN instead). Anchor-inject the countables: for anything a machine can count, hand reviewers Phase 1's deterministic values as a given anchor up front rather than asking them to re-derive it — this keeps reviewers out of the business of re-counting what a script already settled.
• Reverse dry-run for guard/lint scripts: seeded-error injection only checks that a rule catches what it should. In the opposite direction, map every check in a guard, linter, or check script to the rule clause (file:line) that justifies it. Report any check with no mapped clause as a false-positive source candidate - checks left without a basis are what block legitimate work.
• Rule dry-run (third layer): if the target area has rules or guards (linter configs, pre-commit hooks, validation scripts), build an actual mock input (a fixture) and run it through the rule to confirm by execution — not by reading — that it detects or blocks what its documentation claims. Static comparison (does the rule's documentation exist) and content review (does the rule's wording make sense) alone can't prove it fires at runtime — skip this layer and a dead guard (documented but inert) slips through unnoticed.

Phase 3: Judgment — Dismissing False Positives

Personally re-verify every reviewer report before classifying. Common false-positive patterns to check for:

• Training-cutoff confusion: "this date/version can't exist yet" — re-check against the actual current date
• "Already exists but reported missing": any reported "gap" must be re-confirmed to actually be missing via grep before being accepted
• Historical notation mistaken for staleness: an original-version marker or changelog entry is history, not staleness — don't "fix" it
• Number conflicts: reviewer's number vs. the Phase 1 deterministic number → deterministic wins
• Composite-accumulation-gate (death-by-thousand-cuts guard) ([borrowed from PHP-AIO, arXiv 2607.15944v1]): even when every individual finding is separately dismissed as FALSE-POSITIVE/UNCERTAIN/NIT, if the same area (same file/module/component) accumulates 3+ UNCERTAIN findings, or 5+ combined (UNCERTAIN+NIT) findings, flag it separately as an "individually-passed, cumulatively-risky" signal — passing each individual threshold does not mean the composite threshold is also safe (structurally identical to the CRITICAL hard-cap principle in agents/code-reviewer.md). A flagged area is not promoted to CONFIRMED, but must be listed at least once in the Phase 4 addition bucket so the user sees it. [The 3/5 thresholds are initial estimates, subject to recalibration once operational data accumulates.]
• Canary scoring (only for a run that mixed in canaries): save this run's verdicts as [{"file": <path>, "verdict": <final bucket>, "summary": <one line>}] JSON and run python scripts/canary_score.py --manifest <canary-manifest.json> --findings <that JSON>. If a clean control file comes back CONFIRMED, mark every CONFIRMED finding from this run as "needs re-verification" on the coverage map — this does not auto-downgrade them. Drop any finding on a canary file from the fix/addition buckets.