Validate Consistency

Universal Document Consistency Auditor v3.0

Table of Contents

1. Overview
2. Truth Invariant
3. Examples
4. Instructions — Steps 0–5
5. Severity Assignment Rules
6. Bootstrap: Drafting Registry Rows
7. Integration with /release
8. Output
9. Error Handling
10. Resources

Overview

Runs deterministic drift checks across all documentation artifacts, resolves conflicts through a per-fact-class authority registry (sot-map.yaml), and produces a structured report with actionable findings grouped by category and severity. Deterministic checks and LLM-judged checks are structurally separated: judged findings are advisory-only and never block anything.

Truth Invariant

The governed brain arbitrates asserted company/doctrine claims and the rules of arbitration — never generated facts. This validator never reads the brain as ground truth for generated facts: authority for a generated fact lives only in a registry row a human has adjudicated. The brain ingests only human-adjudicated findings — raw findings from this skill are candidate evidence, not truth, until a human adjudicates them.

Examples

Basic audit:

/validate-consistency

Runs the full audit on the current working directory. Loads the authority registry and produces a categorized report.

Targeted check after refactor:

I just renamed a bunch of modules — check consistency

Detects that code changed, runs all 9 checks, surfaces broken cross-references (deterministic) and stale capability claims (advisory).

Pre-release gate (via /release):

/release

Phase 1.6 automatically invokes this skill. Deterministic Critical findings block the release; advisory findings are surfaced for review and never block.

Prerequisites

• Must be run from inside a git repository (or directory with recognizable project structure)
• Read access to all project files
• An authority registry (sot-map.yaml) — optional; without one the skill runs in bootstrap mode (see Step 1)

Instructions

Step 0: Announce

echo "════════════════════════════════════════════════════════════════"
echo "  DOCUMENT CONSISTENCY AUDIT"
echo "════════════════════════════════════════════════════════════════"

Step 1: Load the Authority Registry

The source of truth is a per-fact-class authority registry declared as data — never a global ranking, never auto-detected at runtime.

1. Resolve the registry path: the path supplied by the invoker (argument or explicit instruction) if given; otherwise the default ~/000-projects/intent-os/sot-map.yaml.
2. Read and parse the registry. Print what was loaded:

echo "Authority registry: [path]"
echo "Fact classes registered: [n] — [list of class names]"

1. If the registry is missing or unparseable, announce bootstrap mode: every fact class is unowned. Conflicts are still detected and reported, but no winner is ever named — each is emitted as unowned fact-class — human adjudication needed, and the report includes drafted registry rows for a human to adjudicate (see Bootstrap).

Registry row shape:

version: 1
fact_classes:
  version:                              # key = fact_class (snake_case / dotted)
    owner: package-manifest             # required — the ONE producing surface
    mirrors:                            # surfaces expected to restate the fact;
      - readme                          #   each diffed against the owner only
      - changelog                       #   (star topology — never pairwise N×N)
    determinism: deterministic          # deterministic | llm-judged
    depth_tier: T1                      # T1 value-equality | T2 judged/semantic
    staleness_bound: 0                  # days a mirror may lag before lag = drift
    criticality: high                   # critical | high | medium (severity ceiling)
    volatile: false                     # true = mirrors expected to trail (Info-level
                                        #   replication lag inside the bound)
    adjudicated_by: jeremy              # optional provenance — who made the call
    adjudicated_on: YYYY-MM-DD          # optional provenance — when
  license:
    owner: license-file
    mirrors: [readme]
    determinism: deterministic
    depth_tier: T1
    staleness_bound: 0
    criticality: high
    volatile: false

Resolution rule: when two artifacts disagree on a fact of class F — if the registry has a row for F, the artifact belonging to that row's owner class is correct and the finding is filed against the other artifact. If the registry has no row for F, emit the finding as unowned fact-class — human adjudication needed: list both values and locations, name no winner, never guess.

Reference: See references/sot-registry.md for the full registry specification, fact-class and artifact-class vocabularies, and the bootstrap appendix.

Step 2: Inventory Documentation Artifacts

Scan for all documentation artifacts that will be audited:

# Find all documentation files
echo "Scanning for documentation artifacts..."

Use Glob to find:

• README.md, README.*
• CLAUDE.md
• CHANGELOG.md, CHANGES.md
• 000-docs/**/*.md
• docs/**/*.md
• planning/**/*.md
• VERSION, version.txt
• .github/workflows/*.yml, .github/workflows/*.yaml
• Package manifests: package.json, Gemfile, *.gemspec, pyproject.toml, Cargo.toml, go.mod

Print a summary of what was found:

echo "Documentation artifacts found:"
echo "  - README: [yes/no]"
echo "  - CLAUDE.md: [yes/no]"
echo "  - CHANGELOG: [yes/no]"
echo "  - Canonical docs (000-docs/): [count] files"
echo "  - Other docs (docs/): [count] files"
echo "  - Planning docs: [count] files"
echo "  - CI workflows: [count] files"
echo "  - Package manifests: [list]"

Step 3: Run Drift Checks

Execute ALL applicable checks from the list below. Each check is independent — run every check that has the required files present. Skip checks whose input files don't exist (note the skip, don't error).

Every check carries a Lane tag:

• Deterministic — existence, string-equality, and exact-comparison checks. Findings may be Critical, Warning, or Info.
• LLM-judged (advisory) — checks requiring semantic judgment. Findings are ⚪ Advisory, always — never Critical, never blocking.

Reference: See references/drift-categories.md for category definitions and severity guide.

Check 3.1: Index vs Filesystem (Category 7 — Index/Reference Drift)

Lane: Deterministic (referential integrity — no registry row needed; the filesystem is the definitional referent).

If 000-docs/000-INDEX.md exists:

1. Read the index file
2. Extract all file paths listed in the index
3. For each path, verify the file exists on disk using Glob
4. Scan the 000-docs/ directory for files NOT listed in the index
5. Report: missing files (listed but don't exist), unlisted files (exist but not indexed)

Check 3.2: Version String Consistency (Category 1 — Status Drift)

Lane: Deterministic. Fact class: version-string.

Collect version strings from ALL available sources:

• VERSION or version.txt file
• package.json → version field
• *.gemspec → version attribute
• pyproject.toml → [project] version or [tool.poetry] version
• Cargo.toml → [package] version
• CHANGELOG.md → first version header (e.g., ## [1.2.3] or ## 1.2.3)
• README.md → version badges, install commands with version numbers
• CLAUDE.md → any version references

Compare all found version strings. If they disagree, resolve via the version-string registry row and flag all deviations from the registered authority. No row → emit unowned fact-class — human adjudication needed with every value and location; name no winner.

Check 3.3: README Commands vs CI Workflows (Category 4 — CI/Validation Drift)

Lane: Deterministic. Fact class: ci-commands.

If both README and .github/workflows/ exist:

1. Extract shell commands from README code blocks (look for bash, shell, sh fenced blocks and inline backtick commands in "Getting Started", "Development", "Testing", "Usage" sections)
2. Extract run: step commands from all workflow YAML files
3. Compare test commands, build commands, lint commands (exact string comparison)
4. Flag differences, resolving via the ci-commands registry row; no row → unowned fact-class callout

Check 3.4: CLAUDE.md Doc Table vs Actual Docs (Category 7 — Index/Reference Drift)

Lane: Deterministic (referential integrity — no registry row needed).

If CLAUDE.md references specific files or directories:

1. Extract all file/directory paths mentioned in CLAUDE.md
2. Verify each path exists
3. Flag references to non-existent files or directories

Check 3.5: Stale Phase/Status Language (Category 1 — Status Drift)

Lane: LLM-judged (advisory). Fact class: phase-status.

Search all documentation for phrases that may indicate stale status:

• "no application code" / "no code exists" / "scaffold only" / "placeholder" — flag if lib/, src/, or app/ contains actual source files
• "pre-release" / "not yet released" — flag if git tags show version releases
• "planned" / "upcoming" / "future" — flag if the described feature exists in code
• Phase references (e.g., "Phase: Scaffolding") — cross-reference against actual project maturity

Use Grep to search for these phrases, then validate against the filesystem. The grep is deterministic; deciding whether the language is actually stale is semantic judgment, so every finding here is ⚪ Advisory.

Check 3.6: Capability Claims vs Code (Category 3 — Capability/Behavior Drift)

Lane: LLM-judged (advisory). Fact class: capability-claims.

If README has a features section (look for ## Features, ## Capabilities, ## What it does, bullet lists under these headers):

1. Extract each claimed feature/capability
2. For each claim, search the codebase for corresponding implementation
3. Flag overclaims (feature listed but no code found)
4. Also: scan for significant code modules/classes that have NO documentation mention (underdocumented features)

Mapping a prose claim to an implementation is semantic judgment — every finding here is ⚪ Advisory.

Check 3.7: Cross-Doc Fact Comparison (Category 6 — Cross-Doc Contradiction)

Lane: Deterministic for exact-value facts; LLM-judged (advisory) for the description comparison.

Extract and compare key facts across documents:

• License (fact class license): README license mention vs LICENSE file vs package manifest license field — deterministic
• Language/runtime version (fact class runtime-version): README vs package manifest vs CI workflow matrix — deterministic
• Repository URL (fact class repository-url): README badges/links vs package manifest repository field vs git remote — deterministic
• Description (fact class project-description): README first paragraph vs package manifest description vs CLAUDE.md purpose — semantic equivalence is a judgment call, so report in the advisory lane

Flag any contradictions. Resolve each via its fact-class registry row; no row → unowned fact-class callout.

Check 3.8: Broken Cross-References (Category 7 — Index/Reference Drift)

Lane: Deterministic (referential integrity — no registry row needed).

Scan all markdown files for internal links and references:

1. Find all [text](path) links where path is a relative file path (not URL)
2. Find all explicit file path references in prose (e.g., "see docs/setup.md")
3. Verify each referenced file exists
4. Flag broken references with source file, line number, and dead target

Check 3.9: Planning-vs-Implementation State (Category 5 — Planning-vs-Implementation Confusion)

Lane: LLM-judged (advisory). Fact class: planning-status.

If planning/ directory exists:

1. Read planning documents for listed epics, features, milestones
2. Check which planned items have corresponding code implementations
3. Flag: planning says "future" but code exists (stale planning)
4. Flag: docs present planning items as if implemented (overclaim)

Judging whether code "implements" a planned item is semantic — every finding here is ⚪ Advisory.