Validation Methodology

Validation Methodology

Specifications are claims. Claims require proof. This skill defines how the pipeline transforms claims into proof.

Acceptance Criteria Format

Every module spec receives formal acceptance criteria during Layer 4. Criteria use Given/When/Then format:

### AC-CONFIG-001: Config File Load
**Given** a valid config file exists at the documented path
**When** the application starts
**Then**
- The application reads every declared key from the file
- Unknown keys are logged as warnings, not errors
- Missing optional keys fall back to documented defaults

**Source:** official-docs, source-code, runtime-observation
**Confidence:** confirmed
**Priority:** P0
**Verification:** Automated test

ID Format: AC-{DOMAIN}-{NNN}

• {DOMAIN}: Short uppercase token for the behavioral domain (matches the spec's domain tokens — e.g., CLI, CONFIG, API, AUDIO, PARSER)
• {NNN}: Zero-padded three-digit sequence (001, 002, ...)
• IDs are immutable — retired, never reused

Field Rules

Given/When/Then:

• Use observable terms, not implementation details
• "Given a valid config file exists" not "Given the ConfigLoader is initialized"
• Each Then outcome must be verifiable without implementation internals
• Describe observable state, not UI mechanisms — "system rejects malformed input" not "error banner is visible." Criteria must be testable across any frontend platform.

Priority:

• P0: Must pass. Blocks pipeline. Implementation cannot ship without satisfying.
• P1: Should pass. Reported as warnings. Divergence requires documented rationale.
• P2: Nice to pass. Advisory.

Verification: One of the 5 methods below.

Output Location

workspace/raw/specs/validation/acceptance-criteria/
    _index.md                   # Summary with counts and links
    config-loading.md           # AC-CONFIG-001 through AC-CONFIG-NNN
    cli-flags.md                # AC-CLI-001 through ...
    ...

Verification Methods

Every acceptance criterion specifies how the implementer will verify it:

Quality Gate Criteria

digraph gate_evaluation {
    rankdir=TB;

    "Agent produces output" [shape=ellipse];
    "Check DOD checklist for agent layer" [shape=box];
    "All DOD items pass?" [shape=diamond];
    "Proceed to gate evaluation" [shape=box];
    "Flag incomplete output for review" [shape=box];
    "Check gate criteria" [shape=box];
    "All gate criteria pass?" [shape=diamond];
    "Tag and proceed to next layer" [shape=box];
    "Remediate findings" [shape=box];
    "Attempts exhausted?" [shape=diamond];
    "STOP: Pipeline blocked" [shape=octagon, style=filled, fillcolor=red, fontcolor=white];

    "Agent produces output" -> "Check DOD checklist for agent layer";
    "Check DOD checklist for agent layer" -> "All DOD items pass?";
    "All DOD items pass?" -> "Proceed to gate evaluation" [label="yes"];
    "All DOD items pass?" -> "Flag incomplete output for review" [label="no"];
    "Flag incomplete output for review" -> "Proceed to gate evaluation";
    "Proceed to gate evaluation" -> "Check gate criteria";
    "Check gate criteria" -> "All gate criteria pass?";
    "All gate criteria pass?" -> "Tag and proceed to next layer" [label="yes"];
    "All gate criteria pass?" -> "Remediate findings" [label="no"];
    "Remediate findings" -> "Attempts exhausted?";
    "Attempts exhausted?" -> "STOP: Pipeline blocked" [label="yes"];
    "Attempts exhausted?" -> "Check gate criteria" [label="no"];
}

Gate 1 (spec-verifier, after Layer 3)

• Zero contradictions between specs
• All crypto claims and constants verified against source (not assumed)
• Every behavioral claim has a provenance citation
• Assumed claims are a small minority — most claims have direct evidence. A module dominated by assumptions needs more intelligence gathering before proceeding
• All modules have specs
• Zero uncited behavioral claims in output candidates

Gate 2 (spec-reviewer, after Layer 4)

• Zero implementation leakage in specs
• Zero P0 completeness gaps
• All ACs have IDs and are linked to specs
• P0 ACs have test vectors
• All ACs are testable
• All modules have ACs
• Gap report reviewed
• Zero contamination in output

Implementation Leakage Definition

"Zero implementation leakage" means every identifier, name, and reference in the output specs passes the reimplementor test: "Could the reimplementor reasonably redesign this?" If yes, it's an implementation detail that should have been abstracted. If no, it's an external contract that belongs in the spec.

Implementation details (must not appear in output specs):

External contracts (must be preserved):

See the spec-sanitization skill for the full semantic classification (Implementation Detail vs. External Contract) and rewrite guidance.

Definition of Done Checklists

Layer 1 Agent DOD

• All output files at correct workspace directory for source origin
• All behavioral claims have <!-- cite: --> provenance citations
• Output follows standard format for this agent's intelligence source
• Session JSONL captured to workspace/provenance/sessions/
• No raw source code excerpts longer than one line

Layer 2 Agent DOD

• Feature inventory covers all sources consulted
• Architecture document covers all identified components
• API surface covers all discovered interfaces
• Module map is complete (name, description, priority, dependencies)
• No known gaps (or gaps explicitly documented with [GAP] marker)
• Cross-references to Layer 1 artifacts are valid
• Session JSONL captured