Second Pass Review

Second-Pass Review (Layer 6)

Sanitization is necessary but insufficient. The author of a sanitized spec can't reliably catch their own leaks — the same assumptions that let implementation details slip in during the rewrite also let them slip past a self-review. A fresh read, with no access to the raw analysis, catches drift the author missed.

Why This Exists

Sanitization alone misses contamination in predictable categories:

• Structural leakage: Module names in Part headers, cross-module dependency tables, module counts carried into the output
• Content contamination: Minified identifiers, line number references, IPC channel names, state-store selectors, internal filenames
• Behavioral completeness: Usually passes, because completeness is the easiest dimension to check

Root causes:

1. Sanitization agents copy-paste instead of rewriting
2. Gate 2 flags "zero leakage" but never defines what to look for
3. Grep patterns can't catch descriptive English identifiers
4. Without a second-pass review, authors can't catch their own blind spots

Audit Pipeline

digraph audit_pipeline {
    rankdir=TB;
    compound=true;

    "Layer 5 sanitization complete" [shape=doublecircle];
    "Audit PASS — proceed to implementation" [shape=doublecircle];
    "STOP: Audit failed after 3 remediation attempts" [shape=octagon, style=filled, fillcolor=red, fontcolor=white];

    subgraph cluster_specialist {
        label="Parallel Specialist Reviewers (read every file, apply semantic checklists)";
        style=dashed;

        "structural-leakage-reviewer" [shape=box];
        "content-contamination-reviewer" [shape=box];
        "behavioral-completeness-reviewer" [shape=box];
    }

    subgraph cluster_generalist {
        label="Parallel Deep-Read Generalists (4-5 files each, full contextual judgment)";
        style=dashed;

        "deep-read-auditor-1" [shape=box];
        "deep-read-auditor-2" [shape=box];
        "deep-read-auditor-N" [shape=box];
    }

    "Merge all audit reports" [shape=box];
    "Any FAIL?" [shape=diamond];
    "Dispatch targeted fix agents" [shape=box];
    "Re-audit failing files" [shape=box];
    "Attempts < 3?" [shape=diamond];

    "Layer 5 sanitization complete" -> "structural-leakage-reviewer";
    "Layer 5 sanitization complete" -> "content-contamination-reviewer";
    "Layer 5 sanitization complete" -> "behavioral-completeness-reviewer";
    "Layer 5 sanitization complete" -> "deep-read-auditor-1";
    "Layer 5 sanitization complete" -> "deep-read-auditor-2";
    "Layer 5 sanitization complete" -> "deep-read-auditor-N";

    "structural-leakage-reviewer" -> "Merge all audit reports";
    "content-contamination-reviewer" -> "Merge all audit reports";
    "behavioral-completeness-reviewer" -> "Merge all audit reports";
    "deep-read-auditor-1" -> "Merge all audit reports";
    "deep-read-auditor-2" -> "Merge all audit reports";
    "deep-read-auditor-N" -> "Merge all audit reports";

    "Merge all audit reports" -> "Any FAIL?";
    "Any FAIL?" -> "Audit PASS — proceed to implementation" [label="all PASS"];
    "Any FAIL?" -> "Dispatch targeted fix agents" [label="any FAIL"];
    "Dispatch targeted fix agents" -> "Re-audit failing files";
    "Re-audit failing files" -> "Attempts < 3?";
    "Attempts < 3?" -> "structural-leakage-reviewer" [label="yes"];
    "Attempts < 3?" -> "STOP: Audit failed after 3 remediation attempts" [label="no"];
}

Scope Rules

• READ ONLY from workspace/output/ — auditors check what reached the output without cross-referencing the raw analysis
• Detailed reports with per-finding evidence are written to workspace/raw/audit/ alongside other analysis artifacts
• Top-level summaries with PASS/FAIL verdicts are written to workspace/output/audit/ so downstream consumers see review outcomes
• Summaries describe outcomes, not source-derived examples: a category PASS or FAIL with counts, plus overall PASS/FAIL. Per-finding details live in the raw-side reports for remediation; they don't belong in the implementer-facing output.

Why This Is LLM-Based, Not Grep-Based

Pattern matching is fundamentally the wrong tool for contamination detection. It produces both false positives and false negatives at rates that make it unreliable as a primary mechanism.

False positives are pervasive. Patterns that detect minified identifiers also match regex character classes, priority levels (P0, P1), and standard technology names. Patterns for code structure language match natural English ("cannot function without"). Patterns for module counts match legitimate behavioral quantities ("supports 3 output formats"). An auditor drowning in false positives either wastes time on noise or starts ignoring real findings.

Contextual contamination is invisible to patterns. No regex can detect:

• A legitimate-looking constant name that is actually an internal implementation detail
• Behavioral prose that describes code structure rather than observable behavior
• Tables that mix clean behavioral content with contaminated source references
• Feature flags and telemetry events that look like ordinary string constants

Semantic judgment catches everything patterns miss. The core test is simple: "Would an implementor encounter this exact string without seeing the source code?" An LLM can apply this test to every identifier, name, and reference in a spec, using contextual understanding that patterns lack. This catches minified symbols, internal function names, vendor-specific flags, and structural leakage — all with a single semantic criterion rather than a brittle pattern library.

LLM-Based Audit Protocol

All auditors — specialist reviewers and deep-read generalists — are LLM agents that read files end-to-end and apply semantic judgment. There is no grep triage phase.

Specialist Reviewers

Three parallel agents (structural-leakage-reviewer, content-contamination-reviewer, behavioral-completeness-reviewer) each read every file in workspace/output/specs/ and apply their category-specific checklists. See the role sections below for their checklists.

Each specialist reviewer:

1. Reads every file in workspace/output/specs/ end-to-end
2. For each finding, applies the semantic test: "Would an implementor encounter this exact string without seeing the source code?"
3. Classifies findings by their category checklist (SL-1 through SL-8, CC-1 through CC-12, or BC-1 through BC-9)
4. Produces a structured verdict per file (PASS/FAIL with evidence)

Deep-Read Generalists

In parallel with the specialists, dispatch deep-read auditor agents (one per 4-5 domain files) with this prompt template:

Role: deep-read auditor
Skill: second-pass-review

Read each of these files END TO END:
- workspace/output/specs/domains/<file1>.md
- workspace/output/specs/domains/<file2>.md
- ...

## The Test

For every identifier, name, or reference in the spec, ask:
**"Would an implementor encounter this exact string without seeing the source code?"**

If YES (it's in official docs, CLI help, config schemas, API specs, protocol standards) → CLEAN.
If NO (it came from reading the source code, observing internal state, or reverse engineering) → CONTAMINATION.

This test catches everything: minified gibberish, internal function names, internal feature flags, internal telemetry events, source file paths, and line numbers. No pattern list needed — just judgment.

## Common traps

- **Descriptive camelCase**: `shouldRetryOnTimeout` looks like English but it's an internal variable name. A different developer would call it something else. CONTAMINATION.
- **Feature flags with vendor prefixes**: vendor-prefixed flags (e.g., `<vendor>_experiment_*`, `ff_<scope>_*`) — these are internal, not user-facing. CONTAMINATION.
- **Telemetry event names**: `request_rate_limited`, `worker_shutdown_initiated` — internal analytics. CONTAMINATION.
- **Config property names that look behavioral**: `strictSchemaValidation` — is this the actual config key users write, or an internal state property? Check the config schema spec to verify.

## What is NOT contamination

- Environment variables users set: `DATABASE_URL`, `CACHE_DISABLE_TTL`
- CLI flags: `--format`, `--workers`, `--follow`
- Config file keys users type: `upstreams`, `log_level`, `read_timeout`
- API/protocol field names: `Retry-After`, `Content-Type`, `Authorization`
- Public library/tool names: `jq`, `ffmpeg`, `pandoc`
- Standard tech names: `OAuth`, `PKCE`, `TLS`, `JWT`
- IDE names: `PyCharm`, `GoLand`, `VSCode`

For each file, report:
1. PASS or FAIL
2. If FAIL: exact line numbers and the contaminating text
3. Brief explanation of why it's contamination (applying the test above)

Write findings to workspace/raw/audit/deep-read-<batch>.md

Targeted Fix

For each file that failed any audit (specialist or generalist), dispatch a fix agent:

Role: contamination fixer
Skill: spec-sanitization (contamination taxonomy + rewrite patterns)

Read workspace/output/specs/domains/<file>.md

The audit found these specific issues:
[paste exact findings with line numbers]

Fix ONLY the identified issues. Do NOT rewrite clean content.
For each fix, apply the appropriate rewrite pattern:
- Minified IDs → remove or describe the behavior
- Line numbers → remove the "line NNN" reference, keep the behavioral content
- Module names → replace with behavioral domain description
- Feature flags → "a feature gate controlling [behavior]"
- Part N: headers → behavioral domain name

Write the corrected file back to the same path.

Re-verify

After all fixes, re-run audit (specialists + deep-read) on previously-failing files. Repeat up to 3 times.

Agent Sizing

• Specialist reviewers: 3 agents, each reads all files for their category
• Deep-read generalists: 4-5 domain files each (keeps context manageable)
• Fix agents: 1 file each (focused, targeted fixes)
• Parallelism: All specialist and deep-read agents run in parallel; all fix agents run in parallel

Role: structural-leakage-reviewer

Checks whether the organizational structure of workspace/output/ leaks the original's internal module decomposition.

Why this matters: output specs describe behavior, not the original's architecture. An implementer anchored to the original's module structure will reproduce its design — even when a different decomposition would be better for the new implementation. Module counts, inter-module tables, and domain-to-module mappings all encode architectural decisions that should be the implementer's to make fresh.

Semantic Checklist (8 categories)

Read every file in workspace/output/specs/ end-to-end. For each file, check all 8 categories. Apply the core test: "Would an implementor encounter this exact string without seeing the source code?"

SL-1: Module Names in Headers

Scan all #, ##, ### headers. Flag any header that contains a source module name, internal component name, or internal package name rather than a behavioral domain description.

A header like "Session Management" is behavioral (PASS). A header like "Fx3 Module" or "Part 3: CacheManager" is structural leakage (FAIL).

SL-2: Part N: Headers

The Part N: numbering pattern reveals the original module decomposition count and ordering. Any Part N: header is FAIL. Clean specs use behavioral domain names, not numbered parts.

SL-3: Module Counts

Any reference to a specific count of modules, components, or source files reveals the internal decomposition. FAIL unless the count refers to a user-facing concept (e.g., "supports 3 output formats").

SL-4: Cross-Module / Inter-Module / Module Boundary Sections

Section headings or content that reference module-to-module relationships (e.g., "cross-module", "inter-module", "module boundary", "module interface", "module dependency"). All are FAIL. Clean specs describe behavioral integration, not module interfaces.

SL-5: Module IDs (MOD-NNN)

Internal module identifiers from the analysis module map. Any MOD-NNN reference is FAIL.

SL-6: Interface Consumption Tables

Tables showing which module consumes which module's interface. Look for language like "consumes", "consumed by", "provides to", "depends on module", "module provides".

"Depends on the API key being set" is behavioral (PASS). "Depends on AuthModule" is structural (FAIL).

SL-7: Domain-to-Module Mappings

Any text that maps a behavioral domain back to its source module(s). Look for language like "from module", "source module", "originally in", "derived from module", "mapped from". All are FAIL.

SL-8: Raw Paths

Any reference to workspace/raw/ or files within it. All are FAIL.

Structural Audit Output

Write detailed report to workspace/raw/audit/structural-leakage.md:

• Per-category: PASS/FAIL, match count, file:line evidence for each match
• Summary: total categories checked, total findings, overall PASS/FAIL

Write summary to workspace/output/audit/structural-summary.md:

• Per-category: PASS/FAIL only
• Overall: PASS/FAIL
• Report counts only (e.g., "N findings in M files"); the per-finding examples live in the detailed raw-side report.

Role: content-contamination-reviewer

Checks whether implementation-specific content from the original source survives in workspace/output/.

Semantic Checklist (12 categories)

Read every file in workspace/output/specs/ end-to-end. For each file, check all 12 categories. Apply the core test: "Would an implementor encounter this exact string without seeing the source code?"

CC-1: Minified Identifiers

Short alphanumeric tokens that are artifacts of minification or bundling (e.g., Ab2, sp, r0, Fx3, Qz). Look for 2-3 character tokens that have no English meaning and appear in code-like or identifier-like contexts.

Two-letter abbreviations like "UI", "IO", "ID" are OK. Behavioral acronyms like "API", "CLI", "SDK" are OK. If the token has no English meaning, it's contamination.

CC-2: Line Number References

References to specific source code line numbers (e.g., "line 42", "L157", ":234:"). FAIL unless referencing a user-facing output (e.g., "error reported at line 5 of the config file").

CC-3: Source File Paths

Internal source file paths, import paths, or bundle chunk references. Look for paths containing src/, lib/, dist/, package manager directories, source file extensions in path context, or bundle chunk references (e.g., chunk-042).

FAIL for all internal paths. Clean specs reference user-facing paths only (e.g., ~/.config/app/settings.json).

CC-4: Internal Function/Class Names

Named functions, classes, or methods from the source code. Look for camelCase identifiers that look like function/method names (e.g., parseArgs(), serializeResponse()), and PascalCase identifiers that look like class names (e.g., JobRunner, QueueHandler).

Well-known compound words like "JavaScript", "TypeScript", "WebSocket" are legitimate (PASS). Internal names that a different developer would name differently are contamination (FAIL).

CC-5: Code Structure Language

Phrases that describe internal code organization rather than observable behavior. Look for text that names specific functions, classes, or methods and describes calling relationships between them.

"Calls the API endpoint" is behavioral (PASS). "Calls parseArgs() which returns..." is code structure (FAIL).

CC-6: IPC Channel Names

Internal inter-process communication channel identifiers. Look for IPC-prefixed names, channel string literals, or message-passing identifiers that are internal to the implementation.

Rewrite guidance: Replace with "internal messaging mechanism" or describe the behavioral outcome instead.