Review Agent Harness

Review Agent Harness

Review the operating system around coding agents, not only its files. Separate declared assets, reachable routes, observed task use, and later outcomes.

Route And Scope

Resolve the directory containing this SKILL.md before running its scripts. Select one mode:

• static: inspect the target repository only. Use by default.
• episode: add explicitly authorized Codex or Claude Code JSONL sources.
• longitudinal: compare a validated report with the existing ledger.

Default to inline, read-only output. Write a durable report under the target only when the user explicitly requests an artifact or historical tracking. Never discover user-home Sessions, read Memory bodies, or inspect another provider merely because its files are available.

Use adjacent skills instead when their narrower owner is sufficient:

• codebase-audit for code defects and architecture health;
• repo-agent-context-audit for AGENTS, Skills, and Specs alone;
• skill-lifeguard for one Skill's reliable contract;
• flowguard for running a long task;
• review-gate before landing an agent-generated diff.

Step 1: Freeze The Evidence Boundary

Record target, mode, provider, locale, decision, acceptance boundary, output mode, included sources, excluded sources, and unavailable evidence. Treat a missing required source as unobserved; do not substitute a broader directory, another provider, or remembered results.

Resolve the target before interpreting assets or assigning scores. The collector classifies it as exact_git_root, inside_git_worktree, contains_nested_git_root, or non_git_directory. If the supplied directory contains a nested Git root, stop and retarget that exact repository; do not score the parent as though it were the project. For a Git target, the collector uses Git's tracked and untracked inventory and excludes ignored worktrees and prior review output from repository evidence.

Run static collection from the installed Skill directory:

python3 scripts/collect_evidence.py \
  --target /absolute/target \
  --mode static \
  --locale zh-CN \
  --decision "assess agent-harness readiness" \
  --acceptance-boundary "resolve all five dimensions" \
  --output-mode inline \
  --output /temporary/evidence.json

For Session-informed review, require the user to authorize exact files or an exact root. Use one provider per evidence envelope:

python3 scripts/collect_evidence.py \
  --target /absolute/target \
  --mode episode \
  --provider codex \
  --session-file /explicit/session.jsonl \
  --locale zh-CN \
  --decision "explain the observed verification gap" \
  --acceptance-boundary "separate configured and exercised routes" \
  --output-mode inline \
  --output /temporary/evidence.json

Use --session-root only when that exact recursive scope was authorized. Add --include-request-summaries only when sanitized request summaries are needed for the decision. Read Privacy Boundary and Session Adapters before Session-informed work.

Omit --output to stream evidence to stdout. Inline means no target writes; environment-owned scratch remains allowed. validate_findings.py --input - accepts findings JSON from stdin when the caller already has a stream.

Checkpoint: collection must return agent-harness-evidence; every stage must be available, constrained, not_authorized, not_applicable, unavailable, or unobserved. A depth-limited scan is constrained, never silently complete. Stop on malformed output or an unexplained missing stage.

Copy the collector-owned scope.target_id and complete scope.snapshot (baseline, target_relation, and id) into the findings document. Never author these values manually. The renderer and ledger updater recompute the binding from --target and reject a different local directory or any target state that changed after collection. A previous report, ledger row, branch name, or remembered result is a historical lead only. Recheck any retained claim against the frozen current snapshot and label genuinely historical evidence as such.

Step 2: Run Three Isolated Evidence Passes

Keep the passes logically independent even when one agent runs them in sequence:

1. Task pass: use the current goal, corrections, acceptance boundary, and authorized Episode facts. Do not infer repository mechanisms.
2. Project pass: use static startup, commands, tests, CI, Git, delivery, and recovery evidence. Do not infer Session behavior.
3. Agent-assets pass: use project Rules, Skills, Hooks, settings, and other configured surfaces. Presence and counts are navigation facts only.

Do not launch parallel agents by default. If the user explicitly requests threads, use threads with read-only lanes and bounded evidence packets. A specialist proposes candidates; it does not assign final severity or claim effectiveness.

Read Review Model before classifying the five dimensions. Use present -> reachable -> exercised -> outcome_supported only when each stronger state has direct evidence.

Resolve all 15 stable checks, three per dimension. Assign a score to each dimension only after resolving its checks. The score is an evidence-bounded summary, not a finding: present caps a dimension at 74, reachable at 84, exercised at 94, and outcome_supported at 100; missing or unobserved caps it at 59. Use the lowest applicable check ceiling and retain a short score rationale. Do not compute an overall score.

Step 3: Reconcile Findings

Retain each distinct eligible candidate. Merge only when consequence, root cause, owner, and verifier are the same. The lead alone assigns severity, confidence, primary dimension, verification state, and priority.

Read Finding Contract. Every finding needs:

• an observed consequence or exact governing requirement;
• a bounded evidence reference;
• a cause chain and smallest owner;
• an executable repair route;
• a machine-checkable verifier.

Counts, file absence without a requirement, similarity, theoretical risk, score, or unavailable evidence never create a finding. Critical and High findings require an adversarial check; retain an unavailable check as unverified instead of presenting it as confirmed.

Record each executed verifier in verification_runs with a stable id, purpose, result, exit code, final-state flag, and bounded summary. A confirmed Critical or High finding must cite a final-state candidate_refutation or targeted_reproduction run that supports the claim. Inspect aggregate exit semantics: a child syntax error or failed subcheck paired with aggregate exit 0 is evidence of a false-green verifier, not a passing check.

Author one agent-harness-findings JSON object in environment-owned scratch space, then validate it:

python3 scripts/validate_findings.py --input /temporary/findings.json --strict --json

Fix the findings data, not the validator. Stop if validation does not pass.

Step 4: Report Or Track History

For inline review, render the overview, frozen snapshot, five-dimension scorecard, all 15 checks, structured verification runs, findings, evidence boundary, and at most three priority moves in the response. Do not write to the target.

When durable output is explicitly requested, render atomically:

python3 scripts/render_report.py \
  --findings /temporary/findings.json \
  --evidence /temporary/evidence.json \
  --target /absolute/target \
  --out /absolute/target/.agent-harness-review \
  --json

The renderer refuses to replace an existing run and writes only validated findings.json, privacy-safe evidence.json, and derived report.md.

For longitudinal mode, update the ledger after a fresh review:

python3 scripts/update_ledger.py \
  --findings /temporary/findings.json \
  --target /absolute/target \
  --ledger /absolute/target/.agent-harness-review/ledger.json \
  --json

An absent prior finding remains open with recheck_required until a targeted spot-check produces an agent-harness-resolution-confirmations document. Each confirmation must retain the finding id, verifier, and one bounded evidence_ref. Pass it with --resolution-confirmations /temporary/confirmations.json. Never resolve from finder absence or an id-only assertion.