Test Evidence Review

!bash "${CLAUDE_SKILL_DIR}/../../hooks/yaml-helper.sh" resolve_config --keys automation,workflow,qa.level,testing.strict

Automation mode: Resolve modes.automation (project.local.yaml → project.yaml → default collaborative). Every AskUserQuestion call and every file write follows .claude/docs/automation-modes.md (collaborative asks always · guided major-only · autonomous logs and proceeds; automation_always_ask categories always prompt).

qa.level and testing.strict (resolved above) decide what this review may call a blocker — the same rules /story-done closes stories by, so a BLOCKING item here is one that would stop /story-done:

• qa.level: minimal waives tests. A Logic, Integration or Config/Data story with no test evidence (for Config/Data, no smoke report) is WAIVED (qa.level: minimal) — not MISSING — and raises no BLOCKING item; a test that does exist is still reviewed, but its gaps are ADVISORY. Visual/Feel and UI evidence (the retained screenshots, and the Visual/Feel sign-off) is never waived.
• Gate level per story type: take testing.strict.<type> from the resolved block (Logic→logic, Integration→integration, Visual/Feel→visual, UI→ui, Config/Data→config): true → BLOCKING, false → ADVISORY, unset → a plain-boolean testing.strict: true|false (legacy form) applies to every type; else the Default Gate Level in .claude/docs/coding-standards.md (BLOCKING for Logic, Integration, Visual/Feel and UI; ADVISORY for Config/Data). Every "BLOCKING" below means that story type's gate level.

Test Evidence Review

/smoke-check verifies that test files exist and pass. This skill goes further — it reviews the quality of those tests and evidence documents. A test file that exists and passes may still leave critical behaviour uncovered. A manual evidence doc that exists may lack the sign-offs required for closure.

Output: Summary report (in conversation) + optional production/qa/evidence-review-[date].md

When to run:

• Before QA hand-off sign-off (/team-qa Phase 5)
• On any story where test quality is in question
• As part of milestone review for Logic and Integration story quality audit

1. Parse Arguments

Modes:

• /test-evidence-review [story-path] — review a single story's evidence
• /test-evidence-review sprint — review all stories in the current sprint
• /test-evidence-review [system-name] — review all stories in an epic/system
• No argument — ask which scope: "Single story", "Current sprint", "A system"

2. Load Stories in Scope

Based on the argument:

Single story: Read the story file directly. Extract: Story Type, Test Evidence section, story slug, system name.

Sprint: Read the most recently modified file in production/sprints/; extract the list of story file paths from the sprint plan.

System: Glob production/epics/[system-name]/story-*.md.

If the resolved scope contains ZERO stories, stop here. Report NOT ASSESSED — no stories in scope, name which scope was searched and which path was empty, and route: no sprint file → /sprint-plan new (at workflow: minimal, which has no sprints, /test-evidence-review [epic-slug] instead); a sprint plan listing no stories → /create-stories [epic-slug]; a [system-name] glob that matched nothing → name the glob. Do not continue to Section 3.

Guard the empty scope, not just the per-story unknown. The verdict vocabulary here — ADEQUATE / INCOMPLETE / MISSING — needs a "could not check" value, or an unverifiable story acquires a verdict claiming somebody verified it; that is what NOT ASSESSED is for. But giving the per-story unknown a home does nothing for the empty-scope unknown. With no stories the Section 6 report renders an empty Summary table and ends BLOCKING items: 0 / ADVISORY items: 0 — which reads as everything reviewed, all fine. This skill gates story closure, and coding-standards.md marks Logic, Integration, Visual/Feel and UI evidence BLOCKING, so a false-clean closes stories nobody reviewed.

This is a recurring shape: the sophisticated inner rule present, the outer boundary unguarded. Ask it of any skill that aggregates — what does this emit when the set is empty?

For the resulting story set, collect the fields below with targeted section greps, not a full read of each story:

Grep pattern="## Test Evidence" glob="production/epics/**/story-*.md" output_mode="content" -A 8
Grep pattern="## Acceptance Criteria" glob="production/epics/**/story-*.md" output_mode="content" -A 15

• Story Type (Logic / Integration / Visual/Feel / UI / Config/Data) and the stated evidence path — both live under ## Test Evidence, so the first grep's -A 8 captures them.
• Acceptance Criteria list — the ## Acceptance Criteria block from the second grep.
• Story slug (from the file name) and System (from the directory path) — no read. Full-read a story only when its Test Evidence section is missing or ambiguous. (In Sprint mode, scope the globs to the sprint plan's story paths.)

3. Locate Evidence Files

For each story, find the evidence. Check the story's stated evidence path first — the one Section 2 captured from its ## Test Evidence section. If it exists, use it. Fall back to searching only when no path is stated or it is not found:

Logic stories: search the engine's unit-test root — tests/unit/[system]/ (Godot), Assets/Tests/EditMode/ (Unity), Source/<Module>/Private/Tests/ (Unreal); see .claude/docs/directory-structure.md — for a file whose name contains the story slug or its system. Test files are named per .claude/rules/test-standards.md, not necessarily [story-slug]_test.*.

Integration stories: the same, under the integration root (tests/integration/[system]/, Assets/Tests/PlayMode/, or Unreal's test root)

• Also check production/session-logs/ for playtest records mentioning the story

Visual/Feel and UI stories: Glob production/qa/evidence/ for a retained image (*.png, *.jpg, *.gif) belonging to this story, and for [story-slug]-evidence.*

Config/Data stories: Glob production/qa/smoke-*.md (any smoke check report)

Note what was found (path) or not found (gap) for each story.

4. Review Automated Test Quality (Logic / Integration)

For each test file found, read it and evaluate:

Assertion coverage

Count the number of distinct assertions (lines containing assert, expect, check, verify, or engine-specific assertion patterns). Low assertion count is a quality signal — a test that makes only 1 assertion per test function may not cover the range of expected behaviour.

Thresholds:

• 3+ assertions per test function → normal
• 1-2 assertions per test function → note as potentially thin
• 0 assertions (test exists but no asserts) → flag as BLOCKING (the story type's gate level; ADVISORY at qa.level: minimal) — the test passes vacuously and proves nothing

Edge case coverage

For each acceptance criterion in the story that contains a number, threshold, or "when X happens" conditional: check whether a test function name or test body references that specific case.

Heuristics:

• Grep test file for "zero", "max", "null", "empty", "min", "invalid", "boundary", "edge" — presence of any is a positive signal
• If the story has a Formulas section with specific bounds: check whether tests exercise at minimum/maximum values

Naming quality

Test function names should describe: the scenario + the expected result. Pattern per engine (.claude/rules/test-standards.md): Godot test_[scenario]_[expected], Unity [Scenario]_[Expected], Unreal <Project>.[System].[Scenario].

Flag functions named generically (test_1, test_run, testBasic) as naming issues — they make failures harder to diagnose.

Formula traceability

For Logic stories where the GDD has a Formulas section: check that the test file contains at least one test whose name or comment references the formula name or a formula value. A test that exercises a formula without mentioning it by name is harder to maintain when the formula changes.

5. Review Manual Evidence Quality (Visual/Feel / UI)

For each evidence document found, read it and evaluate the checks below.

A UI story needs no evidence document. When it has none, its retained screenshots are the evidence: apply only Screenshot / artefact completeness (an image of each screen the story changes — a screen a criterion only navigates to, such as the title screen after Quit, needs none of its own) and Date coverage (the images' dates). A UI story is ADEQUATE on those two — never INCOMPLETE for lacking a doc, its criterion linkage or a sign-off.