Verify

/verify

Verify that a change does what its plan said. Report what you observe and keep the receipts. Never fix the code being judged.

This workflow has exactly two halves. Half one creates acceptance criteria and stops for approval. Half two runs only after the user approves those criteria by saying go or an equally explicit instruction.

Hard rules

• Never fix what you judge. Report only.
• Mutation is allowed. Writing to shared or staging systems needs the user to say yes first.
• Provisioning anything that costs money needs the user to say yes first.
• Not checked is always printed, even when empty.
• Expense is not a reason to skip. If a criterion can be driven but costs real setup, say what it would take and let the user decide. Never decide that on their behalf.
• A Not checked reason states why it was not driven. It never asserts that something else covers it, unless it names that thing and says plainly this run did not re-run it.
• Drive the system the way a user does. This workflow does not read or run the repository's unit tests.
• An expectation comes from the plan or from the base commit. Never from the diff. The diff can only expose gaps, and a gap is a question for the user, not a criterion you answer yourself.
• Generated tests stay under the run's tests/ directory until the user explicitly chooses to check them in.
• Make one observation per approved criterion. Do not collapse a harness failure into a behavior failure.

Engine calls

The engine is a local TypeScript package with no installed verify binary. Every invocation must be self-contained because shell variables and working directories do not survive between tool calls.

Use this exact resolution rule at every call site, replacing the verb and arguments as needed:

VERIFY_PIPELINE="${VERIFY_PIPELINE:-$CLAUDE_PLUGIN_ROOT/pipeline}"
(cd "$VERIFY_PIPELINE" && npx --no-install tsx src/cli.ts <verb> [arguments])

CLAUDE_PLUGIN_ROOT is set by Claude Code for an installed plugin. Set VERIFY_PIPELINE yourself only when running from a development checkout. Never hardcode a path to someone's home directory; if neither variable resolves, stop and say so rather than guessing.

The environment helpers are bash scripts resolved the same way:

VERIFY_SCRIPTS="${VERIFY_SCRIPTS:-$CLAUDE_PLUGIN_ROOT/scripts}"
bash "$VERIFY_SCRIPTS/<name>.sh" [arguments]

Once per machine, install the engine's dependencies. The plugin ships TypeScript source and no node_modules, so the first call fails without this. It is a lockfile-pinned install inside the plugin's own directory, it touches nothing in the target repository, and it is safe to re-run:

VERIFY_PIPELINE="${VERIFY_PIPELINE:-$CLAUDE_PLUGIN_ROOT/pipeline}"
if [ ! -x "$VERIFY_PIPELINE/node_modules/.bin/tsx" ]; then
  (cd "$VERIFY_PIPELINE" && npm ci)
fi

Run that before the first engine call of a session. Do not fall back to plain npx, which would fetch an unpinned package from the network.

Optional recorder garnish. Driven criteria get an engine-recorded transcript from their receipts; no PTY is involved. asciinema and agg are attempted only for hand-driven flows, under hard wall-clock limits. A missing, hanging, or broken recorder never blocks verification: record it under Not checked and continue. On a fresh box: brew install asciinema agg, or the equivalent for the platform.

The skill runs in the target repository, not the plugin repository. Resolve every target-repository path with pwd -P and pass absolute paths for --repo, --dir, --criteria, --results, and --claims.

Half one: criteria, then stop

0. The setup contract

First pull anything a fresh worktree is missing from the per-repo shared store (captured login state; a fallback env file stays in the store and is picked up automatically by the environment scripts):

VERIFY_SCRIPTS="${VERIFY_SCRIPTS:-$CLAUDE_PLUGIN_ROOT/scripts}"
bash "$VERIFY_SCRIPTS/shared-store.sh" pull

.verify/setup.json records how this repo boots, seeds, and reports health (written once by /verify-setup from sniffed candidates). If it is missing, ask once: "No setup contract found. Run /verify-setup (recommended), or continue in plain-command mode without a managed stack?" On plain-command consent, write a minimal contract inline:

{"mode": "none", "compose_file": null, "boot": "", "teardown": "", "seed": [],
 "seed_data_files": [], "health_url": "", "base_url": "", "env_file": "",
 "observe": {}, "probes": {}}

Never silently proceed without one. Verify never asks for or stores sensitive credentials; the contract may at most name one of the repo's own local .env files.

1. Find the plan

Look in this order:

1. The current conversation, including an explicit path supplied with /verify.
2. docs/plans/.
3. .omx/plans/.
4. The current pull request body, when available.

Use the newest plan that clearly describes the current change. If no plan is available, ask the user for one and stop. Do not derive criteria from the diff alone.

The diff raises questions. It never answers them.

This is the line that keeps verification black box. An expectation comes from the plan, or from the behavior that existed before the change, or from an assumption you label invented so the user can correct it. Never from the implementation you are verifying.

That is what the source field records, and why invented is a legitimate value rather than an admission of failure. A plan that says "field values persist" without naming a field leaves you a choice. Making it and flagging it loudly is honest. Resolving it by opening the code and testing whatever it happens to do is not, because that criterion cannot fail.

So do not go reading the implementation for things to test. If the plan says "bound every outbound call with a timeout" and does not say what the budget is, the criterion is not "raises at 30 seconds" because you found DEFAULT_TIMEOUT = (5, 30) in the diff. That criterion passes by construction. Ask what the budget should be.

What the diff is for. Once the criteria are drafted from the plan, read the diff once to find gaps. Mark two things:

• additions the plan never asked for
• anything removed or narrowed: a deleted route, a dropped case, a tightened pattern, a reordered rule, a changed default

Each one goes in the approval artifact as a question, naming the change that prompted it and asking what the behavior should be. You do not answer it yourself. A question the user answers becomes a criterion with {"kind": "inferred", "from": "..."} whose expectation is theirs. A question they wave off goes in the uncovered list, where it stays visible.

The difference is between asking "the /confluence/* route was deleted and the plan never mentions it, was that intended?" and deciding for yourself what that route ought to do. The first found a production regression. The second writes a criterion that agrees with whatever the code now does.

Two real examples of what one pass catches. A diff added two clear() calls, in an early return and an error handler, that the plan never asked for; one of them was the bug, and it was on screen while the criteria were being drafted. A different diff deleted an ingress route in the same hunk that added a new one, while the backend still served it; nothing in the plan mentioned the deletion, because a plan records what someone meant to add and is silent on what went out with it.

If an earlier run on the current branch has a criteria.md, show its path and offer to start from it. Stop for the user's choice before replacing or reusing it. A run drafted before criteria carried source.quote and why still reports, with the gap printed where the citation would be, but its criteria cannot be approved again as they stand: add both, from the spec, to every criterion you carry forward.

2. Create the run

Ensure .verify/ is ignored by the target repository. Add .verify/ to its .gitignore when missing.

Create the run and persist its identity in one tool call:

TARGET_REPO="$(pwd -P)"
RUN_ID="$(date +%Y%m%d-%H%M%S)"
mkdir -p "$TARGET_REPO/.verify/runs/$RUN_ID/tests"
printf '%s\n' "$RUN_ID" > "$TARGET_REPO/.verify/current-run"
printf '%s\n' "<path to the plan chosen in step 1>" > "$TARGET_REPO/.verify/.spec_path"

The empty tests/ directory is intentional and must exist even when no test is generated.

.spec_path is how the engine and the second-opinion reviewer get the spec: the engine looks every quote up in it, and the reviewer reads it. When the plan is not a file (it came from the conversation or a pull request body), write it verbatim to $TARGET_REPO/.verify/runs/$RUN_ID/spec.md first and record that path.

Snapshot the working tree now — before any model-driven step (including the second-opinion reviewer) runs with permissions — so the run can later prove nothing outside .verify/ changed (staged changes included):

git diff HEAD > .verify/pre-run.diff
git ls-files -o --exclude-standard | grep -v '^\.verify/' | while IFS= read -r f; do printf '%s %s\n' "$(git hash-object "$f" 2>/dev/null || echo missing)" "$f"; done > .verify/pre-run-untracked.txt

3. List changed behavior files

Choose the branch's merge base. Prefer the PR base or upstream merge base; do not guess a different branch when repository metadata supplies one.

Run:

TARGET_REPO="$(pwd -P)"
RUN_ID="$(cat "$TARGET_REPO/.verify/current-run")"
VERIFY_PIPELINE="${VERIFY_PIPELINE:-$CLAUDE_PLUGIN_ROOT/pipeline}"
(cd "$VERIFY_PIPELINE" && npx --no-install tsx src/cli.ts changed-files \
  --repo "$TARGET_REPO" --base <merge-base>) \
  > "$TARGET_REPO/.verify/runs/$RUN_ID/changed-files.json"

The result includes committed branch changes, staged and unstaged changes, and untracked behavior files. Test files are excluded.

4. Draft criteria

Translate the plan into concrete, observable criteria. Each criterion has:

• id: stable AC1, AC2, and so on.
• title: one behavior.
• plain: reader-facing claim, one sentence with no implementation jargon. Draft it for every new criterion: this is the report-card headline the user approves. The engine keeps the field schema-optional only so older runs can re-render, falling back to title.
• doIt: the intent of the real action, kept short because an approved drive plan is the execution authority when one exists.
• expectIt: a measurable observation.
• source: { "kind": "plan", "ref": "...", "quote": "..." }, { "kind": "inferred", "from": "..." }, or { "kind": "invented", "note": "..." }. For a plan source, ref says where in the spec (a requirement id, a heading, a line) and quote is the spec's own words, copied verbatim, that the criterion was read from. Keep the quote to the sentence or clause that carries the requirement. The engine looks every quote up in the spec and lists the ones it cannot find under the table, so a paraphrase has to be reworded or the criterion relabelled. For an inferred source, from names the diff observation and the answer the user gave. For an invented one, note states the assumption.
• why: one sentence on why this check exists: the bug it would catch, or what would go wrong if the behaviour did not hold. Not a restatement of the title. "The retry schedule is the whole feature" and "over-eager code clears valid selections too" are reasons; "checks that retries work" is the title again. The engine rejects a why that is the title or the plain claim word for word.
• intent: "changes" or "preserves". What the criterion is for.
• baseline: "fail", "pass", "not-applicable", or "unknown". What you expect the base commit to do with it.
• witness: "success" or "refusal". Does this criterion show something working, or show something correctly turned away?
• dependsOn: the parts of the system the criterion drives or observes, from exactly "api", "db", "worker", "browser", "sink", "storage". Half two probes each named part once before judging; a down part marks only its dependents "could not run".
• proof: how a reader will know the check actually ran. One of {"kind": "marker-in-data", "detail": "..."} (the run marker woven into created data — the strongest form), {"kind": "marked-request-rejected", "detail": "..."} (the rejection paired with the marked request), or {"kind": "live-read", "detail": "..."} (a value read fresh during the run, not a stale capture). A criterion you cannot name a proof for is defective: move it to skipped with the reason "no way to prove it ran".
• drive: for criteria reachable through generic command, HTTP, or read-only database surfaces, an ordered plan of exactly these verbs: http, db, wait, run. Each step is { "verb": "...", "args": ["..."], "timeoutSeconds": 60 }; omit the timeout for the 60-second default. Plans are execution authority and run verbatim, so never put shell strings, database writes, assertions, or app-specific verbs in them. UI criteria remain plan-less and hand-driven.
• A driven marker-in-data proof also declares "step": N and optional "expect": "present" | "absent" (default present). The designated step is the only output the engine uses for mechanical proof. expectIt remains the judge's behavioral rubric.

intent is what the criterion is for. baseline is a separate claim about the base commit. Keeping them apart matters, because the obvious shortcut of defining changes as "fails on base" is wrong for real criteria:

• A v1 worker consumes a job written by v2, and a v2 worker consumes a job written by v1. A legitimate criterion for a schema change. It needs both versions running at once, so the base commit alone cannot pass or fail it. intent: changes, baseline: not-applicable.
• With the new flag off, the old behavior is unchanged. Preservation. But the base binary rejects the unknown flag and will not start. intent: preserves, baseline: not-applicable.

changes means the criterion exists to prove the change did something. Its normal baseline is fail.

preserves means it guards something the change could have broken. Its normal baseline is pass, and that is the point rather than a defect. A preserves criterion names what it guards against: the changed route, setting, or file that could plausibly have broken it.

The check is one line: intent: changes with baseline: pass is a defect. That criterion passed before the change existed, so it proves nothing. Everything else is a question of honesty about the baseline, not a rule violation.

Declaring a baseline obliges you to picture the base commit while you are drafting. That is the whole mechanism. Most free passes are visible the moment you ask "would this have passed last week?" and invisible three steps later when the run comes back green.

unknown is a real answer and a cheap one. It goes in the approval artifact as a question. An honest unknown is a much smaller failure than a confident fail that turns out to be a free pass.

witness exists because a set can be all-refusal and look complete. Six criteria that all check something is rejected are satisfied by an implementation that rejects everything. Count the boxes before you submit:

Every new capability needs at least one success + changes criterion: the thing works, end to end, driven the way a user drives it. That box being empty is the most common way a criteria set passes while proving nothing.

Write every criterion as a user would experience it, not as the code is structured. A criterion names a request, a command, a message, or a screen, and the observable it produces. It never names a function, a class, or an internal call. If a criterion cannot be checked without reading source, it is the wrong criterion: rewrite it as something a customer could do.

Good: POST /api/v1/events with a staging key, then read the row: environment_id is the staging environment.

Bad: resolvePayloadEnvironment returns the key-bound environment.

This is why the workflow does not inspect the repository's existing tests. Whether a unit test exists, and whether it mocks something, is a question about the test suite. This workflow answers one question only: does the change work when driven the way a user drives it.

Ask what the laziest implementation that passes would look like. For every criterion, before it goes in the table: could a stub, a constant, or a function that always does the same thing pass this? If yes, the criterion is an observation, not a check.

Real example. A criterion read "a stale selection is cleared from state, URL, and local storage." It passed. Code that clears the selection unconditionally, always passes it just as cleanly, and that was the bug: a valid selection was being thrown away on every page load. The criterion tested the rule in one direction only, and the broken build satisfied it.

So every rule with an on and an off gets both criteria. If a criterion says something is cleared, hidden, rejected, disabled, filtered, or logged out, write its opposite: a valid one is kept, shown, accepted, enabled, passed through, stayed logged in.

The negative side is nearly always where the bug lives, because over-eager code satisfies the positive side by accident:

A plan describes what a change adds. Criteria copied from a plan inherit that blind spot, and nothing in the plan describes what must not break. Writing the opposite criterion is how you get it back.

5. Compute criterion coverage

Write .verify/runs/<id>/claims.json as a mapping from criterion id to changed files it claims, for example:

{
  "AC1": ["src/assets.ts"],
  "AC2": ["src/auth.ts"]
}

Then compute uncovered files from that artifact:

TARGET_REPO="$(pwd -P)"
RUN_ID="$(cat "$TARGET_REPO/.verify/current-run")"
RUN_DIR="$TARGET_REPO/.verify/runs/$RUN_ID"
VERIFY_PIPELINE="${VERIFY_PIPELINE:-$CLAUDE_PLUGIN_ROOT/pipeline}"
(cd "$VERIFY_PIPELINE" && npx --no-install tsx src/cli.ts changed-files \
  --repo "$TARGET_REPO" --base <merge-base> --claims "$RUN_DIR/claims.json") \
  > "$RUN_DIR/coverage.json"

Write .verify/runs/<id>/criteria.json in this shape, copying coverage.json.uncovered into uncoveredFiles:

{
  "criteria": [],
  "uncoveredFiles": []
}

Render the approval artifact:

TARGET_REPO="$(pwd -P)"
RUN_ID="$(cat "$TARGET_REPO/.verify/current-run")"
RUN_DIR="$TARGET_REPO/.verify/runs/$RUN_ID"
VERIFY_PIPELINE="${VERIFY_PIPELINE:-$CLAUDE_PLUGIN_ROOT/pipeline}"
(cd "$VERIFY_PIPELINE" && npx --no-install tsx src/cli.ts criteria \
  --criteria "$RUN_DIR/criteria.json" \
  --spec "$(cat "$TARGET_REPO/.verify/.spec_path")") > "$RUN_DIR/criteria.md"

The rendered artifact shows each plain claim alongside expectIt and includes a Drive plans (what will actually run) section; its verbatim steps and designated proof line are part of what the user approves.

6. Seed script and second opinion, then stop

If any criterion needs volume or precondition data (thresholds, grouping, "with 6 users..."), write a literal $RUN_DIR/seed.sh now: real commands (curl, CLI) that create that data through the application's front door, taking the run marker as $1 and weaving it into every entity. No LLM runs at seed time; this script IS the seed, and the user reviews it. An entry that genuinely needs browser interaction is flagged "needs an agent — approve separately?".

Then get the second opinion — a reviewer that did not write these criteria:

VERIFY_SCRIPTS="${VERIFY_SCRIPTS:-$CLAUDE_PLUGIN_ROOT/scripts}"
VERIFY_ALLOW_DANGEROUS=1 bash "$VERIFY_SCRIPTS/review.sh"

It uses Codex when installed (a different vendor, different blind spots), else a fresh claude -p that sees only the spec and the criteria, and writes $RUN_DIR/review.json (keep/why/codify per criterion plus a missing list). Its output is advice; the user is the tiebreaker. If the reviewer is unavailable, say so verbatim: the criteria were reviewed only by the model that wrote them.

Also compute which relied-on parts have no probe, so the user sees it before approving:

jq -r --slurpfile s .verify/setup.json '
  [.criteria[].dependsOn[]?] | unique
  | map(select(IN("worker","sink","storage") and (($s[0].probes[.] // "") == "")))
  | if length > 0 then "Unprobed parts this run relies on: " + join(", ") else empty end' \
  "$RUN_DIR/criteria.json"

Print criteria.md, the seed script (verbatim), the reviewer's table and missing list, and the unprobed-parts line, including invented specifics and uncovered changed files. Ask the user to edit, correct, or say go.

The table carries each criterion's citation and reason so the user can check every row against the spec without opening it. The engine prints a NOT IN THE SPEC block for any quote the spec does not contain, and rejects a criterion that omits a quote or a why, or whose why is the title restated. A why that says nothing the title did not, in different words, is still a reason to send the criterion back.

Before approval, a plan may be inspected without counting as evidence. Dry-run the fully substituted plan, or exercise one original step into the isolated drafts/ folder:

VERIFY_PIPELINE="${VERIFY_PIPELINE:-$CLAUDE_PLUGIN_ROOT/pipeline}"
(cd "$VERIFY_PIPELINE" && npx --no-install tsx src/cli.ts drive AC1 \
  --repo-root "$TARGET_REPO" --run-dir "$RUN_DIR" \
  --dry-run --criteria "$RUN_DIR/criteria.json")
(cd "$VERIFY_PIPELINE" && npx --no-install tsx src/cli.ts drive AC1 \
  --repo-root "$TARGET_REPO" --run-dir "$RUN_DIR" \
  --draft --criteria "$RUN_DIR/criteria.json" --step 1)

Draft receipts never participate in reconciliation. If an exercise changes the plan, re-render the approval artifact so the user signs the exact steps that will run.

Stop here. Apart from the explicit isolated drafting helpers above, run no verification command and perform no system mutation until the user approves the criteria.

Half two: run and report

Enter this half only after explicit approval. Read the persisted run id; never rely on an earlier shell variable:

TARGET_REPO="$(pwd -P)"
RUN_ID="$(cat "$TARGET_REPO/.verify/current-run")"
RUN_DIR="$TARGET_REPO/.verify/runs/$RUN_ID"
test -f "$RUN_DIR/criteria.json" && test -f "$RUN_DIR/criteria.md"

If the user requested changes to the criteria instead of approving them, update and re-render half one, then stop again.

0. Boot, seed, pipeline check

Boot a throwaway environment and arm teardown before anything else can fail:

VERIFY_SCRIPTS="${VERIFY_SCRIPTS:-$CLAUDE_PLUGIN_ROOT/scripts}"
bash "$VERIFY_SCRIPTS/env.sh" up

From this point every exit path must run bash "$VERIFY_SCRIPTS/env.sh" down — run the rest of half two inside a subshell opening with trap 'bash "$VERIFY_SCRIPTS/env.sh" down' EXIT, or tear down explicitly on every failure branch. Boot and seed are separate verbs precisely so a seed failure cannot leak the stack. up writes .verify/run-env.json with this run's marker and rotates old runs (newest 5 kept).

Seed, in order:

bash "$VERIFY_SCRIPTS/env.sh" seed          # repo seed scripts + user data files

If the approved $RUN_DIR/seed.sh exists, run it with the marker — a failure aborts before judging, because judging against a half-seeded system is how wrong verdicts happen:

MARKER=$(jq -r '.marker' .verify/run-env.json)
VERIFY_SCRIPTS="${VERIFY_SCRIPTS:-$CLAUDE_PLUGIN_ROOT/scripts}"
VERIFY_BASE_URL="$(jq -r '.base_url // empty' .verify/setup.json | bash "$VERIFY_SCRIPTS/expand.sh" --load-env .verify/setup.json)" \
  bash "$RUN_DIR/seed.sh" "$MARKER" 2>&1 | tee "$RUN_DIR/seed.log"

Then prove the pipes work before judging anything:

bash "$VERIFY_SCRIPTS/precheck.sh"

Every part named by any criterion's dependsOn gets one probe (a marker round-trip for the database; any HTTP response for the API; configured commands for worker/sink/storage). A down part taints only its dependent criteria: do NOT drive those — record each as could-not-run with the pre-check's evidence ($RUN_DIR/prechecks/<part>.log). A part with no probe reports unknown and never taints; a FAIL on a criterion relying on an unknown part gets "the failure may be environmental: <part> was never health-checked" appended to its observation.