Test Author

Test Authoring Framework

This skill defines how test authorship is separated from implementation. It exists because an agent that writes both the code and its tests has no adversary in the loop — the tests confirm what was built, not what was intended. The trend record backing this trait names the recurring cost directly: vacuous positive assertions with || isEmpty() escapes, a test oracle computed from the implementation's own formula, assumeTrue wrapped around three real production bugs so they never turned the suite red, and coverage claimed in notes that didn't exist in the test tree. review-quality already names the bias — "the agent that wrote the tests has an inherent bias toward believing they're correct" — this skill is the structural fix upstream of review: an independent authorship step, gated at the point where an oracle can still be frozen before implementation exists to copy from.

Everything below applies whether the test author is a separate dispatch or, in a degraded mode, the same agent operating at a different, declared point in time. The separation is the point — follow it exactly even when it feels redundant with work you can already see.

When query_rules(get, rootId, key:"test-author") succeeds, the served text is authoritative for §3–§10; this skill is the fallback and the Claude-specific addendum.

---

1. When This Applies — The Two Seats

The trait needs-test-author puts two seats on an item, occupied at different phases:

• The plan author, at queue phase, fills test-plan — scenarios, oracle sources, and probes, frozen before any implementation exists. This is the structural lever: an oracle written before code exists cannot be derived from that code, no matter who writes it later.
• The test author, at work phase, fills test-manifest — the actual test files, written after implementation has landed so they compile against the real signatures, but without reading the implementation's reasoning or its own tests.

Same-agent exception (Direct tier, temporal-only): for a single-item Direct-tier dispatch there is no second agent to send the work to. The same agent may occupy both seats, but only under a temporal-only degraded mode — see §11. The gate still applies: test-plan must exist and be frozen before implementation begins, and the manifest must declare the single-actor mode explicitly rather than silently reusing the plan author's context. This is a narrower guarantee than true two-agent separation, and test-independence-audit records it as such (independent-degraded), never as independent.

Outside Direct tier — Delegated and Parallel — the two seats are two dispatches. Never collapse them to save a round-trip; the whole value of this trait is in the second agent's blindness.

---

2. Deriving Numbered Scenarios From Acceptance Criteria

Start from the acceptance criteria in the item's planning note (task-scope, feature-summary, or diagnosis) or the queue-phase spec these criteria live in. Every criterion maps to at least one scenario; a criterion with zero scenarios is a coverage gap, not an implicit pass.

Partition each criterion into happy / failure / edge, mirroring the spec-quality Test Strategy discipline this note composes with — test-plan gives each scenario an oracle and a probe list; task-scope's Test Strategy section should already have named the scenarios themselves, so this step should feel like formalizing, not inventing from scratch. If it feels like inventing from scratch, the queue-phase spec's test strategy was too thin — flag it rather than filling the gap silently.

• Happy — the criterion's primary intended behavior under normal input.
• Failure — what happens when the criterion's preconditions are violated (invalid input, missing dependency, disallowed state transition).
• Edge — boundary values and structural extremes: empty vs. absent vs. null, exact limit values, duplicate entries, ordering sensitivity, maximum depth or size.

A criterion that only yields a happy-path scenario has not been fully partitioned — go back and ask what could violate it or sit at its boundary before treating it as covered.

Stable S-ids. Number scenarios S1, S2, S3… in the test-plan note and never renumber once written test code refers to them — the test-manifest's S-id→test mapping is only auditable if the ids are stable across the plan/manifest boundary. If a scenario is dropped, mark it S4 — dropped: <reason> rather than closing the numbering gap.

Label every scenario EXISTING-SURFACE or NEW-SURFACE. Red-proof is obtained by reverting the fix and running the item's tests. That works only where the scenario binds to a surface that already existed: when the fix introduces the type, parameter, seam or enum constant a test references, reverting it produces a compile failure rather than a behavioral red. Compile-red proves the tests reference new code; it does not prove they detect wrong behavior. Across two bug waves this hit 3 of 10 items and then 2 of 5 — and in the second wave the two affected items were the security fix and the silent-exit fix, the two whose defects mattered most.

• EXISTING-SURFACE — every declaration the scenario touches exists before the fix. A plain revert yields behavioral red; no extra field is needed.
• NEW-SURFACE — the scenario binds to a declaration the fix introduces. It MUST carry a narrowest-revert recipe: the smallest revert that still compiles and still exercises the behavior. Typically keep the new type, parameter or enum constant; revert only its call sites — the technique that recovered genuine behavioral red on 2 of 3 such items in bug wave 1.
• NEW-SURFACE with no revert that can yield behavioral red — say so explicitly and name the substitute verification that replaces it (e.g. "reviewer reads each test body against the implementation and confirms every asserted value traces to its oracle citation"). A substitute declared in the plan is evidence; one produced at verification time is an excuse.

The plan author assigns these labels, not the test author: the test author is blind to the implementation by design (§4) and therefore cannot tell which surfaces are new. Where a dispatch contract's planning seat returns a red-proof-shape field, it carries this same labelling, set before the author is dispatched. The shape actually obtained is recorded in test-manifest (§10).

---

3. Oracle-Derivation Rule

Every scenario's expected result must have a stated oracle source: the spec clause, the documented algorithm, an external reference (an RFC, a library's own documented contract), or an independently-run computation. The oracle is never "what the code returns" and never a value lifted uninspected from the ticket's worked example.

Never encode an illustration as the oracle. A worked example in a spec or ticket exists to build intuition, not to be trusted as ground truth — illustrative examples are frequently approximate, off-by-one, or simplified for readability. If a scenario's only available expected value is an illustration, recompute it independently from the stated rule before writing the assertion, and cite the rule, not the illustration, as the oracle source.

Never read the implementation to decide correctness. This is the rule the whole trait exists to enforce. If you find yourself opening the file under test to see what it currently returns and then asserting that value, stop — you have just converted the test into a change-detector for whatever the implementation happens to do today, including its bugs. This is the exact failure pattern in the trend record: an oracle computed from the implementation's own formula reproduces the formula, not the requirement, so it stays green even when the formula is wrong.

Concretely, an oracle citation looks like:

• "per task-scope §Test Strategy S3: duplicate dependency edges are rejected with DUPLICATE_EDGE" — a spec clause.
• "per RFC 7396 §2: a merge-patch null removes the key" — an external reference.
• "computed independently: SHA-256 of the canonical byte sequence, verified against a second implementation (Python hashlib) run outside the codebase" — an independent computation.

An oracle citation that instead reads "matches current behavior" or "see implementation" is not an oracle — it is a confession that this rule was skipped, and should block the note from being accepted as complete.

A NEW-SURFACE label (§2) does not relax any of this. When a scenario binds to a declaration the fix introduces, the pull toward sourcing the expected value from the new code is strongest — the surface exists nowhere else yet. It still does not qualify as an oracle. Derive the expected result from the spec clause that motivated the new surface, or escalate per §8. A new surface with no oracle available outside the implementation is a spec gap to raise, not a licence to read.

---

4. Blindness Rule — A Capability Boundary

The test author's independence is only real if its inputs are actually restricted. This section is the enforceable half of that — the queue-phase oracle freeze (§3) is the other half.

Why this is not a reading-discipline rule. An earlier version of this section told the author how to read implementation sources narrowly: look up the declaration, stop before the body. That model cannot hold, because no reading tool has a declarations-only mode. Read returns a window, Grep -A<n> returns context lines, sed -n <a>,<b>p returns a range — every one of them will put a function body in front of an author who is trying, in good faith, to resolve a signature. Restraint fails at the tool boundary, so the boundary moves.

The evidence. Bug wave 3 (2026-09), item a3ebd108, consumed THREE test authors:

1. The first called query_notes(operation="list") with no keys= filter, received implementation-notes in the response, and read the implementation files it named — an ingress no reading-method rule had contemplated.
2. The second, working under the previous wave's "use Grep with -A2, never Read" remedy, widened to sed -n <a>,<b>p ranges "to catch wrapped signatures" and read the sentinel function's body.
3. The third was handed every public declaration inline in its dispatch prompt and was barred from src/main by any tool. Zero lookups; 12/12 scenarios plus 7 probes; reviewer verdict independent.

Authors 1 and 2 both self-disclosed and stopped before committing, so no contaminated artifact was ever tracked — the self-disclosure protocol worked, twice. What failed was the reading model, at a price of two burned dispatches. The rules below are the structural replacement.

They are capability rules, not care rules. A breach is a breach whether or not anything useful was seen, and it is disclosed the same way either way.

4.1 Declarations are supplied, not looked up

The dispatch prompt — or the Test author protocol slot of the wave's dispatch contract — MUST paste inline and verbatim every public declaration the author needs: types and data-class constructors with full parameter lists and defaults, function and method signatures, constants, enum values, and any KDoc that carries an oracle or states an invariant (validate() included, per §7 Fixture invariants). The author writes tests against that block and goes looking for nothing further. That includes the block's harness: and runtime call order: lines: build the application under test only as they state, per the dispatch contract's DECLARATIONS block and its rule 4 (a NONE harness the scenarios need is stop-and-ask, §4.4).

A declarations block that is missing entirely is an orchestrator error. Say so and stop (§4.4); do not reconstruct it.

4.2 Hard tool ban on implementation sources

Do not open any file under src/main with ANY tool — Read, Grep, sed, cat, head, Glob preview, an editor, or any shell command whose output includes file content. The ban attaches to the file, not to the intent: "I only wanted the signature" does not make the call permitted, because the tool decides what comes back, not you.

Banned for the same reason: git diff, git show, git log -p and any other view of the implementer's changes on this branch.

src/test stays fully readable. Existing tests, fixtures and harnesses are how you match the codebase's conventions — reading them is expected, not merely tolerated.

4.3 keys= is mandatory on every query_notes call

Every query_notes call carries an explicit keys= filter restricted to the item's queue-phase keys — typically ["task-scope", "diagnosis", "test-plan"]. An unfiltered operation="list" returns implementation-notes and session-tracking in the same response and is itself a breach, whether or not their bodies were read. includeBody=true with a keys= filter is fine; includeBody=true without one is the exact call that burned wave 3's first author.

The same holds for query_notes(operation="search"): scope it to the item, and never search for terms that would rank implementation prose highly.

4.4 A missing declaration means stop and ask

If a declaration you need is absent from the supplied block — or what was supplied does not compile against the test as planned, because a parameter was renamed, a return type reshaped, or a method the plan assumed exists does not — do not derive the corrected shape from context, from a compiler error that quotes surrounding source, or from the diff.

Send the question back: SendMessage to the orchestrator (Parallel/Delegated tier), or ask the user (Direct tier), naming the exact declaration you need. Asking costs one round-trip; the lookup costs the dispatch. This is the escalation path of §8, and a plan-vs-implementation drift found this way is a finding worth recording, not an inconvenience to route around.

4.5 What the author may read

• Queue-phase specification notes (task-scope, feature-summary, diagnosis) and test-plan, fetched with a keys= filter per §4.3.
• The declarations block supplied by the dispatch (§4.1).
• Project documentation (current/docs/, CLAUDE.md) and any external reference cited as an oracle.
• Everything under src/test — conventions, fixtures, harnesses, existing assertions.
• The implementer's changed-file names (git diff --name-only, or the File ownership rows of the dispatch contract) — enough to know where tests belong, not what the files contain.

Never: any file under src/main; any diff, commit or patch content; the implementer's own tests where the dispatch identifies them as such; implementation-notes; session-tracking.

4.6 Self-disclosure on breach — unchanged

If you breach any rule above, deliberately or by accident: stop immediately, commit nothing, and report exactly what was read and when — in your return message and in test-manifest's arbitration record. Delete any draft written after the breach and say that you did.

This protocol is the part of the old model that worked, three times across two waves, and it is unchanged. A disclosed breach costs a re-dispatch. An undisclosed one costs the trait: every later independent verdict on the item becomes unverifiable.

---

5. Red-First Check

Where red is achievable before the fix exists, the test must actually observe it.

Which scenarios "red is achievable" covers is decided by the §2 surface labels, not here. An EXISTING-SURFACE scenario can reach behavioural red by a plain revert. A NEW-SURFACE one cannot — reverting the fix removes the declaration the test binds to, so the run is compile-red, which proves only that the test references new code. For those, red-first means executing the plan's narrowest-revert recipe (keep the new type or parameter, revert its call sites) or, where the plan declared no revert can work, the substitute verification it named instead. Read §2's label definitions before deciding a scenario's red evidence; the shape actually obtained is a test-manifest field (§10).

Bug-fix regression tests: write the test from the diagnosis note's reproduction steps and confirm it fails against the pre-fix code — actually run it red, don't assume the reproduction description implies a failing assertion. A regression test that was never seen red proves nothing about whether it would have caught the bug; this is exactly how assumeTrue wrapping neutralizes a would-be regression test without anyone noticing, because the wrapped test never fails at all, pre-fix or post-fix.

Feature suites (test-after): since implementation already exists by the time the test author writes code, true red-before-fix isn't available. Substitute a mandatory per-scenario line in the manifest: "what specific wrong behavior would this assertion catch?" — name a plausible bug this test would fail against (wrong value, wrong exception, silently-accepted invalid input). If you cannot state one, the assertion is not adding coverage; strengthen it or mark the scenario not-covered: <reason> rather than writing an assertion that would pass against almost anything. "Nothing specific" for this line is a blocking review finding under review-quality.

---

6. Adversarial Probe Catalog

Beyond the scenarios derived from acceptance criteria, run the applicable probes below against the feature's actual input surface. Record every probe attempted in test-manifest — including the ones that found nothing. A probe list with only findings looks like it was written after the fact to justify existing tests; a probe list that also records clean results is evidence the surface was actually exercised.

• Boundary / suffix — values at, one below, and one above a stated limit; strings that are prefixes or suffixes of a recognized token rather than exact matches.
• Alternate separators — the same logical path or identifier expressed with a different delimiter, casing, or encoding than the primary code path expects (e.g. \ vs /, ..%2f).
• Encoded / UNC forms — percent-encoded, double-encoded, or UNC-style (\\?\, \\host\share) variants of path or identifier input, where the surface accepts path-like or URI-like input.
• Mixed case — case variants of identifiers or keys where the underlying store or comparison may be case-sensitive in one layer and not another.
• Empty vs. absent vs. null — three distinct states that are easy to collapse into one code path by accident; each deserves its own scenario if the surface can distinguish them.
• Duplicates / ordering — repeated entries in a collection input, and whether processing order is assumed but not guaranteed.
• Replay / idempotency — repeating the same write or transition twice; confirm the documented idempotency contract (or lack of one) holds.

Not every probe applies to every surface — a pure in-memory computation has no path-encoding surface. Record the ones that don't apply as N/A: <reason> rather than omitting them silently, so a reviewer can tell "not applicable" apart from "forgotten."

---