Tutti Test Audit

Tutti Test Audit

Make every changed test earn its maintenance cost. A green test, a coverage increase, or a larger test count is not a quality verdict.

Read the root and closest scoped AGENTS.md, then read docs/conventions/unit-testing.md before a non-trivial test change or review. Follow docs/conventions/testing.md#validation-selection for commands and validation scope.

Establish the evidence map

Before editing, inspect:

1. the production owner, public entry point, relevant caller and callee;
2. sibling implementations that share the same invariant;
3. existing tests, fixtures, conformance suites, and repository checks;
4. the changed-aware and platform lane that will select the evidence;
5. relevant history for a regression or a suspicious existing test.

For Agent Host lifecycle semantics, also read packages/agent/host/README.md and start with the conformance contract. Do not reimplement lifecycle tests in an adapter.

Pass the authoring gate

Answer all six questions before adding or materially rewriting a test:

1. Protected contract: What observable behavior, invariant, compatibility promise, or prior failure matters?
2. Credible failure: What plausible faulty implementation must make the test fail?
3. Coverage gap: Why would existing evidence not catch that failure? Can an existing table, scenario, or fixture express it without duplication?
4. Owner and observer: Which module owns the decision, and what is the lowest boundary that can observe the risk without mocking it away?
5. Production seam: Does the test require an export, flag, wrapper, global, or injection hook that no production caller needs? If so, move to the real boundary or redesign the owner.
6. Execution: Which blocking lane selects the test, and which native OS must execute it?

A missing answer means the test is not ready. “No new test” is a valid outcome when stronger existing evidence already protects the contract or the proposed assertion only restates a type, literal, implementation detail, or presentation choice.

Choose the proof shape

• Extend an existing table or conformance scenario when it owns the same contract and fixture.
• Add a new scenario when the behavior, failure mechanism, setup, or execution lane is independently meaningful.
• Move upward to a component, narrow integration, conformance, or E2E boundary when a unit double would replace the semantics under test.
• Use a repository check, not a unit test, when source structure, imports, generated artifacts, manifests, or architecture are the actual contract.
• Keep visual styling in visual evidence unless styling changes native behavior such as pointer ownership or window dragging.

Prefer one vertical proof through the owning boundary plus narrow tests for important decision partitions over many stub-backed branch tests.

Build sensitive evidence

For a confirmed bug, capture the failing reproduction before the fix whenever possible. The regression test must fail on pre-fix behavior for the intended reason and pass after the owner-boundary repair. If the old revision cannot be run safely, temporarily inject or restore the faulty decision as a local negative control and report that limitation; never commit the mutation.

For new behavior, first demonstrate that the focused test fails because the contract is absent, not because setup, imports, or fixtures are broken.

For a read-only audit, do not edit production code merely to manufacture a negative control. Name the smallest plausible faulty implementation, determine whether the current assertions would detect it, and label that sensitivity unverified when it cannot be executed safely.

Then:

• assert stable outcomes and important non-effects;
• include neighboring valid, rejection, cleanup, retry, or preservation cases only when they address a credible risk;
• use real isolated SQLite, filesystem, Git, HTTP, IPC, parser, or process boundaries when their semantics are the subject;
• keep external doubles narrow and fail closed on unexpected calls;
• coordinate concurrency with events, channels, barriers, or latches rather than sleeps or lucky elapsed-time bounds;
• control clocks, IDs, randomness, ports, user state, credentials, globals, listeners, timers, processes, and temporary resources;
• assert call counts only when count is itself a contract such as idempotency, single-flight, or at-most-once dispatch.