Test Audit

Test Audit

Three modes, one value bar. Authoring mode gates every new or changed test at write time. Audit mode runs focused sweeps of tests that re-assert source, duplicate stronger proof, couple behavior to implementation, or keep test-only production seams alive. Continue broad audits as separate coherent follow-up PRs; optimize for confidence, not deletion count. Campaign mode prunes one whole subsystem's test surface (every test file a plugin, app, or package owns); before starting one, read CAMPAIGN.md.

Authoring gate

Before adding any test, answer four questions; a missing answer means do not add it yet:

1. What observable behavior, invariant, or independent contract does it protect?
2. What credible regression makes it fail?
3. Why does existing coverage not already catch that failure? Each contract has one primary test owner at the strongest boundary; another layer needs its own distinct risk, such as a transport or lifecycle failure the owner cannot reach. Prefer extending a table-driven case or shared fixture over a near-duplicate test; consolidate duplicated setup in the same change.
4. Does it need a production seam (export, flag, wrapper, injection hook) that no production caller needs? If yes, move the test to the real boundary instead.

Then check the test against every junk pattern; a match fails the gate unless the retention bar names the contract it independently guards. A test that would break under behavior-preserving refactoring is asserting implementation, not behavior; rewrite it at the owning boundary before landing it.

Bug regression tests must fail on the pre-fix code for the intended reason and pass after the owner-boundary repair. A regression test that never demonstrably failed proves the mock, not the fix. One regression at the owner boundary covers the bug; do not replay the same scenario at every layer it crosses.

bb test rules: never mock the database; use createConnection(":memory:") and migrate(db) from @bb/db. Vitest projects come from sharedWorkerProjects in vitest.shared.ts: Node test files share workers (isolate: false), and files that use vi.mock, vi.stubGlobal, fake timers, process.env writes, or global assignments are isolated automatically. Restore any other global state a test changes. Tests follow the no-comments rule.

Junk patterns

The shared checklist for both modes: the authoring gate rejects a new test that matches one, and audits hunt for existing tests that do.

• assertion-free coverage probes;
• self-comparisons and identity copiers;
• copied fixtures, inventories, manifests, or export lists;
• exact source, import, or string greps;
• private predicate or call-shape tests duplicated at real boundaries;
• duplicate invocations of the same contract;
• provider-local replays of shared helpers;
• tests whose only purpose is preserving test-only exports, globals, or wrappers;
• dead production code whose only callers are tests;
• expected values produced by the helper or renderer under test;
• mocks that implement the asserted behavior, or one identical mock standing in for different APIs;
• fakes that skip the real code they front, such as a hand-rolled bb stub or fake provider that never loads the real plugin entry, bridge, or extension;
• fixtures that supply the receipt, admission, or callback ordering the owner should produce, or persistence asserted against a store the path never writes;
• capability tests that restate declared flags instead of exercising the delivery or acknowledgement the flag promises;
• negative controls that pass for an unrelated reason, such as a denial from a different guard or a rejection the production path never reaches;
• names or fixtures that promise more than the input exercises, such as a "retires the window" test asserting the window was not cleared.

Value bar

Tests justify their maintenance cost by protecting behavior, a credible regression, or an independently meaningful contract. In an audit, an existing test that must change for behavior-preserving source reorganization is suspect, not automatically deletable; the authoring gate still rejects new ones.

Before judging a candidate, read the complete test and production owner, its entry point, callers, callees, sibling implementations, overlapping tests, CI routing (turbo.json task inputs and .github/workflows/ci.yml shards), and relevant history. Read AGENTS.md first. When the test claims dependency-backed behavior, inspect the dependency source or types directly.

Discovery

Keep discovery read-only and report evidence before editing. For broad scope, run parallel discovery lanes when available:

• app UI: apps/app (the largest suite), apps/web, apps/mobile, apps/desktop, packages/thread-view, packages/client-core;
• server and data: apps/server, packages/db, packages/domain;
• host and runtime: apps/host-daemon, packages/agent-runtime, packages/provider-bridge-*, packages/host-*;
• plugins and SDK: plugins/ (built-in and provider-* plugins), packages/plugin-sdk, packages/plugin-build;
• CLI, tooling, and end to end: apps/cli, packages/sdk, packages/scripts, packages/bb-app, tests/integration;
• a cross-cutting pattern sweep.

Outside campaign mode, prefer a few high-confidence candidates over a large speculative inventory. Hunt for the junk patterns.

Retention bar

Keep a test when it independently enforces a public API, Plugin SDK, protocol, config, migration, storage, security, platform, default, prompt-byte, generated cross-language, package, release, or architecture contract. In bb that includes server/daemon wire shapes behind HOST_DAEMON_PROTOCOL_VERSION (packages/host-daemon-contract/src/protocol.ts) and Plugin Guide surfaces and anchors (plugins/plugin-api-docs/src/surfaces.ts, anatomy-manifest.json). Also keep:

• call ordering when order is observable behavior;
• regressions with a credible failure mode;
• source inspection when it is the cheapest independent guard: it fails when the contract changes (the user-facing key, byte, or path) and survives an identifier-only refactor;
• a retained test that fails on the baseline: treat it as a possible product bug, reproduce it, and repair the owner rather than deleting it.

Static or slow is not a deletion reason. A test that resembles implementation may still be the independent contract; prove otherwise before removing it.

An unprefixed export of a published @get-bb/plugin-sdk subpath is public API, not a test-only seam, even with zero in-repo consumers. Removing one needs an entry under "Scheduled removals (next major)" in docs/api_to_audit.md.

Provider event shapes and timeline rows are owned by the recorded parity conformance (packages/provider-parity, recordings in packages/provider-bridge-protocol/recordings/) and the provider corpus gates (apps/server/test/provider-corpus/). A plugin test that replays a recorded case is a candidate; one covering a case recordings cannot capture is not.