Auditing Tests

Auditing Iris Tests

Three modes, one value bar. Authoring gates new or changed tests at write time. Audit investigates a few high-confidence candidates for consolidation or deletion. Campaign reviews one subsystem's whole test surface; read CAMPAIGN.md before starting one. Optimize for confidence, not deletion counts or coverage percentages.

Adapted from OpenClaw's test-audit skill. The upstream license is preserved in LICENSE.

Read root and scoped AGENTS.md files before working. This skill adds a test value gate, not a replacement for Iris's ownership rules or verification policy. Use the existing workflow-integration-tests, workflow-regression-tests, and running-compatibility-checks skills for their respective workflows.

Authoring gate

Before adding or changing a test, answer four questions:

1. What observable behavior, invariant, or independent contract does it protect?
2. What credible regression makes it fail or changes its observed report?
3. Why does existing coverage not already catch that failure? Each contract has one primary owner at the strongest useful boundary. Another layer needs a distinct risk, such as a local algorithm invariant versus its integration into compilation, or a process lifecycle failure compiler APIs cannot reach. Prefer extending a fixture or table over duplicating the same scenario.
4. Does it need an export, flag, wrapper, or injection hook no production caller needs? If so, test through the real owning boundary instead.

A missing answer means the test is not ready. Check it against the junk patterns below; a match fails the gate unless the retention bar identifies an independent contract. Behavior-preserving internal refactoring should not break a behavioral test. Explicit representation or output contracts may legitimately constrain refactoring; name those contracts instead of treating all snapshots as junk.

For a bug regression, demonstrate the undesirable behavior before the repair and the intended behavior afterward at the owning boundary. Follow workflow-regression-tests for Iris's failing-fixture/fix history. A deliberately accepted baseline snapshot may pass while recording a bug: the decisive evidence is the reviewed before/after report, not merely the runner's exit status. Runtime regressions must exercise fresh generated code and fail for the intended reason on the buggy compiler. Do not replay one bug at every layer it crosses.

Choose the Iris owner

Compiler fixtures enter through Main.purs. Their checking, diagnostics, semantic, and functional reports protect different observations; sharing one input does not make those reports redundant. Keep generated goldens limited to reachable fixture-owned modules. Use verify.mjs only when execution is the contract; it must test fresh output, never tracked goldens. Use real registry modules and the existing replacements.json mechanism for deliberate substitutes.

Junk patterns

• Assertion-free coverage probes, self-comparisons, and identity copiers.
• Expected values computed by the helper, renderer, or compiler under test.
• Copied inventories, manifests, export lists, or exact source/import greps without an independent contract.
• Private predicate or call-shape tests duplicated by stronger boundary proof.
• Multiple invocations of the same contract without distinct failure modes.
• Source-compilation unit tests duplicating the integration fixture pipeline.
• Handwritten library stand-ins where real prepared registry modules belong.
• Mocks that implement the behavior being asserted or stand in identically for APIs with different contracts.
• Fixtures that supply the ordering, persistence, or evidence the owner should produce, or assert a store the exercised path never writes.
• Capability tests that restate declared flags rather than exercising delivery.
• Tests that exist only to preserve test-only exports, globals, or wrappers; production code whose only callers are tests.
• Negative controls that pass because of an unrelated parser, resolver, checker, or environment failure before reaching the intended contract.
• Names promising more than the inputs and assertions exercise.
• Snapshot acceptance without checking types, diagnostics, locations, recovery, generated code, or runtime results against the intended semantics.

Value and retention bar

Keep tests that independently protect public APIs, language semantics, compiler representations with an intentional contract, LSP payloads, configuration, storage, security, platforms, defaults, generated code, packages, releases, or architecture. Also retain observable call ordering, credible regressions, and source inspection when it is the cheapest independent guard and survives an identifier-only refactor.

Static, slow, snapshot-based, or implementation-adjacent is not a deletion reason. A test that resembles implementation may still be its independent contract. Prove redundancy before removing it. Passing snapshots do not prove semantic correctness. A retained baseline failure may be a product defect; reproduce it instead of deleting the evidence or accepting it away.