Test The Tests

Test the Tests

Passing and high coverage are not evidence, whoever wrote the test — including you, just now. A test proves it runs; it does not prove it discriminates correct from incorrect behavior. Ask:

What fault would this test fail to catch?

The mutation loop

For each test that carries real weight (guards a fix, a regression, an invariant — not a trivial getter):

1. Pick the smallest plausible fault: invert a condition, change < to <=, remove a validation, suppress an error, skip a side effect, hardcode a return value, revert to the prior (buggy) behavior.
2. Apply it to the implementation, run the test, confirm it fails for the right reason — not an unrelated crash.
3. Revert immediately, confirm the tree is clean (git status/git diff), confirm the suite passes again. NEVER leave mutated code in the repository between steps.

Copy the file before mutating it (cp f f.bak) and restore from the copy. NEVER restore with git checkout/git restore while the work under test is still uncommitted — those discard the work along with the mutation. Commit first, or use the copy.

If running the mutation isn't practical, name the smallest mutation that should be tried instead of skipping the exercise — that name is itself the finding.

"Not practical" is a conclusion, not an assumption — establish it as AGENTS.md requires (is the runtime actually missing, is a Linux host available) before falling back to naming the mutation. A "can't run it here" that turns out to be wrong ships tests nobody has ever seen fail.

Common ways a test looks strong but isn't

• The assertion holds under the bug too. A chain assertion like index(a) < index(b) < index(c) can pass under both the fix and the regression it's meant to catch if the scenario doesn't force them to disagree. Reshape the scenario (add a case whose correct answer differs from what the bug would produce) instead of trusting a single run.
• It re-asserts a second recording of the same event instead of a property true by construction — comparing two independently-recorded orderings is racier and weaker than asserting a monotonic counter or a guaranteed dependency order.
• It exercises the unit in isolation, never the real wiring a regression would actually break. Prefer driving the real entry point end-to-end when the risk is in the wiring.
• It asserts implementation trivia (mock called N times, internal helper ran) instead of externally observable behavior.
• Coverage number, not falsifiability. A line being executed says nothing about whether a wrong value on that line would be caught.
• It proves the lab, not production wiring. A probe on a bare context "proved" that cancellation panics past a branch — but production wraps commands in graceful.WithTermination, where Terminate returns normally and the branch executes. Before concluding "X cannot reach this code", rebuild the exact wiring production uses (context construction, env, entry point); a conclusion drawn from simplified wiring describes the probe, not the system.
• An extended test quietly drops what the old one proved. Adding cases to a fixture can remove the conflict that made an earlier property observable. After editing an existing test, re-run the mutations the previous version caught, not only the new ones.
• Ordered behavior tested without conflicts. For precedence lists and fallback chains, a fixture with one candidate per level survives swapping adjacent candidates. Give every level two candidates whose effects differ.