Gating

gating

A gate is a check that blocks. Its only job is to go red when it should.

The characteristic failure is not a wrong check — a wrong check gets noticed. It is a check that cannot fail, which reports PASS forever and is indistinguishable from a working one from the outside. That is what makes this different from ordinary testing: the object under suspicion is the check.

When NOT to use this skill

Scope is ONE check and whether it can be made to fail.

A gate is a thing that goes red. If nothing here can go red, there is no gate to audit.

The three obligations

Every gate owes these. A gate missing any of them is not yet a gate.

1. An anchor outside your own code. Something the check compares against that your implementation did not produce: a published constant, a closed-form answer, a conservation law, a degenerate case with a known result, an independent implementation. A check that compares this run to the last run only ever tells you the code still does what it did. See references/anchors.md.

2. A known-bad it demonstrably rejects. Break the subject the way it would plausibly break, run the gate, confirm red. Until you have done this you have not shown the gate works — you have shown it runs. This is the obligation people skip, because a passing gate feels like evidence.

Two things about known-bads that are easy to get wrong:

• Validate it at the configuration it will run in. A case tuned on a small or fast setting can stop being bad at full size. An "untrained" grid built from one Lloyd iteration was genuinely zero-gain at m=2/K=16 and earned a real +0.10 dB at m=8/K=65536, where one iteration relocates ~63,000 empty cells toward the mode. It passed the fast gate and certified nothing about the real one. This matters more than it sounds, because mutate.py needs a fast gate variant and it is tempting to validate everything there.
• Measure its reach. One known-bad is the floor, not the goal. Name which checks it exercises (known_bad(..., covers=(...))); the harness prints the checks no known-bad reaches. An audited gate had a single known-bad covering 1 of 8 checks — and the check its whole result rested on accepted the same bad case.

3. A written statement of what it cannot catch. Coverage holes are invisible from inside a green run: the gate is silent about the thing it does not look at, in exactly the same tone it uses for the thing it looked at and approved. The author has to assert the hole; nothing else will.

scripts/gate.py enforces obligations 2 and 3 mechanically — it returns exit code 2 (INCONCLUSIVE, not PASS) when a gate registers no known-bad or no coverage limit.

Building a gate

Work in this order. The first step is the one that determines whether the rest is worth anything.

Name the wrong conclusion, not the component. Not "check the quantizer is correct" but "prevent shipping scalar wins at high bit rates when that would really be an optimizer artifact." A gate aimed at a conclusion knows what counts as a near-miss; a gate aimed at a component just exercises the code.

Find an anchor. references/anchors.md lists the kinds, in rough order of strength, with the questions that find each one.

Prefer brackets to point checks. Assert a value lies strictly between two things it cannot legitimately pass: better than a baseline, worse than a theoretical bound. A one-sided check passes for a result that collapsed as readily as for one that is right — which is how an implementation that silently does nothing gets certified.

Derive the tolerance from measured noise. Run the thing several times, see how much it moves, put the threshold outside that. A tolerance picked for comfort tends to land wider than the defect you are trying to catch, and then it swallows it.

Then check it is not too tight to mean anything. The opposite failure is real and less obvious: a margin can be statistically impeccable and practically empty. A paired estimator — scoring both arms on one shared sample so the common fluctuation cancels — is the right way to measure a difference, and its standard error shrinks as the two arms converge. So "beats the baseline by 3 se" degenerates: a codebook perturbed by N(0, 1e-3) gained +0.0001 dB against a 3-se margin of 1.2e-06 and was accepted, while real ones gained 0.35–1.41 dB. The check certified the effect is real, not the effect is worth having. Those are different assertions and need different thresholds — and the second one has to come from an anchor (there, a published lattice codebook), never from the estimator, which knows nothing about what magnitude would matter.

Build the known-bad and confirm red. Then run scripts/mutate.py for the failures you did not think of.

Wire it to a non-zero exit and run it before the thing it gates, not after. A gate that runs after the results are written is a report.

Auditing an existing check suite

Given tests, a linter config, a CI job, or a gate someone already wrote, the question is not "do these pass" but "can these fail". Full procedure in references/auditing.md; the fast version:

• For each assertion, name a concrete input that makes it fail. If you cannot, it is decoration — delete it or fix it.
• Check each oracle's range against the range you actually operate in. A published table that stops short of your regime is a hole with a green light on it.
• Run scripts/mutate.py against the code the suite covers. Every survivor is a behaviour nothing checks.
• Look for assertions whose truth does not depend on the subject at all.
• Look for tolerances wider than the effect being measured.

Scripts

# harness: refuses to report PASS without a known-bad and a coverage limit
python3 scripts/gate.py           # importable; see the module docstring

# mutation pass: which single-token changes does the gate NOT notice?
python3 scripts/mutate.py --target src/codec.py -- python3 calibrate.py
python3 scripts/mutate.py --target grids.py --max 40 -- pytest -q

mutate.py requires the gate to pass on unmutated code first and refuses to run otherwise, because survivor counts against an already-red gate mean nothing. It restores the file even on interrupt, and uses tokenize so string literals and comments are never corrupted. It is the zero-dependency pass that works against any gate command; once it stops finding survivors, mutmut or cosmic-ray go deeper on Python test suites specifically.