Verify

Verify before claiming

Every high-severity finding this repo has produced came from executing something. Every wrong claim came from reading something and reasoning about it.

The rules below are cheap. Skipping them is what produces a confident, wrong answer that a maintainer then acts on.

Rule 1 - execute the pattern, never read it

A regex, a redaction filter, a permission mask or a capability gate is code. Run it against a realistic value and look at the output.

What happens when you don't. utils/logging.py redacts key-value shapes. Reading the pattern suggested "Feed Token: {t}" leaked, so nine call sites were reported as leaks. Executing it showed the alternation contains a bare token, which matches Token: and redacts the value. The finding was wrong in the direction that wastes a maintainer's time.

The same run showed "Access Token obtained: {t}" genuinely leaks, because a word sits between the keyword and the colon. Neither result was predictable by inspection.

uv run python .claude/skills/verify/redaction_check.py \
  "Access Token obtained: {t}" "eyJhbGciOiJIUzI1NiJ9.SEKRET.sig"

Prints LEAKS or redact and exits non-zero when the secret survives, so it drops into a loop or a test.

Corollary: whether a log line leaks depends on data you have not read yet. ?susertoken= and ?token= redact; ?Value1= and ?jKey= do not. You cannot judge a logged URL without opening the code that builds it and learning the real parameter name.

Rule 2 - break the code to validate the test

A test that passes proves nothing until you have seen it fail. Revert the fix, or neuter the guard it depends on, and confirm the test goes red. Then restore.

What happens when you don't. A StrategyBuilder test asserted a tile was absent after an identity reset. It passed. It also passed against the exact bug it was written to catch, because the assertion raced a 400ms debounce that legitimately re-rendered the tile.

Two ways this goes wrong, both seen here:

• Tautological assertion. The test re-implements the predicate inline (is_stale = qty != 0 and updated_at < boundary) instead of calling the function under test. It is then testing Python's < operator.
• Vacuous pass. The assertion is true for a reason unrelated to the fix, so it holds whether or not the fix is present.

When disabling a guard to prove a test, target the exact line. A blind replace(..., 1) hits the first match, which may be a different, pre-existing guard, and then the "proof" proves nothing.

Rule 3 - grep for the sink, not the variable name

A grep that returns nothing is not evidence of absence. It is evidence about your pattern.

What happens when you don't. Searching broker/ for credential variable names inside f-strings found and fixed the direct cases. It was structurally incapable of seeing the larger class, where the secret rides inside something else that gets logged:

• a URL assembled with the token in the query string
• an auth request or response body
• a headers dict
• an exception message (httpx.HTTPStatusError embeds the full URL, so a credential in the URL path leaked on every 4xx, exactly the wrong-credential case)

Before concluding a class is clear, ask what a leak would look like if the secret were never named in the log statement, then search for that.