Writing Defect Reports

Writing Defect Reports

A report is a claim, and it is read by people who will not re-derive it. The cost of an overstated one is not embarrassment — it is a maintainer spending an afternoon on a premise that does not hold, or a "fix" landing for a failure mode that never occurred. The techniques for finding a defect live in verifying-external-behavior (also in this repo's .claude/skills/). This skill is about the step between finding one and publishing it.

The rule underneath all of it: publish the claim you actually measured, at the layer you measured it.

The anomaly is real; the headline may not be

The common failure is not a fabricated bug. It is a genuine internal oddity promoted one layer too far.

You notice that a helper returns a degenerate value for a small sample, trace it into a scoring path, and write the report as "short inputs are wrongly flagged." Then you run real short inputs through the public entry point and the flag never sets — a downstream threshold absorbs the degenerate value, and the only visible effect is noise in a secondary per-feature list. The internal oddity is worth fixing; the headline was false, and a reviewer who tests it will say so.

Before writing the title, run the reproduction at the outermost layer the title names. Then choose one of three honest framings:

The second row is a good report, not a weak one. "This helper returns a value it should not; today a threshold happens to mask it, so there is no user-visible symptom yet" tells a maintainer exactly how to prioritize. Silently keeping the dramatic title because the underlying issue is real is what burns credibility.

Check the code is reachable before calling it broken

Three shapes look like defects and are not — or are, but not the one you were about to describe:

A branch that cannot execute. An early return guarding a lookup that already returns the same value on the missing case; a condition a preceding gate already implies. This is dead code, and "dead code" is the accurate report. Filing it as a correctness bug, or naming a test as though it exercises that branch, misleads everyone downstream — confirm liveness by deleting the line and watching the suite, not by reading it.

A guard whose pattern only matches your fixture. A validity check written against a hand-typed sample can be structurally unable to fire against the real input: a single-line pattern against a generator that wraps its output across indented lines, an exact-string check against a source that varies whitespace. Verify the guard against a captured real input, not the fixture. The consequence matters for the fix, too: replacing synthetic fixtures with real captures deletes that guard's only coverage, so the fix and the fixture change belong in one change, not two.

A file nothing invokes. Repos accumulate scripts that reference paths the repo does not contain and toolchains it does not depend on. Before reporting one as broken, find the caller — the task runner, the workflow, the entry point. If there is none, the report is "this is dead, delete it," which is cheap and uncontroversial, rather than "the build is broken," which is wrong.

Search the project's own records before filing a caveat

The most avoidable report is the one the project already answered. Two measured figures that look inconsistent are usually inconsistent definitions, and the definition is usually written down: a claim ledger row, a docstring, a design note, a closed issue.

Grep for the term before writing "these two numbers do not appear to compute the same quantity." If a record pins the definition, cite it — the caveat you were about to publish reads as an open question the project already closed, and a maintainer has to re-close it.

The same discipline applies to numbers you are quoting. Figures in an older issue may not reproduce, because a dependency the value depends on is unpinned and the installed version has changed. Re-measure before restating, and record it with the command, output, and date — see verifying-external-behavior for why an undated measurement cannot be re-checked, and cross-surface-changes for choosing between two records that cover the same fact.

Quote the protocol next to the figure, not only in whatever artifact produced it: the split, the warmup, the seed, the version. Two documents quoting the same metric under different protocols read as a regression to anyone comparing them, and the reader has no way to tell that they are not comparable.

Your earlier note is a claim, not evidence

Working notes, scratch findings, and a prior comment on the same issue are secondary sources — including your own. They were true about a tree that has since moved, or they were wrong when written.

When a note and the code disagree, the code settles it. Re-derive from the integration branch directly rather than from a checkout that may be behind:

git show origin/main:path/to/file.py | grep -n 'the_symbol'

If two notes contradict each other, do not average them and do not pick the more recent — re-check the tree and then correct the wrong note in place, saying that it was wrong. A knowledge base that records both readings without resolving them is worse than one that records neither, because the next reader will pick one at random.