Deep Audit

Deep adversarial audit

A convergent alternative to slow round-by-round review. Instead of one reviewer finding one or two issues per pass, fan out many independent skeptics over the whole artifact at once, adjudicate what they find, fix everything confirmed, and re-verify. Modeled on the multi-agent methodology behind hard formal-proof efforts (diverse independent portfolio, adversarial throughout, concrete evidence only, synthesize-challenge-repeat).

When to reach for this

• The artifact is dense enough that a single review keeps surfacing new issues each pass (the tell that round-by-round is the wrong tool).
• Correctness is the priority and the cost of a missed defect is high (a paper going to a top venue, a proof, a security-sensitive change, a migration).
• The user asked to "fix ALL of it", "be deeper", "converge faster", "100% rock solid".

Requires the user to have opted into multi-agent orchestration (they asked for a workflow / deep audit / to fan out agents, or ultracode is on). If they haven't, propose it and its rough cost first.

The method

1. Decompose (diverse portfolio). Break the artifact into components by idea, not by section: each independent claim, lemma, estimator, subsystem, invariant. Add cross-cutting failure-mode lenses (see below). Aim for coverage such that every load-bearing claim is attacked by at least one agent that is looking straight at it. Don't tell the agents your favored reading — preserve independence so they don't all converge on the same attractive-but-wrong conclusion.

2. Fan out adversarial finders (one per component). Each finder is prompted to refute, defaulting to "there is a bug," and must ground every claim in the actual text/code (read it, don't paraphrase from memory). Hard rules, borrowed from what works:

• Concrete findings only. Every finding = exact location (file:line / label + quoted text) + one-sentence defect + a failing case (specific inputs/configuration → wrong output, or the exact missing hypothesis).
• Reject status reports, "looks fine", "this is standard/routine", vague optimism, and "the global step is straightforward."
• A fix that re-imposes the same difficulty elsewhere, or assumes its own conclusion, is not a fix — flag it.
• If, after genuinely attacking, nothing is found, the agent must state the specific attacks it ran and why each closed — not just "clean."

3. Adjudicate every finding (independent judge). A separate judge re-opens each cited location and decides CONFIRMED / REFUTED / DOWNGRADED, skeptical of both the artifact and the finding. This kills false positives (misreads, hypotheses that are actually present elsewhere, failing cases that don't arise under the stated conditions) — the step that keeps the fix list honest.

4. Synthesize. Dedup by location, rank fatal > major > minor, and hand back one clean defect list. Nothing is accepted as an issue until it survives this.

5. Fix all confirmed, then re-verify. Apply every confirmed fix (you, in the main loop — fixing needs care and judgment). Then re-audit the touched spots and check that no fix created a new defect. Repeat waves until two consecutive audit passes come back empty (fallback cap: 5 waves; a finding that survives waves N and N+2 goes to the user rather than a third patch). Don't stop after the first wave.

Failure-mode lenses (adapt to domain)

Beyond per-component attacks, sweep these cross-cutting modes explicitly — they are where real defects hide:

• Overclaim: the headline/abstract claims more than the theorems/tests actually deliver.
• Scope creep in a proof: a pointwise result used where a uniform one is needed; a both-correct property stated unqualified; a special-case argument invoked generally.
• Silent hypotheses: a differentiability/density/continuity/boundedness/positivity condition used but never stated (in math), or an un-checked precondition/invariant (in code).
• Edge cases: atoms/ties, endpoints/unbounded support, empty/degenerate inputs, boundary of the parameter space.
• Circularity: an assumption that assumes its own conclusion; a result that cites itself; a citation that gives less than claimed (read the cited source).
• Internal contradiction: a definition/notation used two ways; a table cell contradicting a proposition; main text vs appendix disagreement; a dangling/wrong cross-reference.

The fresh-eyes pass (final gate)

Every targeted wave inherits the blind spots of whoever wrote its prompts: focus hints, fix history, and expected failure modes all prime the auditors toward known territory. After all targeted waves and fixes are done, run one cold audit with little to no context: independent auditors given ONLY the artifact and a minimal instruction ("find concrete defects: location + failing case"), with no cluster assignments, no history, no special-focus lists. Diversify only the entry point (main-text-first as a journal referee would; appendix-first; tables/claims-first; a single deep dive of the auditor's own choosing). Adjudicate as usual. Clean fresh-eyes pass + clean targeted coverage + green mechanical battery is the closure standard; a fresh-eyes finding that targeted waves missed is also a diagnosis of the prompt set — add the missed failure mode to the lenses.

Full inventory — never sample

For a paper/proof artifact: enumerate every formal statement first (grep \begin{theorem|proposition|lemma|corollary} + labels) and assign each proof to a verifier — coverage must be 100% of load-bearing statements, not "a few proofs of the reviewer's choice." Sampling converges linearly and stochastically; inventories converge in one wave. Group tightly-coupled small lemmas into clusters; big proofs get their own verifier. Each verifier returns, besides findings, a steps-verified list and a hypotheses ledger (used-vs-stated; used-but-unstated is a finding).