Math Auditor

Math Audit

Adversarial review of a scientific Julia package for mathematical accuracy and consistency — not software architecture (flag architecture only when it causes mathematical wrongness, e.g. mutation aliasing, accidental type demotion, or inconsistent conventions between modules).

The output is written for the package's own developers: findings must cite exact file:line, state the correct mathematics, and give a concrete failure scenario. A finding that can't survive an attempt at refutation doesn't ship.

What "adversarial" means here

Each reviewer's job is to break the code, not describe it. Concretely, hunt for:

• Wrong equations: update formulas, gradients, covariances, likelihoods that differ from the cited papers or from the docstring's own LaTeX. Derive the correct expression independently and diff it against the code.
• Convention drift: rows-vs-columns for ensemble members, N-1 vs N normalization, factor-of-2 / sign errors, Cholesky L vs U, covariance vs precision, whether noise is added in obs-space or transformed space — especially inconsistencies between modules that must agree.
• Statistical validity: is added noise sampled with the right covariance and scaling (e.g. Δt scaling in stochastic dynamics)? Are means/covariances computed over the right dimension? Deterministic vs stochastic variants actually equivalent in expectation?
• Numerical soundness: unguarded inv/\ on possibly-singular matrices, loss of symmetry/PSD-ness, subtraction-based variance formulas, missing regularization, sqrt of negative-by-roundoff eigenvalues.
• Edge cases the math must survive: ensemble size 1–2, dimension 1 (scalar vs matrix degeneracy), zero variance, NaN/failed ensemble members, empty minibatches.
• Test-math consistency: do the tests actually pin the mathematics (analytic solutions, invariants, convergence rates), or just check shapes and "it runs"? A wrong equation whose test only checks size() is a double finding: the bug and the missing test.

Workflow

1. Partition

List src/*.jl and test/**/*.jl with line counts. Group into 4–8 review units of roughly comparable size, pairing each source module with the tests that exercise it. Group modules that share mathematical conventions together (e.g. all Kalman-update variants in one or two units) so the reviewer can catch cross-module inconsistencies.

2. Fan out reviewers (parallel agents)

Spawn one agent per unit, in a single message so they run concurrently. Each agent prompt must include:

• the exact file list for its unit,
• the "What adversarial means here" hunting list above (copy it in — agents don't see this skill),
• instructions to check code against docstrings/comments and against the standard form of the algorithm from the literature,
• a required output format: a JSON-like list of findings, each with file, line, severity (critical / major / minor / hygiene), claim (one sentence), evidence (the code vs the correct math), failure_scenario (concrete inputs → wrong output), verified (numerical / inspection), and suggested_fix (optional, a few lines).

Tell agents explicitly:

• "Prefer few, well-evidenced findings over many speculative ones — but do report genuine minor inconsistencies. If the module's math is correct, say so and note the strongest invariants the tests pin."
• "When a finding concerns a fixed point, a statistical scaling, or a crash, verify it numerically in a scratch script if cheap — a small linear-Gaussian fixed-point run, a quick Monte Carlo of the statistic, or reproducing the error — and tag it verified: numerical. Numerically verified findings are worth far more than inspection-only ones." (In one audit, the agents that ran code delivered the critical finding pre-verified to 1e-16; the only refuted claim of the run came from an inspection-only unit.)
• "Before claiming anything is 'silent' or 'has no warning/guard', grep for @warn, @error, and throw at the constructors and call sites of the code path, not just the function you are reading — guards often live at construction time."

As each agent's report arrives, save its raw findings verbatim to a scratchpad file (one per unit). A full audit plus verification is long enough that context summarization mid-run can silently lose findings; the scratchpad files are the durable record the report is assembled from.