Audit Choices

Audit Choices

Given a good decision, an agent implements it faithfully; wherever the task is underspecified, it makes the decision itself — silently, and the diff won't flag it. Reviewing thousands of changed lines doesn't scale, and it inspects the execution, which was probably fine. The audit that scales is of the choices: surface every decision the implementer made on its own, judge that list, and record the verdicts.

This is about architecture more than bugs. An implementation can work perfectly and still rest on decisions the user never made — a data shape, a storage location, a dependency, an API contract, a tradeoff of memory for speed — and every one of them is load-bearing for future work. The user needs to know them not because they're wrong, but because they now own them.

This is purely a decision audit — it is not about modifying code, and it can be called at any time. The job is to trace back: walk every step this session has taken, and every step each subagent took (a live implementer traces its own; otherwise reconstruct from its reports, transcripts, and diffs), and surface every single decision that was made on the user's behalf that was not in the original spec or prompt. The ledger of those decisions replaces reading the code as the user's review surface — that is the whole point. Acting on the verdicts (redoing an unsound choice, applying a provisional call) belongs to the caller: the implementing workflow mid-run, or the user after reading the report.

Two ways in, same audit:

• Called by a workflow (per pass or per slice): audit that pass, append its entries to the ledger, and return; the workflow presents the accumulated ledger when it hands back.
• Called directly by the user: audit the whole body of work in front of you (session, branch, or named change) and present the report immediately. Recommend; change nothing.

Workflow

1. Elicit and trace back. When an implementer reports done, ask: "While working on this, which choices did you make that you're not confident of? List all." — but treat the self-report as a starting point, not the boundary: agents under-report. Trace the history yourself — the session's steps, subagent reports, diffs, commits — and collect every decision that is in the work but not in the original spec or prompt. Sweep the architectural categories, not just the suspect fixes: data shapes and formats, storage and naming schemes, API contracts and their error behavior, dependencies added, concurrency/perf tradeoffs, scope interpretations, patterns future code will imitate. Auditing your own session, trace your own steps the same way. Choices the plan explicitly delegated to the implementer are discretion, not audit items.

2. Triage each choice on evidence. Forced by the plan, or invented? Invented ones get the scrutiny: is this the general solution, or a fix shaped to the one failing case? Verdict per choice: sound, unsound, or needs-user — and alongside the verdict, a confidence: how sure the audit is that the user would have made this same call. Confidence is what ranks the report. Reserve needs-user for genuinely user-only calls (taste, product direction, external cost); every needs-user entry records a recommended provisional call that is reversible, so an unsupervised caller can proceed without waiting. The audit never stalls a run: each entry is a judgment handed over for action or review, not a question that halts.

3. State the corrected decision, don't sketch a patch. For each unsound choice, the entry names the decision the work should be redone from — the property that must hold in general — not an edit to layer on top. A patch on top of a bad decision preserves the decision; the redo itself is the caller's, after the ledger is reviewed.

4. Bank every choice in the ledger (below), and promote load-bearing sound ones into the plan's handoff so later passes inherit them as givens instead of re-deciding.

5. Present the ledger: grouped by verdict, ranked by confidence. The audit's deliverable is the ledger, handed to whoever acts next — the calling workflow mid-run, the user at run's end. Each verdict group maps to an action — needs-user (decide, with the provisional calls), unsound (redo, with the corrected decisions), sound (acknowledge: the architecture the user now owns) — and within each group choices are ranked by confidence, least confident first. When the ledger is long, open the report with the two or three least-confident choices overall, whatever their group: the "review these first" line. Sound is not skippable. Only trivial discretion (internal naming, cosmetic calls) compresses to a one-line count.

   Write every entry ELI5 — by default, not on request. The reader didn't live the session: write each entry in the eli5 register — a concrete scenario walked end to end (the triggering event, what the work does today, what the unbuilt alternative would do), every term of art defined at first use, and pseudocode at the level of the decision when the choice is about control flow, ordering, or timing. "A gated ask is dropped, not deferred" is a headline, not an entry. A compressed entry that makes the user ask "explain this one" has failed; the ledger must stand alone without the diff, the spec, or the transcript.