Iterative Refinement

Iterative refinement with verifiable conditions

Improve any system (script, pipeline, prompt, doc, config, dataset) by looping against measurable pass/fail conditions while keeping the main thread's context lean. Task-agnostic. The aim is to make "is it good yet?" a single repeatable command, catch each regression at the edit that caused it, and keep load-bearing reasoning in cheap, auditable steps.

Use this as a toolkit, not a script. Each method below earns its place by what it prevents, and that reasoning is stated inline so you can judge when it applies. Reach for the methods the situation calls for, scale them to the stakes, and adapt or skip what doesn't fit; they compose well, but no fixed subset is mandatory and this skill can't anticipate every task you'll point it at. When a method clearly fits, lean into it fully rather than half-applying it. The judgement of which to use, and how hard, stays yours.

The loop

1. Turn the goal into a pass/fail rubric with explicit thresholds. "Useful" isn't checkable; "contamination < 2%, both naive and effective ratios reported, every input row classified, prints Overall: PASS" is.
2. Bake the rubric into the artifact as a self-check it prints. The artifact computes its own metrics and prints [PASS]/[FAIL] per condition, so the check can't drift out of sync with the code the way an external checklist does. Now any party (you, a subagent, the user) re-verifies with one command.
3. Change one layer, re-run the check. Catch each regression at the edit that caused it, not three edits later.
4. Loop on a fast slice, not the full dataset. Size the slice so a run takes seconds, not minutes. Run the full corpus only at checkpoints, when a layer is structurally complete and at sign-off, since that's where rare signals and period splits actually appear and the cost is justified.
5. Keep a frozen held-out slice the loop never touches, and confirm against it before sign-off. Looping hard on one slice optimises the rubric for that slice (Goodhart on your own metric); the held-out slice is what proves the gain generalised rather than memorised.
6. When all conditions pass but value remains, write a stricter rubric and loop again. Stop when a cycle yields nothing material.

Writing good conditions

• Cheap to evaluate: exit code, grep, a printed PASS line, a line count. If checking costs as much as the work, you won't loop.
• Falsifiable with a number in it. "< 2%", not "low".
• Tied to the goal, not a proxy. A metric can pass while the output is useless, so pair aggregates with a meaningfulness audit (below) that samples real outputs.
• Reconciliation invariants are programmatic gates between steps that catch silent bugs: classified counts must sum to the total; naive and effective measures must bracket reality.
• Track cost alongside correctness. A rubric that watches only accuracy hides regressions in the dimensions that also matter: tool-call count, token consumption, wall-clock, and error rate. A change that "passes" but doubles tokens is a finding, not a success.

Context-window economics

Every byte a tool returns stays in context and taxes every later turn. This is context engineering: treat the window as the scarcest resource, because reasoning quality degrades with depth (context rot) well before any hard limit. Three mitigations, highest leverage first:

1. Think-in-code. To process output (filter/count/parse/aggregate), run code over it and print only the answer; raw bytes never enter the conversation. Reading bytes is justified only when you'll edit them. The context-mode ctx_execute tools do this in a sandbox when available.
2. Return conclusions, not transcripts. A subagent's tool history stays in its context; only its final message returns. A task worth 50 file-reads costs you one paragraph. Forbid the subagent from pasting file bodies or full reports.
3. Locate before reading. Print the line numbers or symbols you need with one targeted query, then read only that span.

Reasoning degrades in bands: roughly 0-100k tokens is peak, 100-150k still strong, 150-200k noticeably softer, and auto-compaction looms around a third of the window. Budget against the bands, not the ceiling. Bring the main thread's irreversible state (task list, strategy notes, decisions) up to date before you approach compaction; a summary written ahead of time survives, working memory you were relying on may not. If a single task can't fit the smart zone, that's a signal to decompose and delegate, not to push through.

Delegation and subagents

Push work off the main thread whenever it produces more bytes than its conclusion is worth.

• Delegate bulk reads/scans (raw bytes stay in the subagent) and well-scoped edits.
• For a delegated edit, give a tight spec: the objective, the exact variables/contracts it may touch, hard "don't touch X" boundaries, the output format, and the SAME machine-checkable acceptance loop you'd run. It inherits your verification standard and returns a compact summary.
• State invariants up front so framing can't drift (for example: assess on effective metrics, never relabel the workflow "degraded", keep output lean). Cheaper to state than to repair.
• Named subagent (fresh context) for research, inspection, or anything needing an unbiased read. Fork (inherits your context) only when accumulated nuance helps and a fresh view wouldn't. Never fork a review, audit, or premise-check; a fork inherits your blind spots.
• Disjoint ownership when running several at once; overlapping write scope means they erase each other.
• Verify what returns: a summary is a claim. Spot-check load-bearing results with a cheap command, or have a second independent agent check.

Fan-out topology

Match it to the write-pattern:

• Sequential single subagent when edits hit the same region of one file; parallelism only causes merge collisions.
• Parallel read-only subagents when subtasks mutate nothing shared. Best fit for a meaningfulness audit: one samples classifier false-negatives, one judges detector precision, one sanity-checks counts, same data, no writes, can't conflict. Send them in one batch so they truly run in parallel.
• Git-worktree agents only when agents must write concurrently, or to race competing variants for a winner. They cost setup and disk; don't use when read-only or sequential suffices.
• Scale the fleet to the task's complexity, and say so when you cap coverage. Rough budget: a simple lookup is one agent and a few tool calls; a broad audit is many agents in parallel. Fan-out multiplies token spend by agent count (a parallel multi-agent sweep can run on the order of 10-15x a single-threaded pass), so "ran 5 of 20" must never read as "covered everything".