Autoresearch Loop

autoresearch-loop

Generalize Karpathy's autoresearch into a domain-adaptive improvement loop. The agent discovers what to measure, then runs a disciplined propose → trial → keep-or-revert loop, maintaining an explicit ledger of what was tried, kept, discarded, and implemented.

Read DESIGN.md once at the start of a run for the full architecture and the domain-specific tensions (metric latency/noise/cost, Goodhart gaming, cost-per-trial, reversibility). The phases below are the operating procedure.

Runtime: the loop's mechanics (run a trial, parse the metric, score confidence, keep/commit or discard/revert) are handled by the arl CLI over a .auto/ session folder — a Claude-native port of pi-autoresearch's tools. Read references/runtime-contract.md for the .auto/ layout, the METRIC name=value contract, MAD confidence scoring, and the arl init|run|log|status commands. Invoke it as node scripts/arl.mjs <cmd> (or arl if on PATH).

When NOT to run

• There is no metric that can be measured repeatably and cheaply enough within a trial budget. A loop with no trustworthy metric chases noise — stop and say so.
• The change surface is irreversible or unsafe to mutate experimentally (production data, customer-facing irreversible actions) without an explicit revert procedure in the adapter.

Phase 0 — FRAME (metric discovery)

Given {project, goal, context} — follow the procedure in references/metric-discovery.md (restate the goal as an outcome → enumerate candidates on the proxy→outcome spectrum → score on six axes → choose primary + guardrails + strategy → red-team for gaming). In brief:

1. Identify or select a domain adapter (adapters/*.md). If none fits, draft an inline adapter following references/domain-adapter-contract.md.
2. Propose candidate metrics (the adapter's menu is a prior, not the answer — reason from the goal). Score each on measurability, latency, noise, alignment, gameability, and cost. Prefer alignment over convenience for the primary.
3. Pick ONE primary objective, a set of guardrail metrics that must not regress, a trial budget (wall-clock and/or cost per trial), and a stop condition (budget exhausted, plateau over K trials, or target hit).
4. Choose the accept/reject strategy for this domain: deterministic-delta (fast, low-noise), significance-test, or bandit (noisy/delayed/expensive — e.g. ads).
5. Name the Goodhart guards (guardrail metrics, holdout, periodic critic).
6. Write runs/<id>/CHARTER.md. Confirm it with the user before spending real budget if trials cost money or touch production.

Phase 1 — BASELINE

Write .auto/measure.sh (and .auto/checks.sh if guardrails require it). arl init with the primary metric + direction, run the baseline (arl run), and record it (arl log --status keep --metric <baseline> --desc baseline). If the baseline can't be measured cleanly and repeatably, stop (see "When NOT to run").

Before proposing any change, profile where the cost actually is, and confirm the benchmark stresses the IN-SCOPE artifact — not a dependency, a native/FFI call, an external engine, the network, or unrelated code. (Validated the hard way on two live runs: once the assumed hot path was wrong twice and 97% of time was in an out-of-scope library; once the in-scope managed code was only 0.4–3.8% of wall-time because a Rust NIF dominated — the correct loop output there was a true negative, "re-scope," not a sub-noise edit. See adapters/code-perf-audit.md → Pitfalls.) Spend one profiling run on the managed-vs-native/dependency split; a loop that optimizes code which isn't the bottleneck produces confident, useless churn — and proving "no in-scope headroom" cheaply is itself a successful outcome.