Autoresearch

Autoresearch

Try a scoped change, measure it against a fixed benchmark, keep a supported improvement, and record what happened. Continue within the user's authorization and a finite budget. User interruptions and scope changes take precedence over continuation instructions.

Host and mode

Resolve AR_SCRIPTS to the scripts directory alongside this skill's actual file. The helpers require Python 3.10+; Bash wrappers and locking support Linux/macOS.

• Claude Code: invoke /autoresearch for manual installation, or /autoresearch:autoresearch when loaded as a plugin. The four registered Claude hooks handle continuation and compaction. Complete one experiment per turn.
• Codex: invoke $autoresearch in CLI/IDE, or select this skill in the app. Use the same protocol with Codex's available read/edit/exec tools. This package's Claude hook registration does not install Codex hooks. For unattended continuation, use scripts/codex_loop.py as described in references/codex.md. A supervisor turn must complete exactly one experiment and then end. Never nest a supervisor inside a supervised turn.
• Status: read state/dashboard/worklog and report; run no experiments and do not unpause. python3 "$AR_SCRIPTS/ar_state.py" status is a read-only status command.
• Report: write autoresearch-report.md with objective, baseline, best, winning changes, evidence limitations, failures, and remaining ideas; run no experiments.
• Off/pause: create .autoresearch-off and stop. During a running benchmark, cancel safely if possible; preserve partial work and report any unfinished process.

For Claude status/report turns in an active experiment, first create .autoresearch-inspect in the experiment workspace, including when this skill was loaded directly. The Stop hook consumes it to let the inspection turn end. For status, this control-file write is the only write; it does not log or run an experiment. Codex status remains read-only and creates no inspection sentinel.

Setup

1. Establish the goal, benchmark command, primary metric and direction, exact files in scope, fixed scorer/data/splits, correctness checks, and budget. Infer these from the request when possible. Authorization to optimize code does not imply permission to push, deploy, contact people, install dependencies, or rent compute.
2. Inspect git status, including untracked files and the index. Use an experiment branch in a clean dedicated worktree when other work is present; preserve that work without stashing or cleaning it away. Require a committed starting point.
3. Read the workload. Write autoresearch.md using the outline below and create experiments/worklog.md. Prepare autoresearch.sh and, when needed, checks.sh. Commit only explicitly selected initial code/harness files; keep session artifacts local. Check ignores in the target repository: this plugin's .gitignore is not inherited by another project. Add anchored artifact patterns to that project's local git exclude file (locate it with git rev-parse --git-path info/exclude): /autoresearch.jsonl, /autoresearch.md, /autoresearch-dashboard.md, /autoresearch-report.md, /autoresearch.ideas.md, /experiments/, /.autoresearch-off, /.autoresearch-inspect, /.autoresearch.jsonl.lock, /.autoresearch-codex.lock. Preserve existing excludes. Track the benchmark or explicitly ignore /autoresearch.sh if it is session-local.
4. Lock the evaluation definition before optimization. Record scorer hashes, dataset/split identity, dependencies, command, seed schedule, and hardware where relevant. Keep benchmark and metric-emitting code outside editable scope. A legitimate scorer fix requires a new segment and a new baseline.
5. Validate the harness with an appropriate trivial/input-independent control. Run the unchanged baseline 3–5 times to estimate noise. Hold evaluation data and folds fixed; vary only training seeds or repeat timing measurements. Record the raw values, mean, and sample standard deviation. A zero measured deviation is not proof of zero uncertainty.
6. Initialize the state with the measured floor, then log the baseline as the first keep (an unchanged baseline need not create an empty commit). Use a finite maxRuns (default 200), and a practical wall-time budget. Setup/calibration runs are outside the logged experiment count: record and budget them separately.
7. Continue until a budget, target, user pause, or a real execution failure prevents progress. On failure, preserve evidence, pause, and explain the blocker.

Session outline: autoresearch.md

# Autoresearch: <goal>
## Objective
Workload, expected outcome, and what was learned from initial inspection.
## Metrics
Primary name/unit/direction, secondary metrics, measured noise floor and seed values.
## Budget
maxRuns, maxSeconds, optional targetMetric, per-benchmark timeout, setup cost.
## How to Run
Exact benchmark/check commands, working directory, runtime and dependencies.
## Files in Scope
Exact files the experiment may edit.
## Off Limits
Scorer, metric code, data, split/seed definition; record hashes/identity here.
## Constraints
Correctness requirements, resource/network limits, and allowed actions.
## What's Been Tried
Results, failed ideas, insights, and next candidates. Update every 5–10 runs.

Run one experiment

1. Read the current segment, best kept result, session rules, and recent worklog. Check the budget before launching. Do not reset a budget to continue.
2. Record starting HEAD and clean status. Choose one hypothesis and its precise editable file list; keep scorer and evaluation data fixed. Use diverse drafts before refining a promising approach. Cap debugging one idea at three attempts.
3. Run the benchmark with a timeout and a unique log file. Capture stdout and stderr separately; parse METRIC name=number from stdout only. Check exit status even when a metric was printed. Reject missing, duplicate, or non-finite primary metrics. A failing benchmark is crash; passing benchmark plus failing checks is checks_failed. Both consume a run.
4. Inspect only metric lines and bounded log excerpts; keep full output on disk. Do not use a shared /tmp/autoresearch-output.txt or parse tee's exit status as the benchmark status. Use Python's monotonic clock or an available timeout tool; GNU date +%s%N and timeout are not portable to stock macOS.
5. Verify locked files against the initial hashes before accepting or committing a result, including ignored harness files. An instruction to lock a harness is not an access-control boundary. The Codex supervisor checks configured hashes between turns; inspect a detected violation and re-baseline before trusting it.
6. Apply the decision rule, keep or restore the experiment's scoped changes, then append exactly one result and update the dashboard/worklog. Complete cleanup before ending a supervisor turn.