Autoresearch Agent

Autoresearch Agent

You sleep. The agent experiments. You wake up to results.

Autonomous experiment loop inspired by Karpathy's autoresearch. The agent edits one file, runs a fixed evaluation, keeps improvements, discards failures, and loops indefinitely.

Not one guess — fifty measured attempts, compounding.

Slash Commands

When This Skill Activates

Recognize these patterns from the user:

• "Make this faster / smaller / better"
• "Optimize [file] for [metric]"
• "Improve my [headlines / copy / prompts]"
• "Run experiments overnight"
• "I want to get [metric] from X to Y"
• Any request involving: optimize, benchmark, improve, experiment loop, autoresearch

If the user describes a target file + a way to measure success → this skill applies.

Setup

First Time — Create the Experiment

Run the setup script. The user decides where experiments live:

Project-level (inside repo, git-tracked, shareable with team):

python scripts/setup_experiment.py \
  --domain engineering \
  --name api-speed \
  --target src/api/search.py \
  --eval "pytest bench.py --tb=no -q" \
  --metric p50_ms \
  --direction lower \
  --scope project

User-level (personal, in ~/.autoresearch/):

python scripts/setup_experiment.py \
  --domain marketing \
  --name medium-ctr \
  --target content/titles.md \
  --eval "python evaluate.py" \
  --metric ctr_score \
  --direction higher \
  --evaluator llm_judge_content \
  --scope user

The --scope flag determines where .autoresearch/ lives:

• project (default) → .autoresearch/ in the repo root. Experiment definitions are git-tracked. Results are gitignored.
• user → ~/.autoresearch/ in the home directory. Everything is personal.

What Setup Creates

.autoresearch/
├── config.yaml                        ← Global settings
├── .gitignore                         ← Ignores results.tsv, *.log
└── {domain}/{experiment-name}/
    ├── program.md                     ← Objectives, constraints, strategy
    ├── config.cfg                     ← Target, eval cmd, metric, direction
    ├── results.tsv                    ← Experiment log (gitignored)
    └── evaluate.py                    ← Evaluation script (if --evaluator used)

results.tsv columns: commit | metric | status | description

• commit — short git hash
• metric — float value or "N/A" for crashes
• status — keep | discard | crash
• description — what changed or why it crashed

Domains

If program.md Already Exists

The user may have written their own program.md. If found in the experiment directory, read it. It overrides the template. Only ask for what's missing.

Agent Protocol

You are the loop. The scripts handle setup and evaluation — you handle the creative work.

Before Starting

1. Read .autoresearch/{domain}/{name}/config.cfg to get:
   • target — the file you edit
   • evaluate_cmd — the command that measures your changes
   • metric — the metric name to look for in eval output
   • metric_direction — "lower" or "higher" is better
   • time_budget_minutes — max time per evaluation
2. Read program.md for strategy, constraints, and what you can/cannot change
3. Read results.tsv for experiment history (columns: commit, metric, status, description)
4. Checkout the experiment branch: git checkout autoresearch/{domain}/{name}

Each Iteration

1. Review results.tsv — what worked? What failed? What hasn't been tried?
2. Decide ONE change to the target file. One variable per experiment.
3. Edit the target file
4. Commit: git add {target} && git commit -m "experiment: {description}"
5. Evaluate: python scripts/run_experiment.py --experiment {domain}/{name} --single
6. Read the output — it prints KEEP, DISCARD, or CRASH with the metric value
7. Go to step 1

What the Script Handles (you don't)

• Running the eval command with timeout
• Parsing the metric from eval output
• Comparing to previous best
• Reverting the commit on failure (git reset --hard HEAD~1)
• Logging the result to results.tsv

Starting an Experiment

# Single iteration (the agent calls this repeatedly)
python scripts/run_experiment.py --experiment engineering/api-speed --single

# Dry run (test setup before starting)
python scripts/run_experiment.py --experiment engineering/api-speed --dry-run

Strategy Escalation

• Runs 1-5: Low-hanging fruit (obvious improvements, simple optimizations)
• Runs 6-15: Systematic exploration (vary one parameter at a time)
• Runs 16-30: Structural changes (algorithm swaps, architecture shifts)
• Runs 30+: Radical experiments (completely different approaches)
• If no improvement in 20+ runs: update program.md Strategy section

Self-Improvement

After every 10 experiments, review results.tsv for patterns. Update the Strategy section of program.md with what you learned (e.g., "caching changes consistently improve by 5-10%", "refactoring attempts never improve the metric"). Future iterations benefit from this accumulated knowledge.

Stopping

• Run until interrupted by the user, context limit reached, or goal in program.md is met
• Before stopping: ensure results.tsv is up to date
• On context limit: the next session can resume — results.tsv and git log persist