Experiment Suite

Experiment Suite

Overview

End-to-end experiment package builder. Single stage, full quality from the start. The agent (Claude Code / Cursor / Aider / Codex / …) writes everything directly using its own tools (Write, Bash, WebFetch, …). This skill contains procedure + reference playbooks + figure-example scripts — no Python runtime, no LLM SDK.

The substantive work is decomposed into reference playbooks under references/:

Also: figure_examples/ — publication-style matplotlib scripts plus a shared style kit the agent can use as starting points.

Read the relevant reference before writing, not after. The full pass does not fit in a single turn — references/00-incremental-execution.md is the only execution mode that completes.

When to Use

• User wants to "design an experiment" for a research question.
• User needs runnable code for a specific task (classification / forecasting / detection / …).
• User wants to compare methods and have a structured report at the end.
• User needs publication-quality figures of experimental results.

When NOT to Use

• User only wants a quick code snippet (write code directly).
• User wants a full paper → paper-writer.
• User wants a literature survey → literature-survey.

Workflow

Step 1 — Understand the question and operating mode

Confirm with the user:

• Research question — what we are trying to answer.
• Task type — classification / regression / forecasting / detection / generation / …
• Mode
  • measured — user has real data or will run code themselves; provide a path to a measured results.json or run train.py against real data later.
  • simulated (default) — agent generates a plausible-shaped, deterministic results.json as a placeholder. Every figure/table caption must say "simulated".
• Framework preference — PyTorch (default), JAX, TensorFlow, or sklearn.
• Compute budget — hours / GPUs available; constrains the code skeleton and hyperparameter plan.

If the user has data and time, push toward measured mode. If not, simulated is acceptable provided disclosures are honest in every artefact.

Step 2 — Set up the run directory

QUESTION="<research_question>"
SLUG=$(python3 -c "import re,hashlib,sys; t=sys.argv[1]; n=re.sub(r'[\\s_]+','-',re.sub(r'[^\\w\\s-]','',t.lower().strip())).strip('-')[:40].rstrip('-'); h=hashlib.sha1(t.encode()).hexdigest()[:8]; print(f'{n}-{h}')" "$QUESTION")
TS=$(date +%Y-%m-%d_%H%M%S)
RUN=output/experiment-suite/$SLUG/$TS

mkdir -p "$RUN/experiment" "$RUN/figures"
ln -sfn "$TS" "output/experiment-suite/$SLUG/latest"

In commands below $RUN = output/experiment-suite/<slug>/latest.

The agent will create five top-level files inside $RUN/:

• experiment_design.md
• data_contract.md
• experiment/{model.py,data.py,train.py,evaluate.py,config.yaml,requirements.txt,README.md}
• results.json
• figures/*.pdf plus their make_*.py source and a manifest.json
• experiment_report.md

Step 3 — Build the package (REQUIRED — this is the whole job)

Open references/00-incremental-execution.md first. Then carry out the six tracks below across many turns, persisting state to $RUN/ after every batch.

3.1 Design — full justification document

Open: references/01-design-depth.md and references/01a-data-contract.md. First write $RUN/data_contract.md as the dataset contract for this run. It must say whether the data are user-supplied, agent-discovered, reused public, controlled, or synthetic fallback. Then write $RUN/experiment_design.md as a real design (≥ 700 words): motivation → hypothesis → datasets → baselines → metrics → ablations → compute budget. Justify every choice.

3.2 Code — actually runnable

Open: references/02-code-quality.md. Fill $RUN/experiment/ with code that an engineer could launch with python train.py --config config.yaml. Real (if minimal) model class, real data loader, real train loop, real eval. The generated data.py and config.yaml are runtime products of this run and should bind to $RUN/data_contract.md, not to a repository-wide hard-coded benchmark. Add a README.md with run instructions.

3.3 Results — honest provenance

Open: references/03-results-protocol.md. Produce $RUN/results.json with a well-formed schema: per-seed entries, per-method per-metric mean & std, ablation block, and a provenance field that names the source.

• measured mode — the user runs experiment/train.py (or supplies a results JSON) and the agent loads it into $RUN/results.json, setting "simulated": false and "provenance": "loaded from <path>".
• measured mode, agent-discovered data — the agent may search for and bind an open dataset itself, but the chosen source, split, and access route must first be written into $RUN/data_contract.md; results.json provenance must point back to that binding.
• simulated mode — the agent writes a deterministic seeded JSON of plausible shape, setting "simulated": true.