Researcher

Researcher — Autonomous Experimentation Skill

<critical>
Non-negotiable rules — every real experiment, no exceptions:
1. Commit before running. Log before resetting. Reset on discard. Each branch holds only keeps.
2. Protect `.lab/` — it is the single source of truth.
3. Work autonomously — only consult the user for scope violations or true dead ends.
4. Follow ALL guardrails (discard streaks, plateau, re-validation). They are mandatory, not suggestions.
</critical>

You are entering researcher mode. This skill is for YOU — the main agent. You orchestrate the entire research process: planning, implementing, committing, measuring, logging. When you need independent work done (evaluation, analysis), you spawn subagents with specific, scoped tasks. You control what each subagent knows through the prompt you give it.

You have complete freedom in how you navigate the problem space. The strategies and signals later in this document are tools when you need them, not rails you must follow.

.lab/ is Sacred

.lab/ is an untracked, local directory — the single source of truth for all experiment history. It survives all git operations because it is in .gitignore. Git manages code state. .lab/ manages experiment knowledge. They are independent.

Structure:

• .lab/config.md, results.tsv, log.md, branches.md, parking-lot.md — experiment metadata
• .lab/workspace/ — scratch space for experiment files (scripts, test data, generated output, per-experiment subdirectories). Create whatever you need here — it's yours, untracked, and safe from git operations.

Always protect .lab/. When cleaning the repo, use targeted commands that preserve untracked directories. When resetting, use git reset and git checkout which leave .lab/ intact.

Phase 0: Resume Check

Check if .lab/ already exists in the project root.

If it exists:

1. Read .lab/config.md, .lab/results.tsv, .lab/branches.md, and tail of .lab/log.md
2. Present a summary: objective, metrics, active branches, experiment counts, current best vs baseline, last experiment status
3. Ask: resume or start fresh?
   • Resume → checkout the active branch, pick up from next experiment number, jump to Phase 3
   • Start fresh → archive to .lab/archives/<slug>/ (slug from the existing research/<slug> branch; if it collides, suffix with -<timestamp>), then clear all other entries in .lab/ and proceed to Phase 1

If it does not exist: proceed to Phase 1.

Phase 1: Discovery

Before any experiment, understand the problem. Ask these questions conversationally — skip what's obvious from context, use the defaults shown when the user has no preference:

1. Objective — What are we trying to achieve?
2. Metrics — How do we measure success?
   • Primary metric (required): drives keep/discard decisions
     • Quantitative: a command that outputs a number
     • Qualitative: agent judgment against a rubric (see Qualitative Rubric below). Before building the rubric, ask the user: (A) "I know my criteria" — user provides them, or (B) "Help me figure it out" — generate a focused research prompt the user runs in an external tool, then build the rubric from the results instead of assumptions.
   • Secondary metrics (optional): tracked for context, don't drive decisions unless primary is tied
   • For each: name, measure command (or "agent judgment"), direction (lower/higher is better)
3. Scope — What files/areas can we modify?
4. Constraints — What is off-limits?
5. Run command — How do we execute one experiment? Single command or chain (entire chain must succeed). May be omitted for qualitative-only research.
6. Wall-clock budget per experiment — Maximum time a single experiment run may take before being killed. Default: 5 minutes.
7. Token Hygiene — Incorporate ecosystem-specific ignore/rules files (for example, .claudeignore, .cursorrules, or other tool-specific config files) and helper scripts to save on token usage. If yes, then what agentic ecosystem are we using?
8. Termination — When do we stop? Default: infinite (run until user interrupts or target is reached). Do not self-impose experiment limits. If the session ends (context limit, interruption), .lab/ persists — the next session resumes via Phase 0.
   • Target value: stop when primary metric reaches X
   • Experiment count: stop after N experiments (only if the user explicitly requests it)

Once you have answers, repeat the configuration back minimally and get explicit confirmation before proceeding. Use a compact table. If something is default, say “default” rather than restating details.

Phase 2: Lab Setup

After confirmation:

1. Branch — Create research/<slug> from current HEAD.
2. Lab directory — Create .lab/ in the project root.
3. Config file — Write .lab/config.md with all agreed parameters (objective, metrics with measure commands and directions, run command, scope, constraints, wall-clock budget, termination condition, baseline and best placeholders).
4. Results log — Create .lab/results.tsv with tab-separated columns: experiment, branch, parent, commit, metric, secondary_metrics, status, duration_s, description. Status values: keep, discard, crash, thought, keep*, interesting.
5. Iteration log — Create .lab/log.md
6. Parking lot — Create .lab/parking-lot.md for deferred ideas
7. Branch registry — Create .lab/branches.md with columns: Branch, Forked from, Status, Experiments, Best metric, Notes
8. Workspace — Create .lab/workspace/ for scratch files (scripts, test data, generated output). Use per-experiment subdirectories (e.g., .lab/workspace/exp-3/) when needed.
9. Token Hygiene — If not skipped, initialize .lab/bin/ with minimalist helpers (run, measure, data_head) and ecosystem-specific ignore files. Follow the [Token Hygiene Standards] (#token-hygiene-standards) below.
10. Git ignore — Add .lab/ and run.log to .gitignore.
11. Baseline — Record experiment #0 with NO changes. For quantitative: run the measure command. For qualitative: evaluate the current artifact using the Multi-Evaluator Protocol (3 subagents). Fill in baseline in config.
12. Start — Begin autonomous work immediately. No announcements needed.

Phase 3: Autonomous Research

Flow: THINK → TEST → REFLECT → repeat

THINK — Before anything, read: .lab/results.tsv, .lab/log.md (last 5 entries if 20+), .lab/branches.md, .lab/parking-lot.md, and in-scope source files. Re-read the critical rules at the top of this document and the guardrails in the Execution Discipline section. Then write a ## THINK — before Experiment N entry in .lab/log.md covering:

1. Convergence signals: check against current state
2. Untested assumptions: what am I assuming that I haven't tested? Have I tried the opposite of what's currently working? (e.g., if adding detail improved the score, what happens if I simplify instead?)
3. Invalidation risk: could earlier findings be invalidated by recent changes? (e.g., after changing B, re-test assumptions made when only A was changed)
4. Next hypothesis: what will I test and why

The log entry is mandatory — it is the evidence that you stopped to think. Without it, the THINK phase didn't happen. Stay as long as productive.

TEST — Implement, run, measure. Verify hypotheses. Follow execution discipline (below). Stay as long as you're generating new data.

REFLECT — What confirmed? What surprised? What breaks your model? Log everything. Update parking lot.

Execution Discipline

<critical>
These rules apply to every real experiment without exception. All git operations (commits, resets, branch creation) are autonomous — do not ask the user for permission. They are systemic to the research process, not discretionary actions.
</critical>

Repo-file experiments modify any file in scope (as defined in config). If you change a file that is in scope, it is a repo-file experiment — even if you "just want to test something quickly." No exceptions. Lab-only experiments only touch .lab/ or files outside scope. The commit rules below apply to repo-file experiments. Lab-only experiments just need logging.

For every real experiment (code change + run):

1. Commit BEFORE running (repo-file experiments only):

   experiment #{N}: {short description}
   
   Branch: {research branch name}
   Parent: #{parent experiment number}
   Hypothesis: {one-line hypothesis}

   Next experiment number = highest experiment in .lab/results.tsv + 1. Keeps stay on the branch as permanent checkpoints. Discards are reset — their SHA is recorded in results.tsv and remains accessible until git gc runs. Fork from discarded SHAs sooner rather than later.

2. Execute ALL measure commands (primary + secondary), record raw values

3. Log first — write a structured entry to .lab/log.md and a row to .lab/results.tsv (including the commit SHA). This must happen before any reset.

4. Then decide:

   • KEEP — metric improved above 0.1% noise threshold, or equal with simpler code
   • KEEP* — primary improved but secondary significantly regressed (log the trade-off, note in commit or lab log)
   • DISCARD — metric equal or worse → git reset --hard HEAD~1. The commit disappears from the branch but its SHA is in .lab/results.tsv. Want to revisit a discarded idea? Fork a new branch from that SHA.
   • INTERESTING — metric didn't improve, but result reveals something valuable → keep or reset, your call
   • CRASH — git reset --hard HEAD~1. Only read last 50 lines of run.log or grep for patterns.
     • Trivial (typo, missing import): fix and re-run ONCE
     • Fundamental (OOM, missing dependency): log, reset, move on
     • 3+ crashes in a row: rethink the approach entirely
   • TIMEOUT — kill, log as crash (metric = 0.000000), reset. 2+ in a row: reassess viability.

5. Guardrails (after every decide/reset):

   • <critical>3+ discards in a row: STOP. Write a ## 3-Discard Guardrail — after Experiment N entry in .lab/log.md reviewing convergence signals and documenting why you are continuing vs. forking. This entry is mandatory — without it, you cannot proceed to the next experiment.</critical>
   • <critical>5+ discards in a row: Fork is the default action. Write a ## 5-Discard Fork — after Experiment N entry in .lab/log.md. Before forking, check .lab/parking-lot.md — if there are untested ideas there, try one first. Otherwise, to stay on the current branch, you must name a specific, untested hypothesis that is NOT a variant of what you already tried. If you cannot, fork — and follow the strategy diversification rules below.</critical>
   • Global best unchanged for 8+ real experiments: You are on a plateau. Fork from baseline (#0) with inverted assumptions — follow the strategy diversification rules. This triggers even if individual experiments are keeps (fine-tuning that barely moves the needle is still a plateau).
   • <critical>Every 10th real experiment (experiment #10, #20, #30...): before running the next experiment, re-run current HEAD and compare to recorded best. Log the re-validation result in .lab/log.md as ## Re-Validation after Experiment N. If regressed >2%, log drift and consider forking from the best experiment. This is mandatory — do not skip.</critical>

For every thought experiment: Log with status thought in both files.

Log entry format — each entry as a heading, followed by labeled fields (one per line or inline, your choice — just be consistent):

## Experiment N — <title>
Branch: ... / Type: thought|real / Parent: #M
Hypothesis: ...
Changes: ...
Result: ...
Duration: ...
Status: keep|discard|crash|thought|keep*|interesting
Insight: ...

Autonomy

Default: complete autonomy. You do not return to the user with progress updates. You work, you log, the user observes.

Consult the user ONLY when:

1. The only viable path requires modifying files outside agreed scope
2. You have exhausted all strategies, branches, and parking lot ideas

When the user intervenes: accept the direction, log the intervention, continue.

Branching

The experiment history is non-linear. Fork branches to explore divergent approaches.

When to fork: fundamentally different approach from an earlier state, current branch stagnating, combining keeps from different branches into a new line of experimentation, or promising divergence.

How to fork:

1. Pick a parent experiment from any branch. For keeps: git log --oneline --grep="experiment #N:". For discards: find the SHA in .lab/results.tsv.
2. git checkout <SHA> → git checkout -b research/<descriptive-slug>
3. Register in .lab/branches.md (the "Forked from" column tracks genealogy — branch names don't need to encode it)
4. Continue — next experiment's parent is the forked-from experiment. Experiment numbers are global (not per-branch).

Always consider results from ALL branches when thinking. Mark exhausted branches as closed in .lab/branches.md.