Auto Research

Auto Research

Optimize learning per unit of time. Start with one fast, revealing task, plan competing hypotheses, test them separately, combine verified winners, then expand coverage. The durable result is a parameter-effect map plus the best verified artifact. An experiment is a checkpoint, not a stopping point.

Workflow

1. Recover the research state. Read the objective, user feedback, project instructions, and existing research records. Reconcile the handoff with actual artifacts and running jobs; collect a pending attempt before launching another. Identify the best verified artifact (the incumbent), active task, regression set, hypothesis queue, and remaining budget. Reuse existing record locations.

2. Clarify success before experimenting. Recover any already agreed criteria; do not ask the user to repeat them. If the evaluator, metric, comparison baseline, or required improvement is unclear, ask focused questions and wait for answers before editing candidates or launching evaluations. Inspect existing artifacts and propose concrete options to make answering easier; do not silently choose what success means. Continue clarification until the answers define a checkable criterion, including evaluation scope, aggregation, and non-regression gates. Restate that criterion in the research contract.

   Distinguish passing correctness gates from achieving the optimization target. Specify whether the target is an absolute score, an absolute change, or a relative improvement against a named baseline. “Improve accuracy by 5%” is ambiguous: from 60%, five percentage points means 65%, while 5% relative means 63%. Clarify the units and direction; record the formula when needed. A baseline's value may be measured next, but its identity and the comparison rule must be settled first. Accept open-ended optimization only when that is the user's intent; do not invent a threshold or substitute any improvement for the requested improvement.

   Then define the smallest useful loop. Record mutable parameters and the fixed evaluator and select one cheap task that exposes the behavior being improved. Measure its turnaround and remove unnecessary setup, models, cases, and implementation work from the inner loop. Prefer an architectural spike when the uncertainty is architectural. Available benchmark tasks are a pool, not a requirement to run them all; choose validation coverage to support the intended claim and user's scope.

   Record the evaluation command, timeout, repair allowance, resource limits, promotion rule, and confirmation plan. Separate research spend from the cost or runtime being optimized. Done means the success criterion is resolved and the next trial is cheap to run with an unambiguous decision rule; do not build a large benchmark matrix before testing the first hypothesis.

3. Establish the active task's baseline. Run the unchanged artifact once, or reuse its compatible recorded result and traces. Verify the evaluator measures the intended outcome; use a known failing control if detection is unproven. Keep the baseline fixed for this task/model/harness; do not rerun it per variant. For noisy measurements, plan only the repeats needed to distinguish an effect, respecting any user instruction to reuse a single baseline and reporting that limitation. Save exact inputs, artifact identity, and raw output.

4. Plan a small hypothesis batch. Before editing, list distinct explanations of the current bottleneck and the parameters or strategies that could test them. Inspect actual failure traces and matched baseline behavior instead of guessing from aggregate scores. For each hypothesis, record:

   • Parameter/change and mechanism: why it might affect the outcome.
   • Predicted observation, falsifying result, and cheapest discriminating test.
   • Fixed comparison parent, expected trial cost, and dependencies or likely interactions with other hypotheses.

   Rank by information gained per unit of time. Test alternatives one by one against the same parent so their effects remain interpretable. Avoid exhaustive parameter grids. Keep the queue short and revise it after results; planning is part of every research cycle, not a one-time backlog or a reason to delay the first trial.

5. Test one hypothesis on the active task. Preserve the incumbent and isolate the candidate from unrelated work. Change one conceptual factor; if a coupled bundle is necessary, attribute the result to that bundle. Record the candidate identity and prediction before running. Use cheap checks first, then the focused task; do not run the broad suite for every variant. Save uniquely identified raw output and inspect relevant traces. Bound crashes and repairs; a repaired candidate gets a new identity. Missing results are unknown, not zero.

   Compare observation with prediction, including regressions and null results. Record keep, discard, inconclusive, or invalid and update the map. Restore the parent between alternatives; retain reproducible snapshots or patches and evidence outside rollback scope. Rejecting a candidate does not disprove its entire strategy: diagnose a promising partial win and plan a targeted follow-up when evidence supports one.

6. Combine and promote. Test compatible winners together against the strongest individual candidate. Do not assume gains add: check interactions, and remove one change at a time when attribution is unclear. A combination that loses to a simpler candidate does not become the incumbent.

   A focused win is provisional. Before promotion, check the candidate against previously solved tasks in the accumulated regression set, using their saved baselines and the declared tolerances. Reject or repair regressions rather than hiding them in an average. Confirm noisy wins with the planned evidence, including all planned attempts and costs. Bank the verified candidate and update the parameter-effect map with interactions and task-specific limits.

7. Expand only after progress. Once the active task meets its local criterion and prior tasks remain green, add one task or a small batch that probes a new failure mode or an uncertain effect in the map. Establish missing baselines only for those tasks. If a new task fails, make it the active task, return to hypothesis planning, and retain the old tasks as regression checks. Do not repeatedly run the expanded suite while debugging that one failure.

   This curriculum changes discovery coverage, not the overall success criteria. Reserve broader evaluation for coverage-expansion checkpoints and final confirmation. A single-task win cannot complete a broader objective.

8. Replan from what was learned. After a hypothesis batch, a new failure, or several non-improving trials, consolidate the map and choose the next most informative batch. Change the suspected mechanism or experiment design when the evidence contradicts it; do not endlessly retune one family or rerun the same tests. Revisit a rejected idea only with a changed premise. Continue without asking whether to proceed after each experiment.