Evolve Design

Step 3 — Size the run

1. What is one unit? Whatever one measurement consumes: rows, a test case, a document section, one input scenario, one grading pass.
2. How big must one be to be stable? Enough that the same candidate scores the same twice. A single item is worse than noisy: it makes the unit binary, and the search skips proposing on a unit it already solves, so every unit at full marks spends an expansion and yields no candidate (measured: one record per unit, eleven of sixteen at 1.0, a run planned for 20 expansions made 5). Put enough in one unit that a good candidate lands between the floor and the ceiling — a few dozen items averaged, not one.
3. How many in the gate — and make it the biggest of the three. Every candidate's score, the one the tree ranks and selects on, is measured on the gate. Too few and the tree ranks on noise for the whole run, and the reported improvement does not survive a re-run. Deterministic scoring 8–12 units; anything with randomness 16–24. Rollout drives the search's own trajectory: four or more, never one (five candidates all scoring exactly 0.6000 was one), and no more than the gate. Test never takes part and is read once at the end: 4–8.
4. Does the candidate learn from data? If it fits before it produces, the fitting volume must match the evaluation volume. Skip when nothing is fitted.

If the total will not fit, shrink each unit — do not cut the gate, which is the only number the search steers by.

expansions must be at least 4 × workers, or the first sweep forks only the root and the tree is flat. By search space: a known defect 4–6; swapping approach or restructuring 12–20; writing something from scratch 20+. Thinking off unless asked — with it on, one whole-candidate rewrite can exceed the proxy limit and return nothing.

search — leave it out unless this task argues against the defaults. They are upstream's and usually right; the parameter descriptions on create_evolve_run carry the ranges and what each does. Set one only for a reason you can give in a line:

• cPuct — lower when the gate is large enough to trust and the budget is small; higher when the score is noisy or coarse or candidates keep tying. A coarse score is a scoring problem first: widen the gate before reaching for it.
• priorExponent — when the failure you expect is a whole-mechanism rewrite (five of eighteen compression candidates replaced a working RLE+Huffman with a from-scratch arithmetic coder, and each scored 0). It cannot move the reported score; that comes from held-out shards.

Checkpoint 3. Show the shape in a few lines: what one unit is, how much is held out, how many expansions with how many workers, and roughly what that costs in time and model calls. Mention search only if you set it, in one line saying why; otherwise leave it out. This is the last point before real money is spent — say so plainly, and default to the smaller option when unsure.

Step 4 — Start it, then report

Call create_evolve_run. Set algorithm to what the user chose. howScored is one sentence for the user: what it measures, how much is held out. risks: at most two, only ones that change a decision; empty is fine.

Checkpoint 4. Report what came back — the probe's two numbers and what the search will do — and then end the turn. Do not wait for it, do not call get_evolve_run to check on it, do not loop. The search runs for minutes to hours; its card streams live progress, and sitting on the turn shows the user nothing they cannot already see and burns the run's own budget. get_evolve_run is for later, when the user asks how it went — then report the actual numbers, never an improvement you have not read.

A refusal is design feedback, not an error. "Cannot discriminate" means the scoring needs harder cases or a more mechanical rubric; "no slope" means the starting point is too strong. Fix it and call again — do not hand the user the server's refusal text as a work order.

Choosing a scoring mode (during step 2)

Take the first that fits. custom_script is the fallback, not the default: you write and maintain the measuring apparatus yourself, so every mistake in it is yours. Judge by what the task is, not by what is in the workspace right now — "there is no table yet" is not a reason to skip dataset_metric when the task is to predict a column; write the table, then use it.

• Cases with known answers, and a number to move → dataset_metric. Deterministic and cheapest, and the framework owns the split so you cannot get it wrong. Metrics: accuracy / mae / r2 / rmse / seconds.
• Correctness pinned down by tests → test_gate. Take it whenever the user says "write tests", "make these cases pass", or describes behaviour case by case. The failure text is the learning signal, and frozenGlobs must include the test paths, or the shortest way to a higher score is to weaken the tests. Shards ≤ half the case count.
• Neither fits → custom_script, which you write: simulations, optimisation heuristics, anything whose quality is a computation with no natural table and no test suite. Read references/custom-script.md before writing the evaluator — it holds the contract, how it runs, and the error rule the probe enforces (a crash must say where, not just what).
• Only another model can judge it → llm_judge. For prose and explanations, anything whose quality is a reading. The most gameable and the only non-deterministic mode; ask once whether it could be a custom_script instead.

Score on a gradient, not a cliff. For hard limits prefer "stop and score what you have" over "violation scores zero": zeroing lands every failure on the same 0 and leaves nothing to climb.

Never reward a property the candidate can fake. Rewarding "gives specific numbers" produces invented numbers; reward agreement with the given source instead.

Environment

Candidates get numpy / pandas / scipy / sklearn and the standard library. Anything else goes in packages — bare names only (optionally ==version), no paths, URLs, or pip options. Your run_shell environment and the candidate sandbox are not the same.