Finetuning

Finetuning

Priors, not rules. Only firm guardrails: held-out eval you never train on, no leakage, trust evo's recorded numbers over the run's self-report. Override anything else against the gate.

Pick the technique by reward shape

Decide on the reward first, technique second. Choosing the comfortable technique over the matching one is the most common failure.

"SFT-then-RL" is not a law. For a competent base model on a verifiable benchmark, RL-from-base often beats SFT-then-RL end-to-end.

Research the literature before the first commit

The decision tree above is the structural prior. The empirical answer for this model on this benchmark usually has a recent paper, blog, or HF Space recipe behind it -- and what beats baseline on a 4B base model in 2026 is not what the agent's pre-training data captures. Before picking the technique for exp_0001 (the first experiment after baseline), invoke evo:ideator with a literature brief:

Task(
    subagent_type="evo:ideator",
    prompt="brief=literature\n"
           "model_family=<e.g. Qwen3-4B-Base, Llama-3.1-8B-Base>\n"
           "benchmark=<name + URL/paper if known>\n"
           "objective=<one line: what beats baseline looks like>\n"
           "constraints=<budget, data sources allowed, gated models forbidden, etc>"
)

The ideator returns ranked proposals with references (arXiv, HF Hub, GitHub, blogs). Read them before picking from the reward-shape table. A paper showing GRPO-from-base works on <model_family> for a similar verifiable benchmark beats applying the table cold.

Run this once before exp_0001, and again whenever the optimize loop hits a plateau (the "stuck across distinct techniques" diagnostic below). Not every subsequent experiment needs a literature pass -- the table + diagnostics carry the rest.

Before committing the budget: smoke-run

Run the full pipeline on ~10 examples for ~1 minute. Must produce: a checkpoint the benchmark can load AND a non-zero eval on a held-out item. If not, the recipe is broken — fix it, don't scale it. dtype mismatch, tokenizer/template drift, OOM at this batch size, empty artifacts dir despite falling loss — all surface on 10 examples. Running longer doesn't surface them differently, just more expensively.

Long training: checkpoint, mid-eval, early-stop in-script

Training for an hour and getting one number at the end is the wrong granularity for evo's tree search. By the time you know the recipe failed, you've spent the budget. Build the verification into the training script, not around it.

Pattern for any training run expected to exceed ~30 min wall-clock:

1. Periodic checkpoint every N steps (e.g. every 0.25 epoch, or every 200 steps — whichever is faster).
2. Mini-eval after each checkpoint on a small held-out subset (5–10 items, not the full held-out — that's reserved for the final committed score). Same scorer as the real eval; the model just sees fewer items.
3. Early-stop on regression: track best mid-eval score; stop if it hasn't improved in patience checkpoints (typically 2). Don't burn 60 more minutes once the trajectory has flattened or reverted.
4. Save the BEST checkpoint, not the last. Early-stop means the current model is probably past its peak; the checkpoint you commit should be the one that scored highest mid-training, not whatever the trainer happened to leave behind.
5. Log every mid-eval score to your tracker (see ## Stream training metrics live). The user watching the live dashboard sees the trajectory build up step-by-step instead of staring at the loss curve hoping it transfers.

HuggingFace TRL: implement as a TrainerCallback on on_step_end — save checkpoint, run the mini-eval via vLLM or HF transformers, compare to best_score, set control.should_training_stop = True on stall. Pattern is one ~30-line class.

Keep vLLM warm across mid-evals when you can (one serve process, reload adapter between checkpoints) — cold-starting vLLM every 200 steps adds 5 min of overhead per checkpoint.

Use a tighter mini-eval subset than the full held-out. The mini-eval is a signal, not the score that gets committed. If the mini-eval scores ≥ baseline on its subset, run the full held-out as the eval-gate scoring pass at the end. If it doesn't, early-stop.

This is Pattern B from the design tradeoff with multi-node staging (Pattern A — break the training into multiple committed evo nodes, each a stage). Pattern B keeps the experiment as one evo node with the verification logic inside the script; it's simpler to write and avoids per-stage vLLM spin-up, at the cost of less tree-search introspection. Multi-stage as separate nodes is preferable when you want the orchestrator to be able to branch alternative continuations from any mid-training checkpoint.

Cap retries at training scale

evo run allows up to max_attempts=3 retries per experiment by default. That budget was designed for second-scale benchmarks where retrying after an edit-bug fix is free. At training scale (~hours per attempt), it's the wrong tradeoff — by attempt 2 you've spent more compute than just trying a fresh hypothesis would cost.

For training-heavy workspaces, set the cap to 1 once at init:

evo config set max-attempts 1

One attempt, one shot. Regression → evo discard → new branch from parent with a different hypothesis. This pairs with the in-script early-stop above: each attempt is single-shot, but its internal verification keeps it from burning the budget on a clearly-failing trajectory.

The "fix-and-rerun" retry pattern still applies for sub-minute benchmarks; leave the default max_attempts=3 there.

Four diagnostics

Stuck at 0 on a verifiable benchmark after 2+ SFT runs. Technique class is wrong, not the recipe. Pivot to RL with the verifier as reward; SFT loss can be healthy while the model emits unparseable output.

Base scores below random before any training (knowledge-heavy benchmark). Model lacks the knowledge, not the format. Post-training shapes existing knowledge; it does not install new knowledge. Right axis: continued pre-training on a domain corpus, distillation from a stronger model that has the knowledge, or retrieval-augmented inference.

delta <= 0 across several committed train moves. Method exhausted on this target. Try a different method, change the data, or improve the harness instead of the weights.

Stuck at the same non-zero score across 3+ experiments spanning distinct techniques. When 3+ committed experiments — across structurally different techniques (e.g. SFT, GRPO, RFT) — all land at the same non-zero score, the bottleneck is not the training method. The most common cause is a train↔verifier objective mismatch: the model has learned to emit answers in one format, but the verifier expects a different one. Examples: training data uses \boxed{X} but the verifier prompt requests ANSWER: X (or vice versa); training uses one chat template, eval uses another; training optimizes step-by-step CoT but the verifier wants the answer alone.

Diagnostic action: spot-check 3 training examples and 3 eval-prompt examples side by side. If a perfect-score training example would NOT pass the verifier (or vice versa), the objective is mismatched. Realign the training data format to the verifier's expected output, OR change the eval prompt (if rules allow). Do NOT try a fourth training-technique variant before doing this spot-check.

What never counts as progress

Five patterns produce a number going up without the model improving. See references/false-progress.md for examples + detection.

1. Training on the held-out set — direct or transitive (public instruction datasets sometimes contain eval-derived items).
2. Embedding eval items in "synthetic" data, even renamed or paraphrased.
3. Generating training data conditioned on per-eval-item failure logs.
4. Submitting a checkpoint you didn't train (off-the-shelf instruct model; parent's checkpoint unchanged).
5. Training a different objective than the verifier scores.

The verifier should catch these. List is here so the train move doesn't produce them.