Eval Loop

Before you start

1. The model package under evaluation must live in a git repository, with evals/<set>/ in the package, beside the model files. Git is the checkpoint mechanism; without it there is no rollback and no run can include improve.

   Keeping the set IN the package is what stops a model edit and its answer key drifting apart: they move in one commit, so fixing a measure and forgetting the golden that depended on it stops being possible. It is safe -- measured on a running server, a cases.jsonl inside a package appears in no model listing, no notebook listing, no package resource, and 404s over HTTP, so an MCP-only answerer has no route to it.

   What it buys differs by target. On a LOCAL Publisher it does not get you free versioning -- sourceContentSha hashes model paths only, so the set needs its own datasetSha. On a hosted target that publishes the whole package directory as an IMMUTABLE version, the set rides inside that version and targetVersion pins model and answer key together; nothing can be edited under a published version, which is what makes it a pin. Check which you have before deciding how much of this you need.

   Look for a set and for prior runs before you author either. A minute of find . -name cases.jsonl, a glance at the set's workdir (<workdir>/runs/, ~/.malloy-eval/<set>-<hash>/runs/ unless eval.toml moves it) and at your host's own transcripts for this repo. Two sessions fourteen minutes apart built the same 29-case answer key from scratch, because the first had committed nothing before it was deleted and the second had no way to know it existed. Roughly a working day was spent twice, and three specific things were rediscovered at cost: the MCP login flow, the retrieval gate's 401, and a judge-rendering bug that had already cost $17 of arm once. The two keys, independently authored from the same questions, differ by about ten points on comparable answers -- which is the available measure of how much a key depends on its author, and a reason to reuse one rather than rebuild it.

   Audit the set's entity ids before the first arm. verify_goldens.py checks that each id names something in the model; check_findable.py checks that a search of its own kind actually returns it. Both are free of model calls. An id that fails either scores a retrieval miss on every run, and the miss reads as the model's fault.

   Commit the set before you spend money on an arm, and keep durable outputs in the repository. A findings document in ~/Downloads is gone the first time somebody tidies up; the set and the write-up belong in git beside the model.

   Runs go in the set's workdir, never inside the model package. A run directory holds a model.malloy snapshot, and a built report is a Malloy package; inside the package under test, either puts that package into loadErrors. The default workdir, ~/.malloy-eval/<set>-<hash>/, is outside git, which suits a measure-only run. A run that will improve and checkpoint needs its ledger kept: set [paths] workdir in eval.toml to a directory in the model's repository but outside the package, and gitignore its servers/ and packages/, which are a server's database and rebuilt reports.

2. The server must be up. Where your host offers retrieval tracing, turn it on and confirm a trace lookup answers, so a call's ranked results can be recovered afterwards.

   Open-source Publisher has no trace lookup. PUBLISHER_MCP_TRACE=retrieval is set by serve.py and makes Publisher write one "Retrieval trace" line per ranked call to its log (stage counts and timings, not the ranked results). Nothing reads that line yet: there is no trace store and no trace tool, so traceId is null on every local attempt. This does not block a scored run, because attribution never depended on it: rankedSummary is copied onto the tool_call event at capture, precisely so the evidence survives without a store. So do not refuse a local run for want of tracing -- an earlier version of this rule did, and it refused every local run there has ever been. Refuse one whose tool_call events carry no rankedSummary, which is what attribution actually reads.

3. Health-check: your host's status check until it reports serving, and inspect loadErrors. A dead database that still answers HTTP is an environment failure, not a model failure. Stop and fix it. Four consecutive environment or no-result attempts means stop the run.

4. Load the set: scrape/import as above, or reuse an existing evals/<set>/. eval.py check --set <set> names every gap in it before anything starts. Never keep two live copies of one set; the set directory in the model repo is the single source of truth, versioned by datasetVersion in set.json.

5. Review goldens before you score. Check how many cases can take a verdict at all, not just how many cases there are: a golden the set stamps provisional, invalid or ambiguous, or a case with no golden, scores verdict: null and stays out of the pass rate. An imported set is provisional throughout by design (skill:eval-import), and the only thing that changes that is verify_goldens.py --promote after a re-derivation through the truth package. A run whose every key is underived is refused rather than spent; a set of bare questions runs, because its answers are what keys get derived from. A verified golden that holds a value but no local artifact stays verified by provenance and is not scorable until you have rows or a scalar to compare (the judge needs both sides). A criteria golden is the exception: it holds no value, so its clauses are both sides. If diagnosis later marks BAD-REFERENCE or AMBIGUOUS-REFERENCE, follow reference/golden-side-door.md. Both are expected in the wild; both are the golden side door below, not improve, and not a sixth step.

6. eval.py run creates <workdir>/runs/<runId>/run.json with the attribution pins (reference/ledger-schema.md): mode, dataset version, the target and the version it served (a local target pins a commit, so commit or stash first; answering from a dirty tree pins nothing), server version, judge version and rubric sha, answerer model, call budget, trace mode. Freeze those for the whole run. Raising a call budget mid-run moved mean outcomes on an unchanged model.

   The call budget is --max-turns, written to run.json as maxTurns. Size it from a pilot rather than taking the default of 30. Run the three cheapest cases uncapped (--max-turns 100) and set the cap at twice their maximum. A set whose questions need two or three sources joined does not fit a cap sized for single-source lookups: on one such set the completed attempts had a median of 17 turns and a 90th percentile of 26 against a cap of 30, and four cases died at it. A cap 15% above the 90th percentile of completed work is not a safety margin. Note also that malloy-analysis tells the answerer to persist -- retry a phrasing, let a small query settle whether a field exists -- so a tight cap and that instruction are in direct conflict.

   Do not change the model and the measuring instrument in the same step. Those are two different axes and only one of them is cheap to separate.

   Batching MODEL edits is fine and expected. Clustering exists so that one edit closes several cases, and an arm per fix does not survive contact with arithmetic: 20 fixes over 100 cases is 2,000 answers, and at the measured $0.33 a case on a proxied warehouse that is $660 of answering to attribute what the acceptance check attributes for the price of the affected cases plus holdout. Budget five arms for a defensible claim (a baseline, two for the A/A, two post-edit), not one per edit, and let skill:eval-improve carry each cluster.

   The instrument is the other axis: the answer key, the judge and its prompt, the answerer model, the skills. Move one of those together with the model and there is nothing left holding still, so the result measures neither. On the run this comes from, nine model commits and a re-derived answer key landed between two arms, and the move from 20% to 39% belongs to no one -- not because two model edits were batched, but because the ruler changed at the same time as the thing being measured. A key repair mid-improve is not forbidden; it ends that comparison, so re-baseline rather than quoting a delta across it.

   The answerer model is instrument too, and it is the one most often left unstated. The same 23-case set read 12 match / 11 near / 0 no_match on one model and 8 / 4 / 15 on a smaller one. No statement about "the agent's" capability means anything until two arms name the same answerer.

7. Generate every answerer prompt from the stored case in cases.jsonl. Never retype the question. A truncated retype is indistinguishable from a real question downstream.

Per question

1. Health-check again.
2. Spawn a fresh blind subagent. Give it only the question text and the Malloy analysis tools. Tell it to follow the malloy-analysis skill. Do not mention eval, gold, scoring, or this skill.
3. Keep a host-side tool-use log for that subagent (name, input path or command, MCP tool name). Publisher traces see MCP only; a Read of a gold CSV is invisible server-side.
4. skill:eval-answer: contamination first, then re-execute, then the judge, then events.
5. skill:eval-diagnose only when this run includes diagnose, only on dev cases, and only after the score event exists.
6. skill:eval-improve only when this run includes improve, and only for owner: model. Then run the acceptance check. On accept, checkpoint.

Report the run, or nobody can read it

A run directory is JSONL. It is a record, not a result, and the console summary scrolls away. Every run ends by producing something a person can open, and that job is skill:eval-report: it builds the servable run package (the case matrix app and the aggregate notebook) and gives the template for the write-up.

Read it at the end of every run, including a run that failed. The standing complaint about this loop is that "a bunch of stuff happens and it is hard to know the actual results", and a ledger nobody renders is why.

Two rules from it are worth repeating here, because they are the ones a conductor skips:

• Separate a MODEL failure from an EVAL failure. A wrong answer and a broken measurement look identical in a pass rate and have nothing else in common. An arm holding a truncated, contaminated or environment-failed attempt has no rate to quote at all.
• Say when a step did not run, and why. improve not running because every cluster came back owner: agent-skill is a RESULT, and it reads identically to having forgotten unless it is written down.

Keep the write-up in the repository beside the set, not in a chat log and not in ~/Downloads.

Checkpoint

A checkpoint is a git commit of the model repository, taken after an acceptance check accepts, so a bad improve direction can be rolled back. It is not a report, and it is not a remote publish.

1. Commit the model files AND the set's ledger in one commit; put the label and the closed issue ids in the message. The run's ledger is in that commit only when the workdir is in the repository (Before you start, item 1).
2. Append the checkpoint event (action: created, label, modelGitSha from the commit you just made, issueIds). The event line itself rides in the next commit; append-only logs trail by one commit and that is fine.
3. Confirm git status is clean for the model files.

Restore: git checkout <sha> -- <model files> (or git revert the checkpoint commits), then reload the package, then append a checkpoint event with action: restored and the sha. Readers return to the model that existed before the bad direction.

Take a checkpoint of the current model before the first improve batch if no commit pins it yet. Rolling back by hand is guesswork.

If reload reports mode: reinstalled, the package was re-fetched from its install location and may have overwritten the restored files. Prefer in-place / watch-mounted packages for this loop.

Out of scope

This loop is local. The ledger is files, the checkpoints are git, you are the conductor. Do not:

• publish the model to a hosted platform as a "true" checkpoint or learning curve
• start a Python orchestrator (loop.py, run_all.py, improve_batch.py) that runs the five steps end to end unattended. You conduct; the scripts are the steps, not the sequencing. There are more than twenty of them and they are not the exception to this: each does one step you invoke and hands back a result you read. What is forbidden is a script that decides what to do next, because every judgement this loop protects lives in that decision
• score by string-diffing rows instead of judging them, or reintroduce a scripted row oracle: one that can pass a wrong answer is worse than none
• wait for a bigger gold set before the loop can run; dev/holdout on what exists beats waiting
• register eval MCP tools or stand up an eval API
• encode unsettled goldens into the model

Prime directives

• The model is the only thing improve edits. No question text, qids, or expected values in any name, doc, or comment.
• You are measuring a model, not reviewing this harness. Read a script when a number you have to report cannot be explained otherwise, and stop there. Do not audit the scripts, propose fixes to them, or hand back tooling critique in place of a result -- a run that ends in harness feedback has not answered the question it was asked. A defect that changed a number gets one sentence in the report; a defect that changed nothing gets one line in chat, once. Fixing it is a different task, and the user starts it.
• When the environment misbehaves, stop. Never diagnose a sick system. The harness is part of the environment: a known-broken measurement does not become quotable by being finished. Measured against this directive, an arm already paid for gets quoted anyway -- it happened three times in one run, once with the confound stated in the same message that started the arm -- so the harness now enforces the parts it can. A run with a truncated or contaminated attempt prints no pass rate and records status: incomplete, and flip_table.py names what one arm left unscored that the other did not. Read run_error on the attempts and the INCOMPLETE line before quoting any number.
• When a subagent disagrees with you, probe. Do not win by authority.
• When a rule here is wrong, change this file and note it on the run.

Related skills

• skill:eval-answer: contamination, judge protocol, events. Its reference/ledger-schema.md is the file contract; skill:eval-judge is the judge.
• skill:eval-diagnose: component, owner, issue events. No edit.
• skill:eval-improve: smallest model edit, probe receipts, no self-accept.
• skill:eval-report: the run package and the write-up a person reads.
• The malloy-analysis skill: what the blind answerer follows. It is installed from the analysis manifest group, not the eval group.