Eval Import

Import questions into an eval set

Questions arrive in whatever shape their author had. This turns them into evals/<set>/, ready for a run, and records what each key is actually worth.

Scope boundary: cases only. This never answers a question, scores an answer, or edits a model. It is skill:eval-loop's scrape step, expanded, and it is where the set's honesty is decided.

The file shapes are defined once, in reference/ledger-schema.md of skill:eval-answer. Read them there; this skill does not restate them. reference/case-format.md covers what is specific to an import: which fields an arriving item maps to, and the four physical formats.

Where the set goes

Two separate rules, and "package" blurs them:

• The same git repository as the model it evaluates, so one commit pins the model and the ledger together. That is what a checkpoint is.
• Never inside the directory tree the answerer's package serves. The answerer can read files in the served tree, and gold there is a contamination path.

So the set is a SIBLING of the package under test, in its repo. A repo with ecommerce/ under test gets evals/ecommerce-questions/, beside it.

Step 1: keep the file as it arrived

Copy it verbatim into evals/<set>/as-received/, and never edit that copy. Not the encoding, not the line endings, not an obvious typo.

It is there so a later disagreement is about the key and not about whether somebody transcribed the question correctly. That disagreement is guaranteed: skill:eval-loop's reference/auditing-an-answer-key.md is about what happens when it arrives, and its first check is the question against this file.

If the source cannot be committed, because it carries names, addresses, internal URLs, or row values somebody pasted into a thread, say so. Record in set.json where the original lives and a hash of it, note under sourceNote that the copy is withheld and why, and tell the user that the byte-for-byte check is now impossible for anyone without the original. Do not redact silently: a redaction you got wrong becomes the provenance record.

Step 2: classify each item by what arrived WITH the question

This is the whole job, and getting it wrong is how an unverified number gets scored as a fact.

Two rules over the whole table:

No golden holding a VALUE imports as verified. verified means two differently shaped derivations agreed through the truth package, and an import has performed neither. provisional goldens are unscorable by design: skill:eval-answer issues verdict: null on them. That is not a gap to work around.

It is also not a dead end, and it used to read like one. The way out is verify_goldens.py --promote, which marks a provisional golden verified once its value re-derives cleanly from the truth package and a second derivation exists. It is the only thing anywhere that writes golden.status. Tell the user that command when you hand over a set of provisionals, because until it is run the set scores nothing. A set that arrives with 60 typed numbers genuinely has 60 unverified numbers, and the number it can honestly produce on day one is a coverage score.

The exception is a golden that holds no value at all, and there are two kinds. Prose criteria (kind: criteria, step 4) have nothing to re-derive: the author's words ARE the key. So does a case whose stated answer is that the data is not there (kind: unanswerable), where the pass is a refusal that names what is missing and a confident number is the failure. Both import verified with verifiedBy: authored_criteria and score immediately. The rule is about numbers, because a number is the thing that can be wrong while looking right.

Watch for the second one in what arrives. A criterion reading "a refusal that names the missing data" is not a rubric clause on a normal case, it is the whole key, and importing it as criteria on a case with a value would make a refusal fail.

A question with no golden is a case, not a reject. Do not hold questions back until somebody derives keys. A set of bare questions already measures whether the model can express an answer at all, and the answers it produces are what the keys get derived from.

Deriving a key yourself, when nothing arrived with the question

The table above is about what ARRIVED. Sometimes nothing did, nobody knows the right number, and the set still needs keys. You may derive one, and the rules are the ones already stated rather than new ones:

• Use your full model access and query until you can defend the number. The agent that gets tested later holds only get_context and execute_query; you are not that agent, and the gap is the point.

• Record the query as canonicalQuery. A number with no query is somebody's guess, which is the row above it in the table.

• Record the entities you used to get there as expectedEntities.required, in the kind:source:name form: the measures, dimensions and views the answer cannot be produced without. reference/case-format.md says to leave this field empty at import because guessing it invents a retrieval expectation nobody stated; you are not guessing, you just used them. That is the only trustworthy producer this field has, and it is what makes retrieval measurable on a set nobody sent keys for. Where the model offers two legitimate routes, write a requiredAnyOf group rather than pick one. verify_goldens.py check 5 audits every id against the model, so a wrong one is a hard finding rather than a silent retrieval miss.

• Derive the list with the model in front of you, not from memory, and not with grep. The model knows what each field IS; an author does not reliably. Ask the question of the model: which entities does an answer to this depend on, and is each a measure or a dimension. Confirm every id with get_context before writing it. A list written from recollection is the most common way a set acquires a retrieval expectation nobody can satisfy.

  Measured: an agent given the model and one question found alternatives a hand-written key had missed on all three cases tried -- carrier name against nickname against code, destination_count against its underlying destination.airport_count. At roughly $0.13 a question. It is better than a person working from notes at finding what the model actually offers.

  It is worse at deciding what to REQUIRE, and the two rules below are why. On the same three cases it proposed demoting two named measures on the grounds that a raw expression answered identically. One of those equivalences was false; the other was true today and is not a reason to demote anything.

• But VERIFY every "this is not strictly required" claim by running both sides. This is the rule that makes the step above safe, and it is not optional.

  The same probe got one badly wrong. Told that a named measure is not required when a raw column plus a plain aggregate answers identically, it wrote that average_plane_size was "equally answerable by avg(aircraft.aircraft_models.seats)". Executed, those are 229 and 196.93. The measure is aircraft.avg(...), an average over distinct aircraft, and the inline version averages over flights. The claim was false for exactly the measure whose entire purpose is to carry that distinction -- and it is the same mistake, on the same field, that produced the only wrong answer in the run this set came from.

  Had that proposal been written into the key, the case would have stopped testing the thing it exists to test.

  So: an equivalence is a claim about DATA, and reading a definition is not evidence. Run both expressions and compare the rows.

  But agreeing today is not a reason to demote the model's entity, and this is the part the probe got backwards. airport_count is count() returns the same 984 as a bare count() right now, and stops doing so the moment anyone adds a null filter or a deduplication to it. The key would not notice: it would go on passing while testing nothing. That applies to every measure that is currently a thin wrapper, which is most of them until the day one is not.

  requiredAnyOf is for two routes the MODEL defines -- a carrier's name and its nickname, destination_count and the destination.airport_count it wraps. Both are entities, both are retrievable, and an answer using either has gone through the semantic layer. That is a real choice and the group records it.

  It is NOT for "the model's entity, or a raw expression that happens to agree". Those are not two routes: one is the route, the other is a bypass that currently works. Keeping the named entity in required is what makes the eval measure what the semantic layer is for, and finding it IS the retrieval behaviour under test -- an agent that searches for the pre-defined entity inherits its definition and its rules, while one that rebuilds from raw columns inherits whatever it thought of.

  What the equivalence check is FOR is reading a run afterwards. When an answer came out right without the entity, comparing the two says whether it got lucky or got it wrong, and a set whose passes are merely lucky is a set of near-misses.

• Then prove each one is retrievable, and record what finds it. check_findable.py --set <set> --mcp-url <mcp> --publisher <rest> --environment <env> --package <pkg> runs two checks and needs no model call. With --publisher it reads the COMPILED model and settles whether each field exists and what KIND it is; then it searches for each entity to confirm the index can actually deliver it.

  This matters more than it looks, because required is what BOTH retrieval numbers are computed against: whether the agent asked for the entity, and whether it came back. An id retrieval cannot deliver makes both fiction, and the case reports a retrieval miss on every run, which reads as a defect in the model or the agent rather than in the key.

  Prefer the compiled model over a grep, and prefer it in both directions. Measured on one package: dimension:flights:flight_count names a field the model really has, so grepping the text passes it, and the compiled model says flight_count is a MEASURE, which no dimension request can return. Conversely dimension:airports:own_type appears zero times in the .malloy and the compiled model declares it, because the source exposes it implicitly from the data; the grep calls it missing and acting on that deletes a good entity. verify_goldens.py check 5 is the grep, it is free and needs no server, and it is not the authority.

  A pass is a floor, not a verdict on the docs: each entity is searched by its OWN NAME, the easiest query that could find it. An entity that answers to its identifier and not to the words a question uses still fails at run time, as not retrieved.

  Writing the list forces the question an author has to answer anyway: what would a reasonable agent search for, and of what type, to find this? If you cannot state that, the agent cannot be expected to guess it.

• status: provisional, never verified, for the reason stated above: you derived it THROUGH the model under test, so a model bug would certify its own key. verify_goldens.py --promote is still the only way out.

• Where two readings are both defensible from the model, do not pick one. A model that documents two conventions for the same population produces two honest numbers, and choosing quietly is how a confident wrong key gets written. Go to Hold an ambiguous golden in skill:eval-loop's reference/golden-side-door.md: record the competing candidates rather than a new number.

• What no query can settle is not a failure to derive. It is step 4's second kind, and those clauses score on day one.

Step 3: run their query, if they gave one

Do it at import, before any run. It is the cheapest finding in the whole loop.

Execute their query with execute_query against the model under test and compare with the number they stated. Three outcomes, three different things learned:

• It returns their number. verifiedBy: authored_query. Keep status: provisional: this proves the number came from that query, and NOT that the query is right. It ran against the model under test, so a model bug certifies its own golden, which is the exact circularity the truth package exists to break. It is still far more than a typed number is worth.
• It returns a different number. A finding on day one, before a single answer was scored. Record BOTH numbers on the case and leave the status provisional. Do not quietly adopt either one: their number may be stale, the model may have moved, or the query may have always been wrong, and which of those it is decides who owns the fix.
• It does not compile. Also a finding, and usually the most informative one: their query names entities this model version does not have. Record the error. This is what a coverage gap looks like before anyone has phrased a question about it.

Never repair their query to make it run. A query that does not compile against this model is evidence about this model.