Eval Report

The four measurements, and why a report carries all of them

A pass rate says an answer was wrong. It does not say where, and on its own it makes every failure look like the model's. A run measures four things, in this order, and each one tells you whether the next is even a fair question:

Coverage first, and a report without it is incomplete. A model that cannot express an answer can never succeed at that question, so a pass rate quoted without coverage cannot distinguish a bad model from an unanswerable set. run_baseline.py measures it by default; if a run skipped it, say so in "Eval failures" and treat the score as provisional.

Cost is a result, not an aside. Report dollars, the turn and call counts, and the entity payload per answer. An agent that answers correctly in 40 turns and 75 delivered entities per question is a different product from one that does it in 8 and 5, and only the report says which you have.

Report them all, because a report that omits them makes every failure look like the model's:

Copy the cascade the run printed. Do not re-derive it. cascade_lines() in run_baseline.py already renders every rung, and it carries one field a hand-made table keeps losing: how many cases answered correctly anyway after stopping on that rung. Paste its block into the report.

A finer measurement must never revise the headline result. This is the rule the re-derivation breaks. Written as a funnel with a shrinking denominator -- 5 of 10 covered, then 3 of the 5, then 3 of the 3 -- the last rung reads as "only 3 of 10 succeeded" on a run where all ten answers were correct. That report went to a reviewer, and the objection was the right one: adding detail about retrieval cannot turn a 10 of 10 into a 3.

Two things stop it:

• Report every rung over all N cases, never over the survivors of the rung above. The rungs are three measurements of the same cases, not a narrowing of them.
• delivered, right is NOT the pass rate. The passing cases are delivered, right plus passed_not_covered plus passed_not_retrieved. A report that omits those last two has dropped the only numbers that reconcile the cascade with the score.

Label a rung for what it measures, not for what a reader will assume. "Can the model express an answer?" is wrong for most of what lands on the no side of the first rung. RULE_UNWRITTEN means the data is present and the model does not encode the rule for combining or filtering it; AMBIGUOUS means several candidates and no doc saying which the question means. The model can express an answer in both -- it does not say WHICH answer is meant. So the rung asks whether the model NAMES what the question needs.

Give the per-entity number too, and label both.

• per CASE, the cascade: did this case get everything it needed
• per ENTITY, recall: of the N entities the answers depended on, how many were delivered

They answer different questions, and the entity number is the more natural read of "how did search do". One run reported 3 of 5 cases and buried 16 required entities, 80% delivered, and the case number was the only thing a reader saw.

• Covered? Can the model express a correct answer at all? Not computed by the run: check_coverage.py reads the MODEL rather than the answers, and the run consumes its report through --coverage. If it was not run, say unmeasured rather than leaving the row out -- an unmeasured coverage label is not evidence that the model covers the question.
• Retrieved? Of the entities the golden answer depends on, how many did get_context hand back. Quote required_count, delivered_count and how many were ranked rather than merely mentioned in a returned source's docs.
• Correct? The pass rate, which is the rung the other two qualify.

Two numbers here are routinely misread, so qualify them in the report or leave them out:

• Entity precision is a PAYLOAD number, not a retrieval-quality one. Report it as what it measures: how much context the agent was handed against how much it needed. 52 entities returned per attempt for the 2 an answer used is a real cost, in tokens and in attention, and it is worth tracking. What it cannot tell you is whether retrieval worked, because the denominator is everything returned and it ignores rank: an entity that came back at rank 3 of 51 scores identically to one at rank 51.
• For quality, report rank. "Required entities came back at median rank 3, 9 of 16 in the top 5" is a statement about retrieval. Precision alone reads as an indictment of it and is mostly a statement about how many fields the package has.
• Report the misses split by cause, not as one recall figure. The run attributes each to never asked (no target of a type that could return it -- deterministic, because target_type is a hard filter on the server, so no documentation could have delivered it) or not retrieved (a compatible target was issued and it still did not come back, which is the docs or the wording, and diagnose separates them). Those have different owners and one recall number hides which you have.
• A miss on a case that PASSED is still a miss, and the report says so. The answer was right by another route; name the route. Measured on one run, 3 of 4 findings were on passing cases and the route was always the same: the agent rebuilding the model's own measure inline, which held for count() and failed on the one measure carrying a grain rule.
• Read the entities that did NOT come back, and what was asked for. That is where the retrieval signal actually is, and the misses are rarely independent. Measured on one run: 5 required entities never came back as ranked results, and 4 of the 5 had the same cause -- the agent sent only source and dimension targets, and target_type is a hard filter, so no measure could be returned however well the model documents it. One defect, five symptoms, and the same one that produced the run's only wrong answer. A per-case list of misses beside what was asked for would have shown it in a glance.
• A PASSING case with recall below 1.0 is usually an expectation defect, not a retrieval miss: the set named one path to an answer the agent reached by another. Report the count and read it as a prompt to check required.

Query errors are worth a section of their own

Malloy the answerer wrote that would not compile or run, whether or not the case passed. These sit on tool_call events as error, and nothing else surfaces them: a case that errored twice and then recovered scores exactly like one that got it right first time, so the effort disappears.

They are the sharpest available evidence on whether the skills and the docs are leading agents astray, because each one names a specific thing the agent believed and the language does not support. Report the count, the distinct error kinds, and for each whether an existing skill already covers it. An error whose fix IS documented, in a skill the answerer did not open, is a finding about the skills rather than about the agent.

eval-diagnose does not see these today: it reads failing cases only, so an error inside a passing case is invisible to it. Say so rather than implying they were triaged.

Say what did not run, and why

A step that did not run is a result, and it is indistinguishable from having forgotten unless it is written down. The common one: improve does not run when every cluster came back owner: agent-skill or owner: dataset, because skill:eval-improve may only touch owner: model. That means the model is not at fault, which is worth a sentence rather than a silence.

Same for diagnose. If it did not run, no failure in this report has an owner yet, and the report says so rather than implying the failures are the model's.

Findings and improvement ideas

Two different things, kept apart.

• Findings are what this run established, each with the evidence beside it. A finding with no artifact behind it is an opinion.
• Improvement ideas are what somebody might do about them, tagged with who owns each: the model, the answer key, the analysis skills, or the harness. An idea is a proposal, not a decision, and the report does not pretend a cluster has been triaged when it has not.

Rank by how many cases each would move, and say when the answer is "nothing": a 90% run whose one failure is a known fan-out trap may need no action at all.

Report facts, do not instruct the reader. "This set has had one arm, so no noise band exists for it" is a fact they can act on. "Do not quote 90% until X" is a lecture about their own number, and they did not ask for one. State what was and was not measured; what to do with it is theirs.

Make the links clickable

A report whose evidence cannot be opened is a report nobody checks.

• Absolute paths, never relative ones. A reader is not in your working directory, and a terminal only makes an absolute path clickable.
• The served URL for the app and the notebook, with the real host and port the run used, not a placeholder.
• Per case, link the FILES, not the directory: artifacts/<qid>/answer.md, artifacts/<qid>/judge.md, artifacts/<qid>/answerer.jsonl. A file:// link to a directory opens nothing in most editors, which is a dead link that looks live.
• Move the run somewhere durable before you link it. A run written under a session scratch directory (/tmp/..., a host's per-session sandbox) is deleted when the session ends, and many editors will not linkify a path there even while it exists. Copy the run directory, artifacts included, next to the report under the set, then link that. A report whose evidence is in a temp directory is a report with no evidence by next week. This was caught twice by a reader clicking and getting nothing.
• Write a local path bare, as /abs/path/to/file, not as [text](file:///...). Terminals and editors linkify a bare absolute path; a file:// markdown link is frequently inert, and an inert link is worse than the path in plain text because the reader has nothing to copy.
• Never link a run LABEL or a qid as though it were a path. faa-v1-baseline-01 is a label; the path is the run directory. A backticked label that looks like a link and resolves to nothing is worse than plain text, and it has already wasted a reader's time.

Related skills

• skill:eval-loop: conducts the run this reports on.
• skill:eval-answer: the verdicts and the ledger schema this reads.
• skill:eval-diagnose: owners and clusters, when it ran.