Agent Eval Cases

Agent Eval Cases

Overview

A case is one task you give the agent, plus the graders that decide whether what came back was acceptable.

That is the whole shape — and note what it is not. It is not an input paired with an expected output. There is no single correct answer string to compare against, which is why the right-hand side is a list of graders rather than a value. Many frameworks offer an expected / expected_output field; reaching for it by default is the most common way to write a suite that measures phrasing instead of behavior.

Three consequences shape everything below:

• A grader can assert on what the agent said or on what the agent did — which tools it called, what is in the store afterwards, whether a gate fired. The second kind is usually the stronger one.
• Graders see only what the run recorded. That record — the final answer, the tool calls, what each returned, what a guardrail stopped — is called the capture here; your harness may have no name for it at all. It bounds every assertion you can write, so find out what is in it before designing a case around one.
• A case is not passed or failed by a person reading it, so whatever you want to be true has to be expressible as code (or as a rubric a model can apply, or — occasionally — as a human's read).

The hard part is not the format. It is knowing which handful of tasks are worth paying a model to run, repeatedly, forever.

The workflow

Steps 1–3 are the ones that decide whether a suite is worth having. Do not skip to step 5.

Step 0 — Confirm where these will run

A case needs something to run it on. Before writing any, find out what exists.

Look for an existing suite first. Search for an eval/evals/evaluation directory, a case or dataset type, *.eval.* files, or a framework dependency. If one exists, write cases in its idiom — its case type, its grader catalog, its repeat convention — and stop looking. See references/vocabulary.md.

If there is none, do not build one unprompted. Name the three options and let the builder choose:

Never scaffold a harness without explicit confirmation. Building one is a separate project with its own design decisions — what to capture, where it runs, how it resets. If the builder wants that, say so and get agreement first.

Hand-grading is not a lesser option to be talked out of. For a small agent it is the right answer, and it stays right for longer than people expect.

Step 1 — Elicit observed failures. Never invent them

You cannot reason your way to a good case list from an empty page, and you must not try. A suite invented at a desk tests the failures someone was able to imagine. That is the wrong set.

Three sources, best first:

• Production traces. Real failures, real phrasings, no imagination required. The best source there is — and the honest reason tracing comes before evals: you cannot distill cases from traffic you never recorded.
• The builder, using their own agent. What makes them wince. What they have caught and corrected by hand.
• Someone who knows the domain. A domain expert who has never seen the code can usually list the expensive edge cases within minutes, because they know which wrong answers cost money.

Ask; do not guess. If the builder cannot name a failure they have actually observed, say that plainly and offer a route to generate observations — a short hand-run probing session, or the domain-expert conversation — rather than inventing a suite that will look thorough and test nothing.

The question set, the triage, and what to do when there is nothing to go on are in references/elicitation.md.

Step 2 — Push each failure down before it earns a case

This is the filter, and it is the step most likely to be skipped.

Observe a failure → push it to the cheapest layer that can catch it → let evals inherit only what will not fit.

A failure that a unit test could have caught is a failure you will pay to re-detect on every run, forever. Work down the layers:

1. Is the payload ambiguous? The fix belongs in the payload. An empty result that means two different things — "this filter matched nothing" and "the thing you filtered on does not exist" — forces the agent to guess, and no eval case repairs that, because it was reasoning correctly over a misleading input. Same for a page that is silently capped rather than marked partial. This is tool-design territory.
2. Is it mechanically checkable? It belongs in the ordinary test suite — a unit test over the tool, or an integration test where real I/O decides correctness. Truncation signalled, errors structured, promised fields present, defaults applied: all provable without a model, in milliseconds.
3. What is left is an eval case. Typically: the agent had correct information and used it wrongly, or chose the wrong tool, or ignored an instruction — and only sometimes, which is the part no assertion over a return value can reach.

This filter assumes layer 2 exists. Check that it does. If the project has no tests over its tools, say so plainly — pushing a failure down to a layer that is not there means nothing catches it, and the case you were about to skip becomes the only guard. Two consequences worth stating to the builder:

• Ordinary tests are the cheaper place to start, and they are outside this skill's scope. A unit test over a tool runs in milliseconds, on every change, for free; an eval case costs a model call every run, forever. A project with neither should usually write those first.
• Until they exist, some cases will duplicate them. That is the right call in the moment — an eval that catches a payload bug beats nothing catching it — but flag those cases, because they are the first to retire once the cheaper layer lands.

Applying the fix and the case together is the honest sequence: a case has to be able to fail for the right reason before the defect underneath it is worth chasing.

Report what belongs at layers 1 and 2; do not go and build it. Those are changes to tool code and test suites, outside a request to write evals — and a payload fix changes the behavior every existing case was written against. Name the failure, say which layer should catch it and why, and let the builder decide whether to take it now, later, or not at all. Carry on writing cases for what is left.

Step 3 — Group by defect class, not by feature

What survives step 2 is a defect class — a way the agent can be wrong — not a feature. Grouping by feature produces one case per capability: expensive, slow, and mostly redundant with tests that already exist.

Defect classes look like this (examples of the shape, not a checklist to fill):

The right set for your agent is whatever step 2 left you, and it will not look like this one. A support agent groups around escalation and scope; a coding agent around destructive edits and test-passing-by-deletion. The axis is the defect, whatever the domain.

Keep the suite small. Five to ten cases is a normal, healthy first suite — start at the low end, since every case is one you pay for on every run. The anti-pattern is believing you need a big one before you start.

Step 4 — Pair every case that pushes with one that bounds

A case asserting "flag old sources as dated" invites a prompt fix that over-rotates into caveating everything — every answer hedged, the agent measurably worse, and the suite still green.

So each case that pushes a behavior gets a partner that bounds it. Here, that partner asserts a recent source is answered with no hedge. Where one case asserts a total is computed by aggregating, its partner asserts a plain listing still uses the listing tool.

Any rule that makes an agent do something needs a case proving it does not do it everywhere. An agent that qualifies every answer is worse than one that occasionally misses — and without the bounding case, that regression looks like total success.

Step 5 — Choose graders

First ask whether you can assert on what the agent did — a tool it called, a gate that fired, what the store holds afterwards. Those are structural facts with one spelling, and they often turn a judge-shaped target into a deterministic one: "asked before acting" needs a rubric, but the gate fired and the store is unchanged is two exact checks, and a stronger claim than any rubric would make.

For what is genuinely left, one question decides most of it:

Is the assertion target atomic?

A number, a tool name, a record count, a URL, an identifier has one spelling → a deterministic check is exact. A claim — "flagged the source as dated", "answered without hedging" — has combinatorially many correct spellings → it needs a judge.

Reach for a judge when the target genuinely has many correct spellings, and not before. Every judge is a second model call, a second thing that can be wrong, and a second thing to debug when a case goes red for no reason.

Having decided you need one, which judge is a second decision — judge calls are usually the dominant cost of a run, and a judge too small for the question produces false reds that look exactly like agent regressions.

Grader shapes, vacuous passes, rubric design, choosing and tiering the judge, and when to stop automating a case are in references/grading.md.