Agent Eval

Measure agent quality you can defend and gate on

Turn "the agent feels better" into a number you can put in a PR check. You own the eval dataset, the scorer mix, the LLM-as-judge calibration, and the block-on-regression CI gate — framework-neutral, provider-neutral.

Do NOT use — route instead

The eval anatomy

Every framework instantiates the same five-stage pipeline. Learn it once; the tool is a detail.

dataset ──▶ runner ──▶ scorers ──▶ metrics ──▶ gate
(JSONL    (calls the   (det / judge  (aggregate +  (pass/fail
golden    system per   / human)      bootstrap CI) exit code)
set)      case)

DeepEval, Inspect AI, and promptfoo are all just opinionated wrappers around this. If you understand the stages you can switch tools without relearning the craft.

Build the dataset first

Build your own golden set — a public leaderboard number is not your number, because identical model weights swing SWE-Bench Verified by 10–20 points just by changing the harness. Measure your task on your data. The dataset is the asset; everything else is replaceable. Rules:

• 50–200 hand-labeled cases per failure mode, not per total. Coverage of how the system fails beats raw volume. 80 real failure cases > 1000 generic ones.
• Never synthetic-only. A set the model wrote will not surface the model's blind spots. Mine real traffic / tickets / transcripts and hand-label.
• Version it in git as JSONL, a first-class reviewed asset — same as code. Diffs are reviewable; relabels are auditable.
• Decontaminate. The eval set must not appear in training data or few-shot examples, or the score is a memorization artifact, not a capability.

Case schema — one JSON object per line:

{"id":"refund-001","input":"Where is my refund for order 4821?","expected":"States refunds take 5-7 business days and asks for nothing already on file","context":["policy: refunds 5-7 business days"],"meta":{"failure_mode":"hallucinated_policy","source":"ticket#4821"}}
{"id":"refund-002","input":"Cancel my subscription and refund this month","expected":"Cancels, refunds prorated amount, confirms no future charge","context":["policy: prorated refund on cancel"],"meta":{"failure_mode":"missed_tool_call","source":"ticket#5190"}}

failure_mode in meta is what lets you slice metrics by mode and find which kind of bug regressed — not just that the aggregate dropped.

Bad: "generate 1000 test questions with GPT and use those." Good: "80 real failure-mode cases pulled from support tickets, hand-labeled, tagged by failure mode."

Choose the scorer — the 60/30/10 mix

Reach for the cheapest scorer that correlates with human judgment. Default mix:

One Scorer protocol, two implementations behind it — deterministic and judge are interchangeable to the runner:

from typing import Protocol

class Scorer(Protocol):
    name: str
    def score(self, case: dict, output: str) -> float: ...  # 0.0–1.0

class JsonSchemaScorer:
    name = "schema_valid"
    def score(self, case, output):  # deterministic, free, no drift
        import json
        try:
            json.loads(output)
            return 1.0
        except ValueError:
            return 0.0

class FaithfulnessJudge:
    name = "faithfulness"
    def __init__(self, judge_model): self.judge = judge_model
    def score(self, case, output):  # judge only where meaning matters
        return self.judge.rate(case["context"], output)  # see judge-design.md

LLM-as-judge you can trust

A score you do not trust is worse than no score: an uncalibrated judge gives false confidence, which is more dangerous than admitted ignorance. Each rule, with its why:

• Judge model ≥ system under test. A weaker judge cannot reliably rank a stronger system — it scores noise.
• The rubric must force a written rationale before the score. Rationale-first judging is what pushes judge–human agreement to ~85% — higher than two humans agree with each other. A bare number is a vibe with a decimal point.
• Pairwise beats pointwise for stability. "Is A or B better?" is more reproducible than "rate A from 1–10," which inflates and clusters at 8–9.
• Swap positions and average. Judges favor whichever answer came first; run A-then-B and B-then-A to cancel position bias.
• Calibrate against human gold and report the agreement before you gate anything on the judge. Not a formality — this is the step that makes every number downstream defensible.

Bad judge prompt: "Rate this answer 1–10." → everything lands 8–9, useless. Good: "Compare answer A and answer B against the reference. First write one sentence on each per the rubric, then output the better label." → forces reasoning, gives a stable signal.

Full rubric templates (pointwise + pairwise), the position-swap harness, the calibration script (agreement / Cohen's kappa vs human gold), G-Eval vs DAG, and the judge bias catalog (length, position, self-preference) with mitigations live in references/judge-design.md.

Agent and RAG scorers

Score the path, not only the destination. Beyond exact/judge:

RAG (DeepEval / RAGAS names):

• Faithfulness — does the answer only claim what the retrieved context supports? Catches hallucination.
• Answer relevancy — does it actually address the question, or drift?
• Contextual recall / precision — did retrieval fetch the right chunks, and not bury them in noise? Separates a retrieval bug from a generation bug.

Agent:

• Tool correctness — right tool, right arguments, right order.
• Task completion / goal accuracy — did it finish the job, not just produce plausible text.
• Trajectory scoring — grade the sequence of steps. A correct final answer from a wrong path will fail differently next time; only trajectory scoring catches it.

The system side of these (how the loop and tools are built) is ../building-agents/SKILL.md; a common system-under-test is ../chatbot/SKILL.md.