Output Eval Validate Judge

Validating LLM Judges

Overview

An LLM judge is only useful if it agrees with human judgment. This skill walks you through calibrating a judge against human-labeled data using True Positive Rate (TPR) and True Negative Rate (TNR) metrics. Do this before trusting any judgeVerdict(), judgeScore(), or judgeLabel() evaluator in your eval suite.

Prerequisites

1. A judge .prompt file — Written following output-eval-judge-prompt
2. ~100 human-labeled traces — With binary pass/fail labels for the failure mode this judge targets. Aim for ~50 pass and ~50 fail. Minimum: 20 pass and 20 fail.
3. Labels stored in dataset YAML — Each dataset has ground_truth.evals.<evaluator_name>.verdict: pass or fail

This process applies only to LLM-based judges. For code-based Verdict.* evaluators, write unit tests instead.

Step 1: Create Data Splits

Split your labeled datasets into three groups:

Organizing splits

Use a naming convention or subdirectories to separate splits:

Option A: Name prefixes

tests/datasets/
├── train_formal_pass_01.yml
├── train_casual_fail_01.yml
├── dev_technical_pass_01.yml
├── dev_ambiguous_fail_01.yml
├── test_simple_pass_01.yml
├── test_contradictory_fail_01.yml
└── ...

Option B: Subdirectories

tests/datasets/
├── train/
│   ├── formal_pass_01.yml
│   └── casual_fail_01.yml
├── dev/
│   ├── technical_pass_01.yml
│   └── ambiguous_fail_01.yml
└── test/
    ├── simple_pass_01.yml
    └── contradictory_fail_01.yml

Splitting rules

• Balance pass/fail in each split — Don't put all failures in dev and all passes in test
• Randomize — Don't sort by difficulty or topic
• Training examples in the prompt — Use only train-split examples as few-shot in the judge .prompt file. Never use dev or test examples — that's data leakage
• Lock the test split — Once created, do not look at test data until final measurement

Step 2: Run the Judge on Dev Set

Execute the eval workflow against only the dev-split datasets:

# Run with cached output on dev datasets
npx output workflow test <workflowName> --cached \
  --dataset dev_technical_pass_01,dev_ambiguous_fail_01,dev_formal_pass_02,...

Or if using subdirectories, list the dev dataset names:

npx output workflow test <workflowName> --cached \
  --dataset $(ls tests/datasets/dev/ | sed 's/.yml//' | tr '\n' ',')

Save the output. You need the judge's verdict for each dataset to compare against ground truth.

Extracting results

Use --json to get machine-readable results:

npx output workflow test <workflowName> --cached --dataset <dev_datasets> --json

The output includes per-dataset, per-evaluator verdicts that you can compare against ground_truth.evals.<evaluator_name>.verdict.

Step 3: Compute TPR and TNR

For the evaluator you're validating, build a confusion matrix from the dev results.

Definitions

Using "fail" as the positive class (what you're trying to detect):

TPR (True Positive Rate) = TP / (TP + FN)

• "Of all the real failures, what fraction did the judge catch?"
• Low TPR means the judge misses real failures (dangerous)

TNR (True Negative Rate) = TN / (TN + FP)

• "Of all the real passes, what fraction did the judge correctly approve?"
• Low TNR means the judge flags passing traces as failures (noisy)

Example computation

Dev set results for check_tone evaluator (42 datasets):

• TPR = 18 / (18 + 3) = 85.7%
• TNR = 19 / (19 + 2) = 90.5%

Why not raw accuracy?

Raw accuracy = (TP + TN) / total = (18 + 19) / 42 = 88.1%

This looks fine, but masks problems. If your dataset were 90% pass (class imbalance), a judge that always says "pass" would get 90% accuracy while catching zero failures (TPR = 0%). TPR and TNR measure what actually matters: catching failures and not crying wolf.

Step 4: Inspect Disagreements

For every case where the judge disagrees with the human label, determine the root cause.

False Negatives (judge missed a real failure)

The judge said "pass" but the human said "fail." For each:

1. Read the trace and the judge's critique
2. Determine why the judge missed it:
   • Criterion too narrow — The prompt defines failure too narrowly. Broaden the fail definition.
   • Missing few-shot example — The failure pattern isn't represented in examples. Add a similar borderline example from the train split.
   • Insufficient context — The judge doesn't have the information needed to detect this failure. Add the missing variable to the prompt.

False Positives (judge flagged a passing trace)

The judge said "fail" but the human said "pass." For each:

1. Read the trace and the judge's critique
2. Determine why the judge flagged it:
   • Criterion too broad — The prompt defines failure too broadly. Tighten the fail definition.
   • Misleading few-shot example — A borderline example is being overgeneralized. Clarify or replace it.
   • Overly strict — The judge applies the criterion more strictly than intended. Add explicit exceptions to the prompt.