Agentsop Metric Design

metric-design — Decomposed, Multi-Criteria Metrics for LLM Pipelines

"It's unproductive to launch optimization runs using a poorly designed program or a bad metric." — DSPy core team [dspy.ai/learn/optimization/overview/]

"LLM judges exhibit self-preference, recency, rubric-order, score-ID, and length biases." — Synthesized from [arxiv.org/pdf/2506.02592, arxiv.org/pdf/2509.26072]

This is a tool skill. It produces a metric function (and a calibration receipt) that other skills consume — DSPy compilers (MIPROv2 / GEPA / BootstrapFewShot), LlamaIndex FaithfulnessEvaluator/RelevancyEvaluator/RetrieverEvaluator, LangGraph eval judges, RAGAS, TruLens. The metric is the optimization target. Get it wrong and every downstream optimizer is theatre.

1. 何时激活 (When to Activate)

Activate when any of these is true:

• Optimization runs: a DSPy / OpenAI-Evals / RAGAS / TruLens job is about to consume a metric(example, pred) -> bool|float. The metric drives gradient-free search; bias propagates into the artifact.
• RAG evaluation: deciding chunk size, reranker, hybrid alpha, retriever-k. A bad metric here picks the wrong chunking strategy and you ship it.
• Agent benchmarks: tool-use, multi-step, planning. A single holistic LLM judge cannot distinguish "wrong tool" from "right tool, wrong args".
• Prompt tuning that has gone past 2 manual iterations: if you're tuning a prompt and have no quantitative metric, you are guessing. Stop the prompt edits, write the metric.
• Production regression test: every chunking / embedding / retriever / prompt PR change needs a metric gate (LlamaIndex OP-10 EvalLoop).

Do NOT activate for:

• One-off exploratory prompt tests where no decision rides on the output.
• Tasks where exact-match / unit-test / schema-validity already gives ≥95% of signal — don't over-engineer.
• Where the user explicitly refuses to commit to any evaluation criteria (then dspy-sop will refuse to compile anyway; this skill cannot help).

2. 核心心智模型 (Core Mental Model)

"The metric IS the model. Change the metric, change the behavior."

A DSPy/GEPA/MIPRO optimizer is a black-box search that maximizes metric(pred, example). Whatever the metric rewards, the compiled prompt will produce. If the metric prefers verbose, hedged answers (which an LLM judge will, by default — judges over-prefer length [arxiv.org/pdf/2506.02592]), the optimizer will produce verbose, hedged answers. A bad metric beats a good optimizer every time.

Three corollaries:

1. Decompose by default. A single holistic LLM-as-judge call ("is this answer good? rate 1-5") collapses orthogonal axes (factuality, tone, length, relevance) into one noisy scalar. Decompose into N orthogonal yes/no sub-judges, then aggregate. Same number of LM calls in the limit, vastly less noise.
2. Bool during compile, float during eval. Same metric function, two return types based on the trace argument. Compile-time bool prevents the optimizer from chasing noise in the middle of the distribution; eval-time float gives gradient for reporting and debugging. [dspy.ai/learn/evaluation/metrics/]
3. The metric must be calibrated against humans. ≥20 spot-checks where you (the human) rate the same examples the metric does. If metric disagrees with human on >20% of cases, fix the metric before any compile. Otherwise the optimizer just learns the metric's bias.

Why holistic judges fail (the catalog)

Full catalog in references/R2-judge-bias-catalog.md.

3. SOP (Standard Operating Procedure)

0. Confirm activation criteria (§1)
1. DECOMPOSE: list orthogonal criteria
2. CHOICE: bool vs float per criterion
3. WRITE: implement sub-judges + aggregator
4. BIAS-TEST: probe for length, position, self-preference
5. CALIBRATE: ≥20 human spot-checks; agreement ≥80%
6. SHIP: hand metric to optimizer / eval loop

Stage 1 — Decompose

Write down the criteria the user actually cares about. For an open-ended Q&A:

• Factual? (yes/no) — answer makes no unsupported claims
• On-topic? (yes/no) — addresses the question
• Concise? (length penalty, scalar) — token count vs budget
• Non-hedging? (yes/no) — no "as an AI language model..." or "it depends" cop-outs
• Cites source? (yes/no) — if grounded retrieval is required

For RAG specifically, copy LlamaIndex's triad: Faithfulness (answer entailed by context), Relevancy (answer addresses query), Retriever quality (MRR / hit-rate on labeled QA pairs). See references/R1-source-evidence.md.

Output of this stage: a checklist of 3–6 sub-criteria, each with type (bool / scalar) and aggregation rule (AND for hard gates, weighted sum for soft).

Stage 2 — Bool vs Float Choice (per criterion)

Per [dspy.ai/learn/evaluation/metrics/]:

Universal rule (DSPy OP-030): wrap the whole metric so it returns bool when trace is not None (compile mode) and float otherwise (eval mode). Same function, two semantics.

Stage 3 — Write the Judge

import dspy

class Assess(dspy.Signature):
    """Assess a single binary criterion. Reply yes/no."""
    text_to_assess = dspy.InputField()
    assessment_question = dspy.InputField()
    assessment_answer: bool = dspy.OutputField()

def metric(example, pred, trace=None):
    # Hard gates (bool sub-judges)
    factual = dspy.Predict(Assess)(text_to_assess=pred.answer,
        assessment_question="Is every factual claim supported by the context?").assessment_answer
    on_topic = dspy.Predict(Assess)(text_to_assess=pred.answer,
        assessment_question=f"Does this address the question: '{example.question}'?").assessment_answer
    non_hedging = dspy.Predict(Assess)(text_to_assess=pred.answer,
        assessment_question="Does this answer commit to a position (no 'it depends' / 'as an AI' hedging)?").assessment_answer

    # Length penalty (deterministic — DO NOT delegate to judge)
    over_budget = len(pred.answer.split()) > 150
    length_score = 1.0 if not over_budget else max(0.0, 1.0 - (len(pred.answer.split()) - 150) / 150)

    if trace is not None:  # compile mode → strict bool
        return factual and on_topic and non_hedging and not over_budget

    # eval mode → float for reporting
    return (factual + on_topic + non_hedging) / 3.0 * length_score

Rules:

• One yes/no question per sub-judge. Combining ("is it factual AND concise?") re-introduces holistic confusion.
• Length is a deterministic scalar, never an LLM call. Judges have systematic length bias [arxiv.org/pdf/2506.02592].
• Use a cheaper / different model family for the judge than the task model (mitigates self-preference). E.g. task = GPT-4o, judge = Claude-Haiku.

Stage 4 — Bias Test

Before calibration, run these probes on a 20-example dev set:

1. Length probe: take 10 good answers, append a redundant paragraph. Does the metric score them higher? If yes, your length penalty is too weak.
2. Self-preference probe: generate the same answer from 2 model families. Does the judge prefer its own family by >10pp? If yes, swap judge family.
3. Position probe (pairwise only): swap A/B order. Does winner flip >10% of the time? If yes, randomize order or average both orderings.
4. Rubric-order probe: list sub-judges in 2 different orders. Does aggregated score shift >5pp? If yes, randomize sub-judge order per call.

Document failures in references/R2-judge-bias-catalog.md for this project.

Stage 5 — Human Calibration

• Sample 20 (or 30 if open-ended) examples spanning the task distribution.
• A human rates each on the same sub-criteria.
• Compute per-sub-judge agreement (Cohen's κ or simple accuracy).
• Threshold: ≥80% agreement on each sub-judge, ≥80% on aggregate.
• If below: fix the metric, do not compile. Common fixes: rephrase the assessment question, switch judge model, narrow the question scope.

This is non-negotiable. Per DSPy Case C: "Never compile against a metric you haven't human-validated on ≥ 20 spot-checks." Garbage metric → garbage compiled program.

Stage 6 — Ship

Hand off:

• The metric function (callable).
• A calibration receipt: {n_spot_checks, per_judge_agreement, bias_probe_results, judge_model_id, task_model_id, date}. Stored next to the compiled artifact. Required for any later audit.
• Recommended optimizer pairing: if sub-judges produce textual feedback (not just bool), pipe into dspy.GEPA for sample efficiency; otherwise MIPROv2.

4. 操作模型 (Operations)

OP-M01 — DecomposeMultiCriteria

• Trigger: Open-ended generation; multiple correctness axes (factuality + tone + length + relevance).
• Action: List 3–6 orthogonal yes/no sub-criteria. One dspy.Predict(Assess) call per criterion. Aggregate with AND (hard) + weighted sum (soft).
• Output: Multi-judge metric callable; one criterion per call.
• Evidence: [dspy.ai/learn/evaluation/metrics/]; DSPy Case C; LlamaIndex Faithfulness+Relevancy+Retriever triad.

OP-M02 — BoolDuringCompileFloatDuringEval

• Trigger: Writing any metric used for both compile (teleprompt.compile) and dspy.Evaluate.
• Action: Inside the metric, if trace is not None: return bool_aggregate; else: return float_aggregate.
• Output: One metric function, two modes; optimizer avoids chasing float noise.
• Evidence: [dspy.ai/learn/evaluation/metrics/]; [dspy.ai/cheatsheet/]; DSPy OP-030.

OP-M03 — ExplicitLengthPenalty

• Trigger: Any LLM-as-judge metric (always).
• Action: Add a deterministic scalar length_score = clip(1 - max(0, len - budget) / budget, 0, 1) multiplied into the final float. Do NOT delegate length judgment to the LLM.
• Output: Length-controlled metric immune to the judge's length bias.
• Evidence: [arxiv.org/pdf/2506.02592]; DSPy SOP step "Add length penalty as separate scalar".

OP-M04 — CrossFamilyJudge

• Trigger: Judge model is in same family as task model (e.g., both GPT-4).
• Action: Swap judge to a different family (Claude, Llama, Gemini, Mistral). Document choice.
• Output: Self-preference mitigated.
• Evidence: [arxiv.org/pdf/2506.02592].

OP-M05 — HumanSpotCheck20

• Trigger: Before any compile / production rollout of a new or modified metric.
• Action: 20 (open-ended: 30) human ratings on the same sub-criteria. Compute agreement. Reject if <80% per-sub or aggregate.
• Output: Calibration receipt; metric is human-validated.
• Evidence: DSPy Case C step 4; [dspy.ai/learn/evaluation/metrics/].

OP-M06 — RAGFaithfulnessRelevancyContext

• Trigger: RAG pipeline; tuning chunk size / reranker / retriever-k / hybrid alpha.
• Action: Instantiate three evaluators: FaithfulnessEvaluator (answer entailed by retrieved context), RelevancyEvaluator (answer addresses query), RetrieverEvaluator(["mrr","hit_rate"]) (does retrieval pull the gold passage). Gate every PR.
• Output: 3-axis RAG quality vector; never a single number.
• Evidence: LlamaIndex OP-10 EvalLoop; [developers.llamaindex.ai/python/framework-api-reference/evaluation/].

OP-M07 — BiasProbeSuite

• Trigger: Before calibration.
• Action: Run length-probe, self-preference probe, position probe, rubric-order probe (§3 Stage 4) on 20 examples. Log deltas.
• Output: Bias-probe report; rejected if length probe ≥+5pp on padded answers or position flips >10%.
• Evidence: [arxiv.org/pdf/2509.26072]; [arxiv.org/pdf/2506.02592].

OP-M08 — TextualFeedbackForGEPA

• Trigger: Sub-judge can articulate why it failed (e.g., "answer was verbose", "missed cited source").
• Action: Return dspy.Prediction(score=float, feedback=str) from the metric. Pair with dspy.GEPA optimizer (not MIPROv2).
• Output: Sample-efficient, reflection-evolved prompts.
• Evidence: [dspy.ai/api/optimizers/GEPA/overview/]; [arxiv.org/abs/2507.19457]; DSPy OP-004.

OP-M09 — RandomizeJudgeOrder

• Trigger: Pairwise judge or rubric with N>3 criteria.
• Action: Per-call: randomize A/B order in pairwise; randomize sub-judge ordering. Or run both and average.
• Output: Position and rubric-order bias mitigated.
• Evidence: [arxiv.org/pdf/2509.26072].

OP-M10 — CalibrationReceipt

• Trigger: Ship a compiled artifact.
• Action: Save JSON next to artifact: {metric_version, n_spot_checks, per_judge_agreement, bias_probe_results, judge_model_id, task_model_id, date, criteria_list}.
• Output: Audit-ready provenance for the metric used to compile.
• Evidence: General audit best-practice; DSPy save(save_program=True) analog.

5. 困境决策案例 (Dilemma Cases)

Dilemma 1 — MIPRO inherited the judge's verbosity bias

困境: Team compiled an open-ended customer-support response program with MIPROv2(metric=llm_judge, auto="medium"). Auto-eval score climbed from 0.62 to 0.84. Production rolled out. Users complained answers were too long. Auditor traced it: the LLM judge gave +0.18 to answers >120 words across the eval set, including some objectively wrong ones. The optimizer faithfully chased that signal.

约束: Re-compile costs ~$30. Cannot re-label dataset. Cannot change judge model (vendor approval cycle).

决策步骤:

1. Diagnose: Confirm length bias on a held-out set: bin answers by length, plot judge-score vs length. Look for monotonic positive slope. (It was there: +0.21 per 50 words.)
2. Patch the metric, not the artifact: Add explicit length penalty (OP-M03) and decompose the holistic judge into factual / on-topic / non-hedging sub-judges (OP-M01). Length is now deterministic, not judge-rated.
3. Re-calibrate on 30 human spot-checks (OP-M05). Agreement: was 64%, now 87%.
4. Re-compile: Same MIPROv2, new metric. Score on new metric: starts at 0.55 (because old prompts over-verbose under new metric), recovers to 0.79 after compile.
5. Production A/B: New artifact has 38% shorter answers, equal factuality, +12% user thumbs-up.

结果: The "score regression" (0.84 → 0.79) was misleading — the old metric was the bug. Real quality improved because the metric finally matched the user.

可提取的操作: OP-M01 DecomposeMultiCriteria, OP-M03 ExplicitLengthPenalty, OP-M05 HumanSpotCheck20. Lesson: when a compiled program is "good on metric, bad in production", the metric is wrong. Don't tune harder — fix the metric.