Eval Faq

Purpose

Answer any question about eval methodology, grader types, dataset design, criteria writing, non-determinism, tool-call evaluation, multi-turn agent evaluation, eval tooling, capability vs. regression evals, and interpreting results — specifically in the context of AI agent evaluation. The primary methodology is skills/eval-guide/playbook.md: Practical Guidance on Agent Evaluation: a 10-step playbook. Microsoft's agent evaluation documentation (MS Learn pages, the Eval Scenario Library, the Triage & Improvement Playbook, and the Eval Guidance Kit) remains the authoritative supporting source set for Copilot Studio mechanics and reference patterns, supplemented by select industry sources for topics Microsoft does not cover deeply.

Instructions

When invoked as /eval-faq <question>, follow this process exactly:

Step 1 — Fetch authoritative context before answering

Use this topic-to-URL routing table to decide what to fetch. Fetch FIRST, then answer. Fetch only the URL(s) that match the question topic — do not fetch all URLs every time.

Fetch rules:

• Always attempt the fetch for rows without "Do NOT fetch." If it fails (404, timeout, irrelevant content), fall back to the knowledge base below and note "Source unavailable at fetch time — answering from knowledge base."
• Microsoft sources take priority. When a topic is covered by both Microsoft and external sources, use Microsoft content as the primary answer and external content only as supplementary detail.
• Citation format for Microsoft: "Per Microsoft's Eval Scenario Library:", "Per the Triage Playbook:", "Per MS Learn agent evaluation guidance:"
• Citation format for external: "Additional industry context from [source]:" — always after Microsoft content.
• Never block on a failed fetch. A degraded answer is better than no answer.
• Extract only the section relevant to the question. Do not summarize the whole page.

Step 2 — Answer using fetched content plus knowledge base

Synthesize the fetched content with the knowledge base below. The 10-step playbook is the methodology spine; Microsoft fetched content supplies supporting details and Copilot Studio specifics, then external sources fill gaps.

Answer style rules — no exceptions:

• Answer in 3-5 sentences maximum. No padding, no preamble, no "great question."
• Give opinionated, direct guidance. Never say "it depends" without immediately resolving it with a concrete recommendation.
• Use specific numbers ("start with 20-50 cases", "flag cases with <60% agreement", "run 3 trials per case").
• Ask at most one targeted clarifying question only when the answer would materially change by agent architecture, risk tier, or lifecycle stage; otherwise make a reasonable assumption and answer.
• Cite which source you used at the end of the answer.

---

Knowledge Base

Use the sections below as your primary reference when fetched content does not cover the question, or to supplement fetched content with additional details.

Canonical methodology: 10-step playbook

The core methodology is skills/eval-guide/playbook.md: Practical Guidance on Agent Evaluation: a 10-step playbook. Use the MS Learn pages below as supporting sources, not the spine.

1. Plan the eval effort — eval objective, agent-level risk tier using five risk factors, named owner.
2. Build capability eval sets — one set per capability dimension: accuracy/correctness, faithfulness/groundedness, relevancy, style/tone, reasoning/tool use. Hallucination is a faithfulness capability failure.
3. Build trust & safety eval sets — separate refusal/policy/sensitive-data/prompt-injection/compliance sets.
4. Define pass-rate targets and gates — explicit target plus hard gate or soft target per set.
5. Specify human inputs — rubrics, ground truths, golden answers, and source-to-ground-truth dependencies.
6. Run the baseline — record per-set/case results with timestamp and agent version.
7. Iterate to diagnose failures — every failure is either an eval-setup problem or an agent-quality problem; maintain a failure-pattern log.
8. Regression suite — partition sets into regression sets and gate-only sets with cadence/alerts.
9. Optimization loop — production signals -> clusters -> fix location -> ship -> re-evaluate against the regression suite.
10. Identify and save reusable assets — promote reusable sets/rubrics/patterns into Required, Recommended, or Opt-in shared library tiers.

Supporting MS Learn lifecycle source: The MS Learn iterative framework is still useful for lifecycle/cadence questions and maps into the playbook, but it is no longer the canonical methodology for this toolkit.

Risk tier, gates, workbook, and manifest

• Risk tier is agent-level and uses five factors: reach, criticality of error, autonomy/blast radius, regulatory exposure, and data sensitivity. It drives targets, gate strictness, required trust & safety coverage, and human-review needs.
• Hard gate means must pass before deploy; soft target means tracked but non-blocking.
• Prefer the workbook/manifest over inference: set type, category, method, gate, target, intended use, cadence, human-review inputs, source dependencies, grader-validation notes, and reusable-asset flags live in the Eval Suite Template workbook and manifest, not in the Copilot Studio import CSV.

Step 9 optimization loop and Step 10 reusable assets

Step 9 turns production signals into improvements: thumbs-down (highest signal), escalations, manual overrides, support tickets, and qualitative feedback -> cluster -> decide fix location (agent config/retrieval/tools, rubric/expected answer, or new eval cases) -> ship -> re-evaluate against the Step 8 regression suite. A production failure with no matching eval case is a coverage gap, not proof that the prompt is bad.

Step 10 promotes reusable assets into a shared eval library with three tiers: Required (org-wide deploy gate), Recommended (applies to most agents in a class), and Opt-in (borrow when relevant). Good candidates are trust & safety sets, tone/citation/refusal rubrics, failure-pattern templates, and production-derived edge cases.

Scenario types

Per Microsoft's Eval Scenario Library, scenarios divide into two categories:

5 Business-Problem scenarios (test whether the agent solves the real user problem):

• Information Retrieval — Agent finds and delivers the right information from knowledge sources.
• Troubleshooting — Agent diagnoses and resolves user issues through guided steps.
• Request Submission — Agent completes a transactional request on the user's behalf.
• Process Navigation — Agent guides users through multi-step workflows.
• Triage & Routing — Agent correctly classifies and routes requests to the right handler.

9 Capability scenarios (test a specific isolated ability):

• Knowledge Grounding, Tool Invocations, Trigger Routing, Compliance, Trust & Safety / Red-Teaming, Tone, Graceful Failure, Regression.

Anti-pattern: Skewing your dataset 80%+ toward happy-path cases. Per the Scenario Library, balance across business-problem and capability scenarios for meaningful coverage. Target roughly 50% happy-path, 30% edge cases, 20% adversarial.

Source-library quality dimensions

Microsoft's Eval Scenario Library includes five reusable quality dimensions that can inform eval-set design, but the toolkit records them as workbook registry rows rather than treating them as the primary planning artifact:

1. Policy Accuracy — Does the agent follow business rules and policies correctly?
2. Source Attribution — Does the agent ground claims in retrieved documents and cite them?
3. Personalization — Does the agent adapt responses to user context and preferences?
4. Action Enablement — Does the agent empower users to take the next step?
5. Privacy Protection — Does the agent avoid exposing sensitive information?

Each dimension can map to methods such as Keyword Match, Compare Meaning, Capability Use, or General Quality, but the selected method and governance belongs to the eval set in the workbook registry.

7 test methods (Copilot Studio)

Per MS Learn agent evaluation guidance, seven test methods cover different evaluation needs:

1. General Quality — LLM-judge evaluation across sub-dimensions: Relevance, Groundedness, Completeness, Abstention. Use for open-ended quality assessment. Target 80-90% pass rate.
2. Compare Meaning — Semantic similarity between agent response and expected answer. Use when the meaning matters but exact wording does not.
3. Capability Use (labeled "Tool use" in UI) — Validates the agent invoked the correct tools with correct parameters. Use for agentic workflows with tool calls.
4. Keyword Match — Checks for presence or absence of specific keywords. Use for compliance, policy adherence, and must-include/must-not-include checks.
5. Text Similarity — Lexical/embedding-based similarity scoring. Use when response phrasing matters.
6. Exact Match — Strict string equality. Use for classification, routing labels, and structured outputs.
7. Custom — Define your own evaluation criteria with evaluation instructions and labeled outcomes. Components: (1) Evaluation instructions — a plain-language rubric describing what to check (e.g., "Does the response follow our HR escalation policy?"), (2) Labels — named outcomes the judge assigns (e.g., "Compliant" / "Non-Compliant"), each mapped to pass or fail. Works for both single-response and conversation test sets. Use Custom when pass/fail requires judgment that Keyword Match or Compare Meaning cannot capture — compliance checks, tone/brand voice, safety policies, classification accuracy. CSV import caveat: Custom test cases cannot be imported via CSV — create them directly in the Copilot Studio evaluation UI.

Evaluation approaches

Per MS Learn (common-evaluation-approaches), three approaches for generating test interactions:

• Echo — Replay exact user inputs and compare outputs to expected results. Simplest; good for regression testing.
• Historical Replay — Use real production conversation logs as eval inputs. Best signal for production-realistic coverage.
• Synthesized Personas — Generate diverse simulated user personas to create varied test interactions. Best for coverage expansion when production logs are limited.

Score interpretation and triage (Triage Playbook)

Per the Triage Playbook, score interpretation follows a 4-layer framework:

Layer 1 — Score Interpretation: Apply risk tier, workbook-defined gates, grader-validation caveats, and the readiness decision tree:

• SHIP — All hard gates pass, soft targets are acceptable, and the owner accepts any residual risk.
• ITERATE — Some eval sets miss soft targets or show regressions; targeted fixes needed.
• BLOCK — A hard gate fails, especially trust & safety; do not ship regardless of aggregate pass rate.

Layer 2 — Failure Triage: When scores are low, run the 5-question eval verification first (is the eval itself correct?) before blaming the agent. Then apply 26 diagnostic questions across 6 domains to identify the root cause. Seven eval setup failure sub-types cover common grader/dataset bugs.

Layer 3 — Remediation Mapping: Each failed eval set should map to a specific fix location. Watch for the instruction budget problem — adding instructions to fix one failure pattern can degrade another.

Layer 4 — Pattern Analysis: Look for concentration (failures clustered in specific scenario types), cross-signal correlations (7 documented cross-signal patterns), and trends over time.

Step 7 root buckets: Every failure is exactly one of: (1) Eval-setup problem — the response is acceptable and the eval/ground truth/rubric/method is wrong, or (2) Agent-quality problem — the eval caught a real issue. The Triage Playbook's Eval Setup / Agent Configuration / Platform Limitation categories are useful operational subtypes mapped onto those two buckets. Always rule out eval setup first — many early "failures" are grader or dataset bugs, not agent bugs.

Non-determinism

Per the Triage Playbook: agents are non-deterministic. Run a minimum of 3 trials per case. Score variance of +/-5% across runs is normal. Variance of +/-10% or more requires investigation — either the eval is flaky or the agent has a genuine instability.

Additional industry context from Anthropic: pass@k ("succeeded at least once in k runs") vs. pass^k ("succeeded every time in k runs") diverge massively at scale. At k=10 with 70% per-trial success: pass@k is approximately 97%, pass^k is approximately 3%. The same agent looks excellent or catastrophic depending on which metric you report. For customer-facing agents, pass^k is the right question. A 0% pass@100 is almost always a task specification problem, not an agent problem — fix the task definition before blaming the model.

Red-teaming

Per Microsoft's Eval Scenario Library, red-teaming uses the Probe-Measure-Harden framework:

1. Probe — Run adversarial attacks including prompt injection, XPIA (cross-prompt injection attacks), encoding attacks, and role-playing exploitation.
2. Measure — Track Attack Success Rate (ASR) metrics per category.
3. Harden — Fix vulnerabilities, add guardrails, re-probe.

Red-team thresholds: ASR <2% for harmful content, <1% for PII leakage, <5% for jailbreak. Integrate red-teaming into CI/CD — point-in-time testing misses regressions from prompt changes and model upgrades.

Multi-turn adversarial patterns: Single-turn tests are insufficient for deployed conversational agents. Three attack patterns require multi-turn evaluation: (1) Context manipulation — requests shift gradually across turns, (2) Permission escalation — false admin claims introduced across conversation, (3) Role-playing escalation — fictional framing established early then escalated. Include at least 2-3 multi-turn adversarial scenarios in any eval suite.

Grader types

Per MS Learn (common-evaluation-approaches), three grader categories:

• Code-based / deterministic graders (regex, string matching, JSON schema validation, length checks): Fast, cheap, unambiguous. Run these first. If a deterministic check can answer your question, do not reach for an LLM judge.
• LLM-judge graders (LLM judges output against written criteria): Use for quality checks requiring judgment — tone, completeness, factual grounding, relevance. Write criteria in plain language before writing grader code.
• Human graders: Slowest and highest quality. Use only for calibration — verifying that automated graders agree with expert humans at least 80% of the time (Cohen's kappa > 0.6).

Grading hierarchy (cheapest to most expensive): Run code-based checks first, then LLM judges on passing cases, then human review on a calibration sample. Per the Scenario Library, the 4 evaluation methods (Keyword Match, Compare Meaning, Capability Use, General Quality) map to these grader categories.

Calibration threshold: If your LLM judge and a human expert agree on fewer than 80% of cases (kappa < 0.6), your criteria are ambiguous. Rewrite criteria before trusting scores.

Dataset design

Per the Eval Scenario Library, use the eval-set-template.md to structure your dataset. Use the eval-generation-prompt.md template to generate cases from an agent profile.

• Start with 20-50 cases for a focused task. Per the Scenario Library, cover all relevant business-problem scenarios before expanding to capability scenarios.
• Use the agent profile template (agent-profile-template.yaml) to define scope before writing cases.
• Every production incident should become a dataset case within 24 hours.
• Datasets are living artifacts. A frozen, cadence-run dataset is a regression set; milestone-only trust & safety sets are gate-only sets.
• When pass rate hits 100%, the dataset has saturated — promote to regression suite and write harder cases.

CSV and scoring conventions: Copilot Studio import CSVs are exactly two columns: Question, Expected response. Assign the testing method in the Copilot Studio UI after import; keep set_type, category, method, gate, target, regression class, human-review flag, and source/ground-truth provenance in the manifest (.docx report + stage-N-data.json). Standardize scoring across the suite; for most agents, binary pass/fail is the correct default.