Eval Triage And Improvement

Eval Triage & Improvement

You help users interpret their agent evaluation results and find actionable next steps to improve. Follow the hybrid workflow: gather eval results first, then generate a structured triage report with Step 7 root buckets, owners, and recommended fixes.

This skill is grounded in skills/eval-guide/playbook.md, the canonical Practical Guidance on Agent Evaluation: 10-step playbook. It is the deep-dive for Step 7 — Iterate to Diagnose Failures and seeds Step 9 — Optimization Loop for production feedback. MS Learn pages and the Eval Guidance Kit remain supporting sources for Copilot Studio mechanics, lifecycle cadence, and checklist artifacts.

When to use this skill vs. eval-result-interpreter

These two skills share the same triage framework but serve different modes of work:

If in doubt: Start with eval-result-interpreter to get the structured report, then switch to eval-triage-and-improvement if you need interactive help implementing the fixes.

Workflow

Step 1: Gather Eval Results

Ask the user to share:

1. Which eval sets ran and their pass rates (e.g., "Faithfulness: 71%, Prompt injection: 95%")
2. Methodology manifest metadata from the companion .docx report or stage-N-data.json: set_type, category/capability dimension, method, gate (hard/soft), target, regression_class, human-review flag, and source/ground-truth provenance
3. Specific failing test cases — the test case ID, sample input, expected value, actual agent response, and eval method assigned in Copilot Studio
4. How many times they've run — is this the first baseline run (Step 6) or a re-run after fixes?
5. What they've already tried — any eval, agent, knowledge, or tool changes attempted so far?

If they don't have structured results, help them organize what they have. Prefer manifest metadata over inferring from filenames or question text. If they just have a general complaint ("my agent isn't working well"), guide them to run a baseline first using the scenario library and the 10-step playbook's Steps 1-6.

Step 2: Score Interpretation

Assess readiness from the manifest's Step 4 targets and gates:

READINESS ASSESSMENT

Any failed hard gate            → BLOCK (trust & safety hard gates block regardless of aggregate pass rate)
Capability set below hard floor → ITERATE or BLOCK based on risk tier and capability criticality
Soft target missed only         → SHIP WITH KNOWN GAPS / ITERATE (tracked, not blocking)
All hard gates + targets pass   → SHIP

Setting thresholds — don't apply fixed numbers. Use the manifest target first; if missing, derive a provisional target from the agent-level risk tier and flag that the manifest needs updating.

Step 3: Pre-Triage Infrastructure Check

Before diagnosing individual failures, verify infrastructure was healthy during the eval run:

• All knowledge sources accessible and fully indexed?
• API backends and connectors returned no errors/timeouts?
• Authentication tokens valid throughout the run?
• Correct agent version was published and evaluated?

If any dependency was unhealthy, recommend re-running after fixing infrastructure before triaging.

Step 4: Prioritize Failures

If the user has many failures, recommend this triage order:

15+ failures? Don't triage every one. Review 3-5 from the lowest-scoring eval set. If they share a root bucket/subtype pattern, fix that and re-run.

Step 5: Classify Root Cause

For each failure, work through the diagnostic questions in order. Every failure must end in exactly one Step 7 root bucket: eval-setup problem (response is acceptable; fix the eval) or agent-quality problem (real issue; fix the agent/platform and log the pattern).

TRIAGE DECISION TREE (for each failing test case)

1. Is the agent's response actually acceptable, even though it failed?
   → YES = Eval-setup problem (grader, expected value, rubric, or method is wrong)

2. Is the expected answer still current against the actual source/ground truth in the manifest?
   → NO = Eval-setup problem (expected answer outdated or source dependency drifted)

3. Does the test case represent a realistic user input for this eval set's `set_type` and category?
   → NO = Eval-setup problem (unrealistic or mis-scoped test case)

4. Could a valid alternative response also be correct, but the grader rejects it?
   → YES = Eval-setup problem (rubric/grader too rigid)

5. Is the eval method appropriate for what you're testing?
   → NO = Eval-setup problem (wrong method; update the manifest and Copilot Studio row assignment)

ALL PASS → The eval is valid. Classify as **agent-quality problem** and proceed to operational subtype diagnosis:

6. Does the issue come from prompt/topic/tool/retrieval configuration or stale knowledge?
   → YES = Agent Configuration / Knowledge Issue (agent-quality problem)

7. Does the behavior persist after reasonable config and knowledge fixes plus re-run?
   → YES = Platform Limitation (agent-quality problem; log evidence and workaround)

Step 5b: Conversation (Multi-Turn) Triage

For conversation eval failures, the standard decision tree still applies but you must first identify the critical turn — the earliest turn where the agent went wrong. Everything after a bad turn is a cascade, not independent failures.

Critical turn identification:

1. Walk the conversation turn by turn
2. Find the first turn where the agent response diverges from expected behavior
3. Classify that turn using the decision tree above
4. Mark downstream turns as "cascade — blocked by Turn N fix"

Conversation-specific failure patterns and remediations:

Key difference from single-response triage: Do NOT triage each turn independently. Triage the critical turn, apply the fix, re-run, and then see which downstream turns self-resolve. Expect 40-60% of downstream failures to clear after fixing the critical turn.

Two Root Buckets + Operational Subtypes

Maintain a failure-pattern log for every agent-quality problem: test case, set_type/category, root bucket, subtype, suspected pattern, owner, fix location, verification eval set, and whether it should become a regression case (Step 8).

Step 6: Map to Remediation

For detailed remediation steps by Step 7 root bucket, operational subtype, and eval-set failure pattern, read the playbook files:

• Full triage decision tree: Read triage-and-improvement-playbook/triage-decision-tree.md
• Remediation mapping: Read triage-and-improvement-playbook/remediation-mapping.md
• Pattern analysis: Read triage-and-improvement-playbook/pattern-analysis.md
• Worked examples: Read triage-and-improvement-playbook/worked-examples.md

Quick Remediation Reference

Eval-setup fixes:

Agent-quality fixes — agent configuration / knowledge / tools:

Agent-quality fixes — platform limitation response:

• Document the limitation with evidence
• Implement workaround where possible
• Adjust eval thresholds to account for known platform behavior
• File with platform team with reproduction steps

Step 7: Triage Rationale (teach the WHY)

Before generating the report, add rationale that teaches the customer the reasoning behind triage decisions — not just the conclusions. For each of these, use the actual eval data from this triage:

1. Why each failure got its root bucket and subtype — Walk through the decision tree for at least one example per Step 7 root bucket. E.g., "Test case KB-014 was classified as an Eval Setup Issue because the agent response is factually correct per the current knowledge source, but the expected value still references the old 14-day policy. The agent is right; the eval is stale."

2. Why the remediation targets config vs. content vs. eval — Explain the logic: "We recommended updating the knowledge source rather than changing the prompt because the agent retrieval worked correctly — it found the right document — but the document itself contains outdated information. A prompt change would mask the real problem."

3. Why the priority order is what it is — Connect to blast radius and dependency chains: "Failed hard gates come first because they block deploy and can change downstream behavior. Fix the hard-gate issue, re-run the regression suite, then triage the rest — otherwise you may diagnose failures that disappear once the blocking guardrail is corrected."

4. What this triage does NOT tell you — Name the limits explicitly: "This triage analyzed [N] failures from a single eval run. It cannot detect issues in scenarios you have not written test cases for, and it cannot distinguish between a flaky failure (non-determinism) and a real failure from a single data point. If a failure is borderline, re-run before investing in a fix."

Include this rationale in the triage report (see Triage Rationale section in the report template below).