Kayba Stage 5 Action Plan

Stage 5: Action Plan

Triage each insight and produce a concrete, prioritized action plan.

Inputs

• eval/stage1_insights_summary.md — insights from Kayba
• eval/stage2_domain_context.md — domain context
• eval/baseline_metrics.md — the evaluation rubric
• eval/baseline_metrics.json — baseline values
• eval/compute_baselines.py — measurement code

Read all files before starting.

Process

1. Triage each insight

For each insight/skill, answer three questions in order: Is it valid? Is it already handled? Is it a code fix or prompt fix?

1a. Validity check

• Does it describe a real, recurring problem visible in traces — or noise from a one-off edge case?
• Is it actionable — can the agent actually change this behavior given its tools and context?
• If not valid → verdict: discard with a one-sentence reason.

1b. "Already handled" verification

Do not rely on memory or assumption. Run these checks and cite what you find:

1. Grep the codebase for 2-3 key terms from the insight (tool names, error strings, behavioral keywords). Example: for an insight about cancellation eligibility, grep for cancel, eligibility, criteria.
2. Read the existing system prompt text — check AGENT_INSTRUCTION in the agent file and the domain policy file. Quote any existing language that addresses this behavior.
3. Verdict:
   • If existing text partially covers it → keep as a strengthening fix, note what's missing.
   • If no existing coverage → keep.
   • If existing prompt text already covers the behavior thoroughly AND the baseline metric is >= 95% → discard (cite the existing text and metric). A high baseline alone is NOT sufficient to discard — if the metric is below 95%, there are still failures to fix. An 87% baseline means 1 in 8 attempts still fails; that is worth fixing.

1c. Code-vs-prompt decision tree

Walk through this tree for every non-discarded insight:

Q1: Can the agent fix this by following different instructions?
    (Does it have the right tools, correct data in tool responses,
     and sufficient context to behave correctly?)
  │
  ├─ YES → PROMPT FIX
  │        The agent has everything it needs but acts wrong.
  │        A system prompt addition would fix it.
  │
  └─ NO → Q2: What is the agent missing?
           │
           ├─ Tool doesn't exist, schema is wrong, API returns
           │  incomplete data, infrastructure drops information,
           │  timeout/error not surfaced to agent
           │  → CODE FIX
           │    Name the file, function, and specific change.
           │
           └─ The agent has partial information but the prompt
              can't fully compensate (e.g., needs a new tool
              but a heuristic prompt workaround exists)
              → PROMPT FIX (primary) + CODE FIX (optional)
                Note both. Mark the code fix as "optional" with
                a one-sentence justification for why it's lower priority.

Ambiguity default: When genuinely uncertain, default to prompt fix and add a note: "Classification uncertain — defaulting to prompt fix. Revisit if prompt change doesn't move metrics." This is safer because prompt fixes are cheaper to test and revert, and Stage 7 handles prompt fixes and code fixes through different paths.

Use the reflector's reasoning from Stage 1 insights — it often explicitly identifies root causes that clarify the code-vs-prompt distinction.

2. Consolidate related insights

Before writing recommendations, merge insights that are redundant. Two insights should merge when ALL three conditions hold:

1. Same target behavior — they describe the agent doing (or failing to do) the same thing.
2. Overlapping fix text — the prompt instructions you'd write for each would share >50% of their content.
3. Addressing one substantially addresses the other — fixing insight A would fix >80% of the cases described by insight B.

When NOT to merge — two insights about the same tool or domain area but different failure modes should remain separate. Example: "agent doesn't check cancellation eligibility" and "agent doesn't execute cancellation after user confirms" both involve cancel_reservation but are completely different behavioral failures with different prompt fixes. Keep them separate.

For each merge, document:

• Which insight IDs are combined
• Which insight's framing is primary (use the one with stronger trace evidence)
• What, if anything, is lost from the secondary insight (add it as a sub-point)

3. Write specific recommendations

For each insight (after merging):

• Discards: one sentence on why it's not valid or actionable.
• Code fixes: what code/schema/infrastructure to change. Name the file, the function, the specific change. If Stage 7 needs to find the right code location, give it enough to grep for.
• Prompt fixes: the exact instruction text to add to the system prompt, where it should go (e.g., appended to AGENT_INSTRUCTION, added to domain policy, or as a standalone skill block), and why this wording over alternatives.

4. Assess risk per fix

For each non-discarded fix, assess whether the change could break currently-working behaviors:

For Medium and High risk fixes, add a one-sentence mitigation: what to watch for, or how to word the prompt to preserve working cases.