Agent Instructions Evaluator

Agent Instructions Evaluator

Overview

Evaluate agent instructions or agent definitions for operational achievability in production settings. This skill focuses on runtime reliability and runtime efficiency rather than writing quality, identifying issues from hidden state, conflicting rules, vague scope, brittle exact phrasing, underspecified tool behavior, instruction overload, and performance friction caused by excessive or contradictory runtime reasoning.

Use this skill when you need:

• A practical evaluation report focused on runtime reliability
• Evidence-backed prompt review with concrete recommendations
• A reusable report artifact that can be shared with reviewers
• Per-dimension scoring that preserves nuance rather than averaging away critical issues

Core Principle

Score the artifact not by how much behavior it describes, but by how much behavior the agent can reliably execute. More rules do not automatically make a better prompt—more rules often lower achievability. An instruction set that is technically understandable but expensive to reconcile at runtime should still be considered lower-achievability, because slow, unstable, or tool-heavy execution reduces production reliability.

Evaluation Workflow

<Steps>
<Step>
**Gather all relevant data**

Before beginning analysis, collect all available metadata and context. The raw JSON files produced by the extraction scripts are a required deliverable — save them into the eval/ folder alongside the report files. Every report in the set must reference them.

Path note: All scripts/ paths below are relative to the skill directory (skills/agent-instructions-evaluator/). Run these commands from that directory, or prefix the path accordingly (e.g. skills/agent-instructions-evaluator/scripts/extract_agent_info.py).

1. Extract agent metadata (if evaluating a YAML file):

   # Preferred — single command; auto-scans the project tree for tool source files
   python scripts/extract_agent_info.py <agent.yaml> \
       --search-root /path/to/project \
       --output-dir eval/ --json
   
   # Alternative — explicit toolkit directory (faster on large trees)
   python scripts/extract_agent_info.py <agent.yaml> \
       --search-root /path/to/project \
       --tools-root /path/to/toolkit \
       --output-dir eval/ --json
   
   # Alternative — use pre-extracted tool JSONs from step 2 below
   python scripts/extract_agent_info.py <agent.yaml> \
       --search-root /path/to/project \
       --tools-dir eval/ \
       --output-dir eval/ --json

   --tools-root / auto-scan behaviour:

   • When --tools-root is supplied, the script recursively scans that directory for tool source files.
   • When neither --tools-root nor --tools-dir is supplied, the scan runs automatically against --search-root (or the agent YAML's directory if --search-root is also absent). This means in most cases a single command is sufficient.
   • .py files are included only when they contain at least one @tool, @flow, or @<any>.tool() decorated function (e.g. @mcp.tool() in MCP server source files) — all other Python files are silently skipped.
   • .json files are included only when detect_json_tool_type() identifies them as an agentic workflow (spec.kind == "flow") or Langflow format — all other JSON is silently skipped.
   • When both --tools-dir and --tools-root are supplied, --tools-dir entries take priority; --tools-root fills gaps.

   This produces eval/agent_<name>_extracted.json — a full snapshot including:

   • name, display_name, kind, llm, context_variables, guidelines_count

   • instructions_length, instructions_chars, instructions_est_tokens

   • skill_catalog_est_tokens, collaborator_routing_est_tokens, tool_list_est_tokens, tools_spec_est_tokens

   • resolved_tools — each agent-level tool with spec_chars, spec_est_tokens, type, file_path, resolved

   • resolved_collaborators — each collaborator with display_name, description, kind, llm, tools, skills, instructions_length, guidelines_count, routing_est_tokens, collocated

   • skills — each skill fully resolved: description, name_length, description_length, name_too_long, description_too_long, unmatched_placeholders, allowed_tools, resolved_allowed_tools (with per-tool spec_est_tokens), allowed_tools_spec_est_tokens, body_chars, body_est_tokens, catalog_est_tokens, tool_binding_shadows, scripts, references, skill_file

   • Always pass --output-dir eval/ so the JSON is saved alongside the report files.

   • Co-located collaborator YAML files (same directory as the agent) are found automatically; remote files require --search-root.

   • Tool: extract_agent_info.py

2. Extract individual tool metadata (optional — only needed when --tools-root auto-scan is insufficient, e.g. tools in a remote registry or a non-standard format):

   python scripts/extract_tool_info.py <tool.py|tool.json|tool.yaml> --output-dir eval/ --json

   Auto-detects file type:

   • Python .py: @tool decorator → regular Python tool; @flow decorator → Python flow tool
   • JSON .json: spec.kind == "flow" → WxO Agentic Workflow; data.nodes (list) → Langflow workflow
   • YAML .yaml/.yml: kind: knowledge_base → WxO Knowledge Base; kind: mcp → MCP Toolkit

   This produces eval/tool_<name>_extracted.json per tool. Pass the containing directory to extract_agent_info.py via --tools-dir eval/ to enrich all tool entries.

   • Tool: extract_tool_info.py

   Token estimation rules applied during extraction:

   • All estimates use character count ÷ 4 (≈4 chars/token).
   • For Python @tool / @flow functions: spec = function name + docstring + each parameter serialised as its JSON Schema representation (not the bare type string). Literal["a","b"] → {"type":"string","enum":["a","b"]}; list[str] → {"type":"array","items":{"type":"string"}}; unknown custom classes → {"type":"object"} (conservative fallback). Return type is excluded — output schema is not injected into the agent context.
   • For WxO Agentic Workflow JSON: spec = name + display_name + description + input_schema only. Output schema is excluded.
   • For tools that cannot be resolved (definition file not found): a 200-token fallback estimate is used. This appears in the output as ~200 est. tokens (fallback — definition not found).

3. Identify missing tool definitions: Note which tools/collaborators are referenced but not available for inspection.

4. Organize the data: Create a complete picture of:

   • What the agent instructions say
   • What tools are actually available
   • What parameters those tools accept
   • What context variables exist
   • What guidelines constrain behavior
   • What skills are attached and what each skill brings (allowed-tools, scripts, references)
   • Which tools are retrieval/knowledge-base surfaces, their passage-count limits, and where they are referenced (agent instructions, collaborator, or skill body)

   When reviewing tool metadata, flag any tool as a potential retrieval surface if:

   • Its kind is knowledge_base, OR
   • Its name or description contains any of: search, query, retrieve, lookup, knowledge, kb, rag, find, fetch, document, semantic
   • For flagged tools: note any top_k, max_results, limit, or num_passages parameter and its default value. If none exists, record the passage count as unbounded.

Only after gathering all data, proceed to analysis. This ensures:

• Tool grounding assessment is based on actual tool signatures, not assumptions
• Execution feasibility is evaluated against real capabilities
• Recommendations are specific and actionable

  </Step>

<Step>
**Understand the input**
Accept any of these input types:
- Raw system prompt
- Instruction block for an agent
- watsonx Orchestrate native agent YAML
- External agent definition
- Design document describing agent behavior
- Partial excerpt from a larger prompt or policy

If the input is partial, state that the evaluation scope is partial and score only what is visible.

Processing watsonx Orchestrate Agent YAML

When evaluating a watsonx Orchestrate native agent YAML file, extract and evaluate these key sections:

1. Instructions field (instructions:): This contains the primary agent prompt. Evaluate this as the main instruction content for all dimensions.

2. Guidelines field (guidelines:): These are structured rules that supplement the instructions. Count these as additional constraints and conditional logic. Each guideline typically adds:

   • 1 conditional branch (condition → action)
   • 1+ critical constraints if the action contains MUST/NEVER/ALWAYS language
   • Potential tool triggers if the action specifies calling a tool/collaborator

3. Collaborators list (collaborators:): Each entry names a subordinate agent the supervisor can delegate to. The script now resolves each name to its YAML file (co-located first, then --search-root) and extracts: display_name, description, kind, llm, tools, nested collaborators, skills, instructions_length, guidelines_count, and collocated flag.

   Recursive traversal: Collaborators are themselves agents and may have their own collaborators: lists. Resolve and analyze collaborators at all depths — not just the immediate children of the root agent. For each resolved collaborator, check its own collaborators: list and apply the same resolution, CO-1 through CO-7 analysis, and report generation recursively. Stop only when a collaborator has no further collaborators: entries, or when a YAML cannot be resolved (note it as unresolved). Use a visited-set (by agent name or file path) to detect and break cycles — if a collaborator's YAML has already been resolved at a higher level, record the back-edge as a dependency loop (CO-4 violation) and do not recurse into it again.

   For each resolved collaborator at every depth, apply the Collaborator Health checks (CO-1 through CO-7) described below — these are the exact parallel of the SK-1 through SK-7 skill checks, applied to the collaborator's agent YAML instead of a SKILL.md body.

   CO-1 — Single Responsibility: Does the collaborator agent cover exactly one domain or capability? Count the distinct tool categories and top-level behavioral sections in its instructions:. More than one primary workflow is a smell; three or more is a violation.

   CO-2 — Distinct, Non-Overlapping Scope: Do any two collaborators share coverage for the same user intent, topic, or trigger condition? Compare each collaborator's description field and instructions: scope statements against every other collaborator in the parent agent's collaborators: list. Overlapping descriptions force the supervisor to make an ambiguous routing decision — this is a systematic reliability failure, not an edge case. Flag every pair with detectable overlap, quote the overlapping text, and rate the overlap: Exact, High, Moderate, or Low.

   CO-3 — Routing Clarity (Name + Description): Is the collaborator name specific enough to be meaningfully different from all other collaborator names? Is the description field clear, specific, and complete enough that the supervisor — without reading the instructions body — can decide whether to route to this collaborator for a given user turn? Check: does the description explicitly state what intents it covers AND what intents it does NOT cover (boundary conditions)?

   CO-4 — No Cross-Collaborator Dependencies (including dependency loops): Does the collaborator's instructions assume that another collaborator has already run, set state, or returned a value? Look for references to "after X agent", "the result from the previous agent", or implicit state assumptions that could only come from a prior collaborator execution. Collaborators must be independently invocable.

   Additionally, check for dependency loops across the full collaborator set. A dependency loop exists when collaborator A depends on state from collaborator B, and collaborator B depends on state from collaborator A — or any longer chain. Use the same directed-graph / cycle-detection approach as SK-4.

   Signals that indicate a dependency edge from collaborator A to collaborator B:

   • Collaborator A's instructions reference a tool exclusively available in collaborator B's tools: list
   • Collaborator A's instructions reference state or outputs that can only be produced by collaborator B's execution path
   • Collaborator A's instructions contain phrases implying B must have already acted (e.g., "once the account is selected", "after authentication")
   • Collaborator A's collaborators: list nests collaborator B (creating a direct dispatch dependency)

   CO-5 — Instruction Complexity Budget: Apply the same complexity analysis used for agent instructions to each collaborator's instructions: and guidelines:. Collaborators are just agents with their own instruction surface — the same rules apply:

   • Count lines of instruction content (apply Rule E thresholds)
   • Count nested if/then branches (apply Rule C thresholds)
   • Count active operational rules per turn (apply Rule F budget)
   • Count MUST/NEVER/ALWAYS/EXACTLY constraints (apply Rule B)
   • Count implicit state variables (apply Rule A) Report each collaborator's complexity signals separately from agent-level counts.

   CO-6 — Collaborator Correlation and Consolidation: Are any two collaborators so closely related in domain or trigger conditions that they are likely to be dispatched in the same turn or in immediate succession? Assess each collaborator pair for:

   • Adjacent trigger intents: Intents that users commonly express together may span two collaborators, forcing multi-hop dispatch per turn.
   • Cross-instruction tool references: A collaborator's instructions that reference tools exclusively available in another collaborator's tools: create coupling that requires co-dispatch.

   Note on shared tools: Two collaborators sharing tools is common and expected — it is not itself a consolidation signal. Do not recommend merging collaborators solely because they share tools.

   Consolidation recommendation rule: Recommend merging two collaborators into one when:

   • They have CO-2 Moderate or High overlap AND their intents plausibly co-occur in a single user turn, OR
   • One collaborator's instructions explicitly require tools available only in another collaborator's context.

   CO-7 — Collaborator Architecture Performance Surface: Assess the aggregate runtime cost of the collaborator architecture as a whole (see Rule N thresholds):

   • Collaborator count: total number of collaborators (>5: Medium; >10: High)
   • Per-collaborator instructions size: lines per collaborator (>100: Medium; >150: High)
   • Routing ambiguity: average number of plausible collaborator candidates per typical user turn (>2: Medium; >4: High)
   • Multi-collaborator turns: estimated proportion of user intents that plausibly require ≥2 collaborator calls (>20%: Medium; >40%: High)
   • Re-dispatch frequency: evidence of multiple collaborator hops per turn in the supervisor's instructions (any confirmed: Medium)
   • Nested collaborator depth: any collaborator that itself has collaborators (depth > 1: Medium; depth > 2: High)
   • Combined instruction cost: estimated tokens across all collaborator instruction bodies that could be involved in one turn (>2,000: Medium; >4,000: High) Report each component with its measured value and risk rating, then produce an overall collaborator dispatch overhead rating (Low / Medium / High).

   Collaborator depth notation: When reporting multi-level collaborators, qualify each collaborator's identity with its depth and parent, e.g. account_agent (depth 2, parent: supervisor_agent). Cross-collaborator checks (CO-2, CO-4, CO-6) are applied within each depth level — compare siblings only. Parent-to-child relationships are already captured by CO-4 (dispatch dependency), not CO-2 (sibling overlap).

   Structural checks for each resolved collaborator:

   • description: Does it give the supervisor enough routing signal (CO-3)? Vague descriptions cause misdirected dispatch.
   • tools:: The tool list available to the collaborator. Check whether every tool referenced in its instructions is present. Missing tools are execution gaps (feeds CO-1 and CO-5).
   • collaborators:: Nested collaborators. Each nested entry adds another dispatch hop and potential CO-4 dependency edge.
   • instructions_length / guidelines_count: Directly feeds CO-5 complexity budget.
   • collocated: Whether the YAML was found co-located with the parent agent or resolved via --search-root. Remote collaborators may not be available for inspection in all environments.

4. Tools list (tools:): Count the number of tools referenced. Add these to the tool-required behaviors count.

5. Context variables (context_variables:): Note which variables are available. Check if the instructions require tracking additional state beyond these variables.

6. Skills list (skills:): A list of skill names that the agent can load_skill into context at runtime. Each skill is a dynamic instruction module — the agent swaps the active skill's SKILL.md content into its context window when routing to that domain.

   Skill directory structure (resolved by extract_agent_info.py):

   <skill-name>/
   ├── SKILL.md           # frontmatter: name, description, allowed-tools
   ├── scripts/           # optional: Python scripts (.py) available to the skill at runtime
   │   └── *.py
   └── references/        # optional: reference files (any extension) loaded at runtime
       └── *

   For each skill, apply the following checks. These are Skill Health criteria — distinct from the five agent-level dimensions but equally important for production reliability.

   SK-1 — Single Responsibility: Does the skill do exactly one thing? A skill body that covers multiple unrelated intents, sub-topics, or tool categories is doing too much. Count the distinct tool calls or workflows described in the body. More than one primary workflow is a smell; three or more is a violation.

   SK-2 — Distinct, Non-Overlapping Scope: Do any two skills share coverage for the same user intent, topic, or trigger condition? Compare each skill's description frontmatter and body scope statements against every other skill in the agent's skills: list. Overlapping descriptions force the agent to make an ambiguous routing decision at every turn where both skills could plausibly apply — this is a systematic reliability failure, not an edge case. Flag every pair with detectable overlap, quote the overlapping text from both descriptions, and rate the overlap: Exact (same intent, same words), High (same intent, different words), Moderate (shared edge cases or boundary conditions), or Low (tangential but distinct).

   SK-3 — Routing Clarity (Name + Description): Is the skill name specific enough to be meaningfully different from all other skill names? Is the description frontmatter clear, specific, and complete enough that the agent — without reading the body — can decide whether this skill applies to a given user turn? A description that requires the agent to already know the domain details to understand it is circular. A description that is vague enough to match multiple domains is ambiguous. Check: does the description explicitly state what intents it covers AND what intents it does NOT cover (boundary conditions)?

   SK-3a — Frontmatter validation (hard limits that prevent the skill from loading):

   • Name length: The name field must be ≤ 64 characters. A name exceeding 64 characters will prevent the skill from importing. Flag immediately as a hard failure — the skill will not be reachable at all.
   • Description length: The description field must be ≤ 1024 characters. Past this limit the skill will not load. Flag immediately as a hard failure.
   • Unmatched {{placeholder}} tokens: Any {{identifier}} in the description or body with no matching param entry quietly becomes a hole — the model reads a sentence with a gap in it. Scan both the description and the body for {{...}} patterns and verify each has a matching param. Flag unmatched placeholders as a hard failure.

   SK-4 — Cross-Skill Dependencies, Mid-Body Handoffs, and Dependency Loops: This check covers three distinct cases. Assess each independently.

   Case 1 — Backward assumption (violation): Does the skill body assume that another skill has already run, set state, or returned a value? Look for references like "after X skill", "the result from the previous skill", "the intent identified earlier", or implicit state assumptions that could only come from a prior skill's execution. Skills must be independently executable — a skill that requires a prior skill to have run cannot be reliably activated in all valid routing paths.

   Case 2 — Mid-body load_skill (violation): Does the skill body issue a load_skill call before its own work is complete, and then expect to resume its own steps after? When the new skill loads, the current skill's instructions are gone — there is no returning. Any steps written after a mid-body load_skill are unreachable. Detect this by checking whether the load_skill call appears before the skill's terminal step and whether subsequent steps depend on coming back to this body.

   Case 3 — Terminal handoff (document, do not flag as violation): A skill body may issue a load_skill call as its last action, after all of its own steps are complete. This is a valid one-way handoff — the skill has finished its work, and the next skill picks up from the conversation state. Document these handoffs neutrally in the report (note the target skill). If a chain of terminal handoffs exists (A → B → C → …), note the chain depth — multiple hops mean each skill's instructions are replaced in sequence, which is a "telephone game" risk worth flagging for the author to assess. Document; do not penalise.

   Dependency loop check (violation): Build a directed graph using forward load_skill pointers: draw an edge A → B if skill A's body contains a load_skill call to skill B (mid-body or terminal). Check this graph for cycles. A cycle means whichever skill loads second erases the one that loaded first — neither finishes. Any cycle is a violation regardless of where in the body the load_skill call appears.

   How to detect loops:

   1. For each skill, find every load_skill call in its body — note both mid-body and terminal calls.
   2. Draw a directed edge for each: skill A → skill B.
   3. Check the full graph for cycles of any length (A → B → A, or A → B → C → A, etc.).
   4. Any cycle is a loop violation — report it as a separate finding, naming all skills in the cycle.

   Signals that indicate a dependency edge from skill A to skill B:

   • Skill A's body contains an explicit load_skill call targeting skill B (mid-body: violation; terminal: document)
   • Skill A's body references state, context variables, or outputs that can only be set by skill B's execution path
   • Skill A's body contains phrases like "after selecting the product" or "once the account is identified" where the selection/identification is the job of another named skill
   • Skill A's body references a tool that is exclusively in skill B's allowed-tools (and not in A's own allowed-tools or the agent's top-level tools:)

   SK-5 — Instruction Complexity Budget: Apply the same complexity analysis used for agent instructions to each skill's SKILL.md body. Skills are just agent instructions scoped to a domain — the same rules apply:

   • Count lines of instruction content in the body (apply Rule E thresholds)
   • Count nested if/then branches (apply Rule C thresholds)
   • Count active operational rules per turn (apply Rule F budget)
   • Count MUST/NEVER/ALWAYS/EXACTLY constraints (apply Rule B for exact phrases)
   • Count implicit state variables (apply Rule A) A skill that individually exceeds any Rule C/E/F threshold is a complexity risk, regardless of how the agent-level instructions score. Report each skill's complexity signals separately.

   SK-6 — Skill Correlation, Consolidation, and Tool-Shadowing: Assess each skill pair for sequential-load coupling, and separately assess the agent's tool declarations for shadowing gaps.

   Tool declaration model (check this first): Every tool any skill uses must be declared in agent tools: — that is the authoritative registry. A skill's allowed-tools restricts which declared tools are visible in the skill's context. This produces three categories:

   • Agent-only tools (in agent tools:, not in any allowed-tools): callable at base level ✓
   • Skill-filtered tools (in agent tools: AND in one or more allowed-tools): callable only inside those skill contexts — shadowed at agent base level ✗
   • Skill-only tools (skill-filtered AND not referenced in agent instructions:/guidelines:): informational only — no execution gap because the agent never intended to call them at base level

   Correlation checks (per skill pair):

   • Adjacent trigger intents: Intents that users commonly express together in one message (e.g. "show my balance and last transactions") may span two skills, forcing sequential load_skill calls. Each call replaces the active skill body — the first skill's instructions are gone by the time the second loads.
   • Cross-body load_skill references: A skill body that instructs the agent to load_skill another skill before the first body's own work is complete creates an instruction-loss risk (see SK-4). This is a coupling signal, not just a performance signal.

   Tool-shadowing execution gap (Trigger 3 — High severity): Cross-reference every tool named in the agent's instructions: or guidelines: against shadowed_resolved_tools. If any match, the agent has written an instruction it cannot execute at base level — the tool is declared but unreachable outside a skill context. This is a deterministic silent execution gap. Flag every affected tool individually. Fix: either remove the tool from the relevant skill's allowed-tools (making it callable at base level), or move the agent-level instruction into the skill body where the tool is accessible.

   Zero-tool base coverage (Trigger 3a — intentionality check): Check whether active_resolved_tools is empty (all declared tools are skill-filtered). If so, the agent has no tools before any skill loads. This is a valid and common design for a pure skill-routing agent — it is not an error. Check whether the agent instructions: contain an explicit statement confirming that every user intent is handled through a skill. If yes: record as confirmed, no note needed. If no such statement exists: raise a lightweight intentionality note asking the author to confirm. Either way, Trigger 3a does not lower any dimension score.

   Consolidation recommendation rule: Recommend merging two skills into one when:

   • They have SK-2 Moderate or High overlap AND their intents plausibly co-occur in a single user turn, OR
   • One skill's body issues a load_skill call to the other mid-workflow (indicating inseparable sequential dependency).

   SK-7 — Skill Architecture Performance Surface: Assess the aggregate runtime cost of the skill architecture as a whole using these components (see Rule N in signal-rules.md for thresholds). Because only one skill body is in the prompt at a time, the cost is per-load, not per-turn-sum.

   • Skill count: total number of skills (>5: Medium; >10: High)
   • Per-skill body size: lines of instruction per skill body (>100: Medium; >150: High)
   • Routing ambiguity: average number of plausible skill candidates per typical user turn (>2: Medium; >4: High)
   • Load transitions per turn: estimated number of load_skill calls per turn for multi-step intents (>1 per turn: Medium; >2 per turn: High)
   • Re-load frequency: evidence that the agent reloads the same skill within a single turn (any confirmed: Medium)
   • Per-load token cost: estimated tokens for the largest skill body likely to be loaded in one turn (>2,000: Medium; >4,000: High) Report each component with its measured value and risk rating, then produce an overall skill load overhead rating (Low / Medium / High).

   SK-8 — Redundant or conflicting platform content: Does the skill body or the agent instructions contain content that the platform already injects, or describe fixed platform behavior the author cannot change? Both produce token waste and risk in-context contradictions when the author's description drifts from actual platform behavior.

   Canonical item list and triggers: see Rule J2 in signal-rules.md. Items are organized into three categories: A (platform-injected content), B (fixed behavior the author cannot override), and C (content misplaced in the wrong location). Apply each trigger as defined there.

   Effect: Category A — remove (already provided). Category B — remove (conflict risk). Category C — move to the right location or omit.

   Structural checks for each skill:

   • description (frontmatter): Does it give the agent enough signal to decide when to load this skill (SK-3)? Vague descriptions cause misdirected routing.
   • allowed-tools: The explicit tool allowlist for this skill context. Check whether every tool the SKILL.md body instructs the agent to call is present here. Tools referenced in SKILL.md body but absent from allowed-tools cannot be called — this is an execution gap (feeds SK-1 and SK-5).
   • scripts/: Python scripts uploaded to the skill and available for the agent to invoke as sandboxed compute — no network access, no file-system access. Check all of the following:
     • Does the body instruct the script to call an external API or read a file? Flag as an execution gap — that work must move to a tool.
     • Does the body reference the exact script path (e.g., scripts/calculate_fee.py)? If not, the invocation may fail silently.
     • Do the argument names in the body match the script's declared parameters exactly? Mismatches are silent failures.
     • Does the body reproduce the script's logic in prose (e.g., re-stating the fee formula in text)? Flag — the model may do the math itself instead of calling the script, producing inconsistent results.
   • references/: Reference files (lookup tables, policy docs, etc.) available to the skill at runtime. They are read on demand and stay in context until the skill changes. Check all of the following:
     • Is each reference pointer conditional (e.g., "read X if the customer is on a legacy plan")? A bare pointer with no condition either gets read every turn or never — flag as ambiguous.
     • Does the body ever instruct reading the same reference twice? Flag — it is already in context.
     • Does the body put lookup tables or policy detail inline instead of in a reference? Flag — inline detail inflates the body size permanently; the detail should live in the reference and be read on demand.
     • Does the body use the exact reference path? Vague pointer names can fail silently.
     • Is the workflow logic in the body and the detail/lookup tables in the references? If the split is reversed, correct it. Counting rules for YAML:

• Prompt length: Count only the lines in the instructions: field (exclude YAML structure, metadata, and guidelines)
• Critical constraints: Count MUST/NEVER/ALWAYS/EXACTLY in both instructions: and guidelines: sections
• Nested conditionals: Count if/then branches in instructions: plus each guideline's condition→action pair
• Tool-required behaviors: Sum of collaborators + tools (e.g., 14 collaborators + 1 tool = 15 tool-required behaviors)
• Exact phrases: Count "Respond exactly:", "Say:", and similar requirements in instructions: and guidelines:
• Collaborator routing behaviors: Each collaborator adds at least 1 conditional routing decision (when to dispatch) plus the full instruction surface of that collaborator's own agent YAML
• Per-collaborator complexity: Count lines, nested branches, active rules, and implicit state for each collaborator's instructions: independently — report these separately from agent-level counts (CO-5)
• Collaborator overlap pairs: Count the number of collaborator description pairs with detectable scope overlap (CO-2)
• Correlated collaborator pairs: Count pairs where shared tools or adjacent intents make co-dispatch likely (CO-6)
• Collaborator performance surface: Compute the seven CO-7 components and their aggregate risk rating
• Skill routing behaviors: Each skill listed adds at least 1 conditional routing decision (when to load_skill) plus the full instruction surface of that skill's SKILL.md body
• Per-skill complexity: Count lines, nested branches, active rules, and implicit state for each skill's SKILL.md body independently — report these separately from agent-level counts (SK-5)
• Skill overlap pairs: Count the number of skill description pairs with detectable scope overlap (SK-2)
• Correlated skill pairs: Count pairs where adjacent intents make sequential loading likely, or where a mid-body load_skill creates inseparable coupling (SK-6 Triggers 1 and 2)
• Tool-shadowing execution gaps: For each tool named in agent instructions: or guidelines:, check whether it also appears in shadowed_resolved_tools. Each match is a SK-6 Trigger 3 violation (High severity — agent instruction it cannot execute at base level). Count each affected tool separately.
• Zero-tool base coverage: Check whether active_resolved_tools is empty — SK-6 Trigger 3a (intentionality check only; valid design pattern; raise a lightweight note if no explicit confirmation in agent instructions; no score impact either way)
• Skill-only tools: Count tools in skill_only_tools — informational only; not a violation (SK-6 Trigger 3b)
• Skill performance surface: Compute the six SK-7 components and their aggregate risk rating

Important: Use the utility scripts (extract_agent_info.py, extract_tool_info.py) to extract tool, collaborator, and skill metadata before scoring the "Execution & Tool Grounding" dimension. Both scripts live under scripts/ relative to this skill file — not the top-level workspace. extract_agent_info.py now resolves both collaborator agent YAMLs (co-located first, then --search-root) and skills (SKILL.md). Pass --search-root pointing at the project root when collaborator YAMLs or SKILL.md files are not co-located with the agent YAML. If collaborators, tools, or skills are referenced but their formal definitions cannot be extracted, note this as a limitation and proceed — recommend that the user provide definitions and re-run for a complete assessment. </Step>

<Step>
**Extract evidence**
Use the gathered data to identify and count:
- Exact phrase requirements
- Nested conditional branches
- Implicit state requirements
- Critical constraints (MUST, NEVER, ALWAYS, EXACTLY, etc.)
- Exception clauses
- Subjective classifiers
- Tool-required behaviors
- Hard conflicts between rules
- **Knowledge base / retrieval surfaces** — for each tool identified as a retrieval surface in Step 1:
  - **Passage count limit**: stated `top_k`/`max_results`/`limit` value, or "unbounded" if none
  - **Call condition**: is the tool called unconditionally on every turn, on most turns, or conditionally? Quote the trigger phrase from the instructions.
  - **Number of KB calls per turn**: count how many times the tool (or different retrieval tools) appear in a single turn's workflow
  - **Location**: agent instructions, collaborator instructions, or skill body (note the skill name and its estimated load frequency from SK-7)
  - **Estimated payload**: stated passage count × ≥500 tokens (conservative lower bound; actual chunk size is a runtime variable not visible from static analysis — state this as an approximation and flag the exact value as unverifiable without runtime inspection), or flag as unknown if unbounded

For each resolved collaborator, additionally extract:

• CO-1: Number of distinct tool categories / top-level behavioral workflows in the collaborator's instructions
• CO-2: Any overlap with other collaborators — pairs, quoted evidence, overlap rating
• CO-3: Whether the description explicitly states covered intents AND boundary conditions (what it does NOT cover)
• CO-4: Any cross-collaborator state assumptions — explicit references to another collaborator's output or implicit state; plus: build the dependency graph across all collaborators and check for cycles — list any loop found with all collaborators in the cycle named
• CO-5: Per-collaborator Rule A/B/C/E/F signal counts (lines, branches, active rules, exact phrases, implicit state)
• CO-6: Correlated collaborator pairs — shared tools, adjacent intent co-occurrence, cross-instruction tool references; consolidation recommendation (yes/no with justification)
• CO-7: Collaborator architecture performance surface — all seven components with measured values and risk ratings; overall collaborator dispatch overhead rating

For each resolved skill, additionally extract:

• SK-1: Number of distinct tool calls / primary workflows in the skill body
• SK-2: Any overlap with other skills — pairs, quoted evidence, overlap rating
• SK-3: Whether the description explicitly states covered intents AND boundary conditions (what it does NOT cover)
• SK-4: Case 1 — backward state assumptions (explicit or implicit references to another skill's prior execution). Case 2 — mid-body load_skill calls (note position: before or at terminal step; flag if before). Case 3 — terminal load_skill handoffs (document neutrally, note chain depth if > 1 hop). Plus: build the forward-pointer graph across all skills and check for cycles — report any loop as a separate finding with all skills in the cycle named.
• SK-5: Per-skill Rule A/B/C/E/F signal counts (lines, branches, active rules, exact phrases, implicit state)
• SK-6: Correlated skill pairs (Triggers 1–2) — adjacent intent co-occurrence, cross-body load_skill references; tool-shadowing execution gaps (Trigger 3 — High) — for each tool named in agent instructions:/guidelines:, check shadowed_resolved_tools for a match (each match = agent cannot execute that instruction at base level); zero-tool base coverage (Trigger 3a — intentionality check, no score impact) — is active_resolved_tools empty? if yes, is skill-only operation confirmed in agent instructions?; skill-only tools (Trigger 3b — informational) — tools in skill_only_tools (filtered but never referenced in agent text); consolidation recommendation (yes/no with justification)
• SK-7: Skill architecture performance surface — all six components with measured values and risk ratings; overall skill-load overhead rating