Kayba Stage 2 Domain Context

Stage 2: Domain Context Gathering

Understand the agent's world — what it does, what tools it has, and what "success" looks like.

Inputs

• TRACES_FOLDER — path to directory containing trace JSON files

Process

0. Detect trace format

Before reading traces, identify the framework that produced them. Read 1 trace file and check:

Record the detected format in the output under Trace Format. All subsequent trace-reading steps use the field paths appropriate for that format.

If the format is unrecognized, note the top-level keys and structure, then proceed best-effort with field names found in the data.

1. Detect architecture

Read 2-3 traces and determine if this is a single-agent or multi-agent system:

• Single agent: one agent_info entry, one conversation thread, tool calls from one identity
• Multi-agent / router: look for multiple agent_info entries, routing tool calls (e.g., transfer_to_*, delegate_to_*), sub-conversation arrays, or distinct system prompts per agent identity

If multi-agent: document each agent separately (name, role, tools, handoff triggers) and note the routing logic. The remaining steps apply per-agent.

2. Find the system prompt

Use a fallback chain — stop at the first hit:

1. Config files — grep for keys: system_prompt, system_message, instructions, AGENT_INSTRUCTION, SYSTEM_PROMPT in YAML/JSON/TOML/Python/JS files
2. Source code — search for prompt template strings, f-strings, or .format() calls that build the system message (look in agent implementation files)
3. Trace extraction — read 3 trace files from {TRACES_FOLDER}:
   • Check info.environment_info.policy (tau2-bench format)
   • Check first message with role: "system" in the messages array
   • Check raw_data fields for system-level content
4. Not found — if none of the above yields a system prompt, explicitly record SYSTEM_PROMPT_STATUS: NOT_FOUND in the output and flag this for the orchestrator. Do not fabricate or guess.

When found, record both the prompt content and its source location (file path + line, or trace field path).

3. Extract tool definitions

Two-pass approach: source code first (ground truth), then traces (usage evidence).

Pass 1 — Source code discovery:

• Search for tool/function definition patterns: @tool, @is_tool, def tool_, function schema arrays, OpenAPI specs, tools=[] arguments
• For each tool, extract from source:
  • Name
  • Input parameters with types and defaults
  • Return type / output schema (document the structure, not just "returns a dict")
  • Side effects: READ (no state change), WRITE (mutates state), GENERIC (neither)
  • Validation rules the tool does NOT enforce (critical — grep for comments like "API does not check", "agent must enforce")

Pass 2 — Trace usage evidence:

• Read ALL traces (if <= 20) or a stratified sample (see step 4 for sampling)
• Extract every unique tool_calls[].name from assistant messages
• Extract every role: "tool" response to document actual output shapes
• For each tool, record one example input/output pair from traces

Reconcile the two passes:

• Tools in source but NOT in traces = "available but unused" — flag these; they may be relevant for edge cases the agent should handle
• Tools in traces but NOT in source = possible dynamic tools or external APIs — investigate

Output the full tool inventory as a table with columns: Name, Category, Input Schema, Output Schema, Observed in Traces (Y/N), Unvalidated Rules.