Agent Observability Trace Rca

Backend

Detection — At the start of every invocation, before taking any action, determine which backend to use:

1. If the user passed --backend pup anywhere in their invocation → use pup mode immediately, regardless of whether MCP tools are present. Skip steps 2–4.
2. Check whether MCP tools are present in your active tool list. The canonical signal is whether mcp__datadog-llmo-mcp__list_llmobs_evals appears in your available tools.
3. If MCP tools are present → use MCP mode throughout. Call MCP tools exactly as named in this skill's workflow sections.
4. If MCP tools are absent → check whether pup is executable: run pup --version via Bash. A JSON response containing "version" confirms pup is available.
5. If pup responds → use pup mode throughout. Translate every MCP tool call to its pup equivalent using the Tool Reference appendix at the bottom of this file.
6. If neither is available → stop and tell the user:

   "Neither the Datadog MCP server nor the pup CLI is available. Connect the MCP server (claude mcp add --scope user --transport http datadog-llmo-mcp 'https://mcp.datadoghq.com/api/unstable/mcp-server/mcp?toolsets=llmobs') or install pup."

--backend pup is accepted anywhere in the invocation arguments and is stripped before passing remaining args to the skill logic.

pup invocation rules:

• Invoke via Bash: pup llm-obs <subcommand> [flags]
• pup always outputs JSON. Parse directly — no content-block unwrapping (unlike MCP results, which may wrap JSON in [{"type": "text", "text": "<json>"}]).
• If pup returns an auth error, tell the user to run pup auth login and stop.
• Parallelization: issue multiple Bash tool calls in a single message (one pup command per call).
• Time flags: pup accepts bare duration strings (1h, 7d, 30m) and RFC3339 timestamps. Do not use now--prefixed strings — strip the prefix when converting from a skill --timeframe argument: now-7d → 7d, now-24h → 24h, now-30d → 30d.
• --summary on pup llm-obs spans search strips payload fields to essential metadata only. Use it in bulk/search phases where content is not needed.

Invocation ID: At the very start of each invocation, before any MCP tool call, generate an 8-character hex invocation ID (e.g., 3a9f1c2b). Keep it constant for the entire invocation.

Intent tagging: On every MCP tool call, prefix telemetry.intent with skill:agent-observability-trace-rca[<inv_id>] — followed by a description of why the tool is being called. On the first MCP tool call only, use skill:agent-observability-trace-rca:start[<inv_id>] — instead (note the :start suffix). Example first call: skill:agent-observability-trace-rca:start[3a9f1c2b] — Phase 0: discover configured evals for task-cruncher to infer analysis mode

Agent Observability Trace RCA — Root Cause Analysis from Production LLM Traces

Diagnose why an LLM application is failing by searching production traces and walking the span tree from symptom to root cause. The skill automatically selects the best analysis mode based on available signals:

The mode is announced (never asked) in the first user-facing output with a one-line override hint.

Methodology

Resolve → Search → Observe → Open Coding → Axial Coding → Root Cause Navigation → Recommendations

Usage

What's wrong with <ml_app> over the last <timeframe>
Why is <ml_app> failing today
Analyze eval failures for <eval_name> on <ml_app>
Look at the errors on <ml_app> over the last <timeframe>
Root-cause low scores on <eval_name>

Inputs

If neither ml_app nor eval_name is provided, ask the user.

Available Tools

Eval discovery & overview

Trace & span exploration

Key get_llmobs_span_content patterns

How to use search_llmobs_spans

Always include @ml_app:"<ml_app>" in the query string — the structured ml_app parameter is unreliable and can return spans from other apps. Do not rely on the structured parameter alone.

Useful query fragments — combine with space (AND):

Dedicated params (span_kind, root_spans_only, ml_app) work alongside query, but query takes precedence over tags.

Parallelization rules

1. get_llmobs_span_details: Group span_ids by trace_id, chunk each trace's span_ids into batches of at most 20. Issue ALL chunks for a page in a single message.
2. get_llmobs_span_content: Each call is independent — always issue ALL in a single message.
3. get_llmobs_trace / find_llmobs_error_spans / get_llmobs_agent_loop: Parallelize across different traces in a single message.
4. Pipeline parallelism: Start get_llmobs_span_details for page 1 results immediately — don't wait to collect all pages.

Analysis Workflow

Output discipline: Phases 0–5 are internal analysis. The only user-facing outputs during these phases are the Phase 1 Signal Summary and the mandatory checkpoints at Phases 2 and 3. Do NOT narrate reasoning, summarize intermediate findings, or output Phase 4 deep-dive results as prose. All detailed findings go exclusively into the Phase 6 report.

Phase 0: Resolve Inputs & Infer Mode

First: check for classification context. Scan the conversation for a # Session Classification Summary header. If found → enter Step 0S below and skip all remaining Phase 0 steps and Phase 1 entirely.

Step 0S — Extract Failure Bucket from Classification Output

The canonical handoff format is the Per-Unit Details table inside the # Session Classification Summary section. Extract one row per unit:

Failure bucket = all rows where verdict is no or partial.

• < 5 entries → note low confidence, proceed anyway.
• Empty → report "No failures found in the classification output" and stop.

Present this overview before proceeding:

## Classification Overview (from agent-observability-session-classify)

**ml_app**: <from summary header>  |  **Classified**: N  |  **Failures (no+partial)**: F  |  **Pass rate**: X%

| Failure Mode | Count |
|---|---|
...

Proceeding to Phase 2 using F failure traces. Mode inference bypassed — classification verdict is the signal.

Then skip Phase 1 and jump directly to Phase 2. Carry forward:

• Phase 2 reasoning input: (trace_id, span_id, detail) tuples — same structure as eval reasoning or error messages
• Phase 4 navigation: use app_type from each trace block to choose the span navigation strategy
• Phases 2–7: run completely unchanged — the failure bucket structure is identical regardless of source

Standard resolution (no classification context):

1. If neither ml_app nor eval_name provided → ask the user. If eval_name is provided but ml_app is not → also ask for ml_app (eval names are not globally unique; without it, span searches return results from all apps sharing the eval name).
2. If timeframe not provided → default to now-24h.
3. Resolve failure_filter (before mode inference):
   • "errors" → force Error Signal mode
   • "low scores on <eval>" → treat as eval_name=<eval>, then continue inference
   • "high latency" → note for Phase 1 (sort by duration post-fetch); continue inference
   • Tool/span name → note as @name:<x> query fragment for Phase 1; continue inference
4. Resolve mode (skip if mode was explicitly provided):
   • eval_name given → Eval Signal
   • User explicitly mentioned errors/exceptions/crashes → Error Signal
   • Otherwise → call list_llmobs_evals_by_ml_app(ml_app):
     • Evals returned → Eval Signal
     • No evals → Error Signal (announce auto-selection in Phase 1)
5. When eval_name is multi-valued, note for Phase 1: run parallel per-eval searches and merge+dedup by (trace_id, span_id).

Phase 1: Find Problematic Spans

Three mode-specific paths. All end with a Signal Summary that labels the mode and includes a one-line override hint.

Mode switch handling: At any checkpoint, if the user says "switch to [error|eval|generic] mode", re-enter Phase 1 with the new mode. Phase 0 inputs do not re-resolve.

Auto-pivot: If the selected mode finds no data (0 evals configured, 0 error spans in timeframe), announce the pivot to Generic and proceed — do not stop and ask.

Eval Signal path

Step 1a: Eval overview (parallel)

For each eval, call both in a single parallel batch:

• get_llmobs_eval_aggregate_stats(eval_name, from, to)
• get_llmobs_evaluator(eval_name)

Interpret aggregate stats:

• total_count == 0 → Note "no data." Skip this eval (or pivot to Generic if it's the only one).
• Boolean pass_rate == 1.0 → Note "100% pass." Skip unless it's the only eval.
• Boolean with failures → Note counts and pass_rate. Continue.
• Score with assessment criteria → Note distribution and pass/fail counts. Continue.
• Score WITHOUT assessment criteria → Infer failures: bottom quartile, or below median if bimodal. Label as "inferred failures" in report.
• Categorical with assessment criteria → Note top_values and pass/fail. Continue.
• Categorical WITHOUT assessment criteria → Infer from context (e.g., "error", "incomplete", "off_topic" are likely failures). Ask user if genuinely ambiguous.

Interpret eval config:

• Config returned (custom) → Store prompt_template, assessment_criteria, parsing_type, output_schema.
• Config nil (OOTB) → Note prompt is not inspectable.

Calibration cross-check: When two evals share a name prefix but differ in type (e.g. foo-boolean and foo-score), compare their pass rates on overlapping spans. A discrepancy >20% is an Evaluator Calibration Discrepancy — flag it in the report.

Step 1b: Collect failure spans

For each eval:

1. search_llmobs_spans(query="@evaluations.custom.<eval_name>:*", from, limit=50). When multi-valued, issue one search per eval in parallel — merge result sets, dedup by (trace_id, span_id).
2. Paginate until ≥15–20 failures OR no more pages. Cap at 200 spans total.
3. get_llmobs_span_details per trace_id batch (follow Parallelization Rules).
4. Extract per row: assessment, value, reasoning, span_id, trace_id, span_kind, content_info.
5. Separate into pass/fail buckets using thresholds from Step 1a.

JSON-type eval fallback: If @evaluations.custom.<eval_name>:* returns 0 spans but get_llmobs_eval_aggregate_stats confirmed total_count > 0, the eval is JSON-type and scores are not indexed on this field. Fall back to: search by the span name or span kind that the eval targets (check get_llmobs_evaluator for the span filter), then inspect output payloads for JSON verdict fields via get_llmobs_span_content(field="output").

Step 1c: Signal Summary (Eval Signal)

## Signal Summary: `{ml_app}` · Eval Signal
(Inferred from {N} configured eval(s). Say `switch to error mode` or `switch to generic mode` to change.)

**Timeframe**: {from} → {to}

| Eval | Type | Total | Pass Rate | Status |
|------|------|------:|:---------:|--------|
| eval_1 | boolean | 4,891 | 37.3% | ⚠ Investigating |
| eval_2 | score | 1,200 | — (inferred threshold) | ⚠ Investigating |
| eval_3 | boolean | 500 | 99.2% | ✓ Healthy |

**Collected**: {pass_count} passing, {fail_count} failing.

For a single eval, collapse to a single-line header instead of a table.

Error Signal path

Step 1a: Sample error spans

search_llmobs_spans(query="@ml_app:\"<ml_app>\" @status:error", from=timeframe, limit=50). Paginate until ≥30 error spans or no more pages.

Step 1a.5: Soft error scan

MCP tool spans sometimes report @status:ok but carry "isError": true in their output payload — these are invisible to @status:error queries and can outnumber hard errors.

Call search_llmobs_spans(query="@ml_app:\"<ml_app>\" @status:ok", span_kind="tool", from=timeframe, limit=20). For a sample of 5–10 results, call get_llmobs_span_content(field="output") in parallel. If any payloads contain "isError": true, add MCP soft errors as a separate row in the error frequency table with the note: (status:ok but isError:true in payload — not queryable via @status:error).

Step 1b: Group by error type

Group spans by error_type tag → frequency table. If error_type tag is absent on some spans, supplement with the error.type field from get_llmobs_span_details (fetched in Step 1d).

Step 1c: Fetch stack traces (parallel)

For the top 3–4 error types by count, pick 2–3 representative trace IDs each. Call find_llmobs_error_spans(trace_id) in parallel across all selected traces. Extract:

• Error message and stack trace
• Origin span kind and name
• Whether errors propagate from children to parents (cascade) or are isolated

Step 1d: Fetch span details

get_llmobs_span_details on representative spans for each error type (follow Parallelization Rules). Extract content_info, span_kind, duration.

Step 1e: Signal Summary (Error Signal)

## Signal Summary: `{ml_app}` · Error Signal
(No evals configured — analyzing runtime errors. Say `switch to eval mode` or `switch to generic mode` to change.)

**Timeframe**: {from} → {to}  |  **Total error spans sampled**: {N}

| Error Type | Spans | Cascade? | Origin Span Kind |
|------------|------:|:--------:|-----------------|
| TimeoutError | 42 | Yes | tool |
| APIError 429 | 18 | No | tool |
| ValueError | 7 | No | llm |
| MCP soft errors (isError:true) | 23 | No | tool |

Generic path

Step 1a: Eval health check (when evals are configured)

If list_llmobs_evals returned evals in Phase 0, call get_llmobs_eval_aggregate_stats for each enabled eval in parallel. Flag any enabled eval with total_count: 0 as Broken Eval Configuration — include in the Signal Summary anomaly table as a High severity row.

Step 1b: Broad span search

search_llmobs_spans(query="@ml_app:\"<ml_app>\"", root_spans_only=true, from=timeframe, limit=50). Apply failure_filter narrowing if present (tool/span name → @name:<x> query; "high latency" → sort result set by duration after Step 1c). Paginate until ≥30 spans.

Step 1c: Fetch span details

get_llmobs_span_details per trace_id batch.

Step 1d: Rank by structural anomalies

Partition spans using heuristics:

• Top decile by duration (latency outliers)
• Agent spans with >N tool/LLM iterations (long-running loops)
• Retrieval spans returning 0 documents (RAG miss)
• Workflow spans whose child set is missing an expected step (compare against median child layout)
• Token efficiency: Check if non_cached_input_tokens ≈ input_tokens across LLM spans. If the app has stable system prompts (>1k tokens) and cache hit rate is 0%, flag as High severity — enabling cache_control: ephemeral on the system prompt would cut input token costs by 60–90%

Step 1e: Signal Summary (Generic)

## Signal Summary: `{ml_app}` · Generic
(Analyzing structural anomalies. Say `switch to eval mode` or `switch to error mode` to change.)

**Timeframe**: {from} → {to}  |  **Sampled**: {N} root spans

| Anomaly Type | Count |
|---|:---:|
| Latency outliers (>p90) | 12 |
| Long agent loops (>8 iterations) | 5 |
| RAG retrieval misses | 3 |
| Zero prompt cache utilization | All LLM spans |
| Broken eval configurations | 2 |

Phase 1.5: Determine App Profile & Where the Root Cause Lives

Inspect content_info and span_kind across collected spans. Drives Phase 4 strategy.

App profile (from content_info):

LLM Experiments traces: If root spans have span_kind: experiment and carry input, output, and expected_output structured fields, you are looking at a Datadog LLM Experiments trace. Each span represents one dataset record run. Read quality signal from the root span's input/output/expected_output fields via get_llmobs_span_content — not from LLM sub-span messages, which may contain stub or placeholder content. Evaluations attached to experiment spans are computed by the Experiments framework at run time and may not be registered as online Datadog evaluators (get_llmobs_evaluator will return 404 for them).

Where the root cause likely lives — by symptom span kind:

Key insight: The signal — eval verdict, error message, latency outlier — flags one span in isolation. It's a symptom report, not a diagnosis. The root cause often lives in a different span: a parent that gave bad instructions, a sibling that provided bad context, or a child that made a wrong decision. Phase 4 navigates the tree to find it.