Telemetry Analyzer

Agent Telemetry Analyzer

This skill exports agent telemetry traces from watsonx Orchestrate and analyzes them directly from raw OTel JSON or Langfuse-format trace JSON to produce structured bug reports with root-cause analysis and fix recommendations.

Two trace formats are supported:

• OTel JSON — produced by the classic TracesController.export_trace_to_json() ADK path and local IBM telemetry servers. Top-level key: traceData.resourceSpans.
• Langfuse JSON — produced by the agentops-v3 REST API (GET /v1/agentops-v3/traces/<id>) and the export_traces_agentops_v3.py script. Top-level key: observations (array of GENERATION / CHAIN / SPAN objects).

Always detect the format first before reading any span data. See the detect_format() helper in Step 3C and the normalization table in Step 4.

Setup: See README.md for prerequisites (ADK installation, .env configuration, and environment activation) before running any scripts.

What this skill does

1. Identifies the trace source — remote hosted env (wxO Traces API or agentops-v3 REST API), local env with IBM telemetry, or JSON files already on disk (OTel or Langfuse format)
2. Exports trace files — discovers traces by agent name or ID, paginates through results, and downloads them as OTel JSON (via ADK TracesController) or Langfuse JSON (via export_traces_agentops_v3.py / search_traces_agentops_v3.py)
3. Analyzes for bugs — detects the JSON format, normalizes spans to a common structure, and scans for hard errors, LLM failures, tool call failures, agent logic bugs, flow issues, token anomalies, and cache efficiency
4. Produces a bug report — structured HTML report with an executive summary, trace summary table, critical issues, warnings, observations, and fix recommendations (multi-trace analysis only — a single-trace request goes straight to the conversational flow report)
5. Produces conversational flow reports — thread-scoped reports that reconstruct multi-turn conversations and surface per-turn failures

When to use this skill

Use this skill when you need to:

• Debug a failing or misbehaving watsonx Orchestrate agent
• Understand why an agent returned an empty or incorrect response
• Trace LLM token usage and identify prompt bloat or context overflow
• Identify tool call failures, retry loops, or hung async flows
• Analyze a batch of traces and surface the most impactful issues
• Reconstruct a full multi-turn conversation from thread-scoped traces

How to invoke

Ask questions like:

• "Analyze the telemetry for the My Agent traces from the last 20 minutes"
• "What went wrong in trace f492e71b957ccec2d07096ae99395d19?"
• "Export and inspect all traces for the Supervisor agent in the last 2 hours"
• "Why did this agent return an empty response?"
• "Show me the conversational flow for this trace"

Command-line usage

Standard scripts (_adk.py) — wxO Traces API / local IBM telemetry

# Search traces and save IDs for bulk export
python scripts/search_traces_adk.py --agent-name "My Agent" --last 1h --all \
    --save-ids data/trace_ids.txt

# Export all discovered traces
python scripts/export_traces_adk.py --ids-file data/trace_ids.txt

# Export a single trace by ID
python scripts/export_traces_adk.py --trace-id <32-char-hex-id>

agentops-v3 scripts (_agentops_v3.py) — agentops-v3 REST API

# Search traces for an agent (last 20 minutes by default)
python scripts/search_traces_agentops_v3.py --last 20m

# Narrow or widen the time window
python scripts/search_traces_agentops_v3.py --last 2h
python scripts/search_traces_agentops_v3.py --start 2025-01-01T09:00:00Z --end 2025-01-01T21:00:00Z

# Collect every trace in the window and save IDs for bulk export
python scripts/search_traces_agentops_v3.py --last 1h --all \
    --save-ids data/trace_ids.txt

# Quick count before committing to a full export
python scripts/search_traces_agentops_v3.py --last 2h --count

# Export a single trace by ID
python scripts/export_traces_agentops_v3.py --trace-id <32-char-hex-id>

# Bulk-export from a saved ID list
python scripts/export_traces_agentops_v3.py --ids-file data/trace_ids.txt

Analysis capabilities

Trace export and discovery

• Cursor-based pagination through the wxO Traces API with automatic 429 retry
• Supports --limit N, --all, and --count modes for controlling result volume
• Saves trace files as data/<trace_id>.json — OTel format via export_traces_adk.py, Langfuse format via export_traces_agentops_v3.py
• Works with remote hosted environments and local servers started with --with-ibm-telemetry
• search_traces_agentops_v3.py / export_traces_agentops_v3.py use the raw agentops-v3 REST API (bearer token from active env, no ADK SDK calls)

Bug detection

Detected from normalized spans, supporting both OTel (traceData.resourceSpans[0].scopeSpans[0].spans) and Langfuse (observations[]) formats:

• Hard errors — any span with status.code == STATUS_CODE_ERROR
• LLM failures — finish_reason == "length", empty answer.task output, zero output tokens (via llm.token_count.completion / gen_ai.usage.output_tokens attributes)
• Tool call failures — empty ToolMessage content in *.tool spans, retry loops, duration_ms > 30 000 ms
• Agent logic bugs — missing answer.task span, empty final answer, repeated agent.task cycles (multi-step loop), empty collaborator output
• Flow issues — has_error on any span, widget_result.task or collaborator.task with empty output
• Token anomalies — cumulative token totals exceeding 50 000, high input:output token ratio
• Cache efficiency — total cache-read tokens, cache hit rate below 20 % on multi-turn threads, zero cache reads across ≥ 3-turn threads, cache-write/read imbalance

Conversational flow analysis

• Detects trace format and extracts thread_id from the correct location:
  • OTel: traceloop.association.properties.thread_id or thread.id span attribute
  • Langfuse: metadata.attributes.thread_id or sessionId top-level field
• Searches the local trace cache and the API for all traces belonging to a thread
• Reconstructs user message, routing decision, collaborator steps, and final response per turn
• Flags turns where the agent returned no response or encountered an error

Output format

The analyzer produces:

1. Bug Report (HTML artifact for 3+ issues, inline for fewer)

   • Report header with agent ID, name, environment, date, and UTC time window
   • Executive Summary with KPI stat cards (for 10+ traces)
   • Categorical trace summary table grouped by routing pattern or error type
   • Per-trace detail table with start time, span count, token counts, and status
   • 🔴 Critical Issues, 🟠 Warnings, and 🟡 Observations tables with affected trace IDs
   • 🛠️ Fix Recommendations with root cause and actionable steps per issue

2. Conversational Flow Report (HTML artifact, thread-scoped)

   • Thread metadata: thread ID, turn count, agent, environment, time window, overall outcome
   • Per-turn timeline: user message, routing decision, collaborator steps, response delivered, token counts, error flags
   • Full span timing breakdown across all turns in the thread
   • Root cause summary comparing what worked vs. what failed

Supporting files

Scripts in scripts/ (created on first use by Step 1 if not already present):

Use the *_agentops_v3.py scripts when the environment uses the agentops-v3 REST API. Use the standard scripts for environments backed by the classic wxO Traces API.

Step-by-step workflow

Step 1: Verify helper scripts are present

File placement rules — apply throughout all steps:

• All Python scripts written to disk must be saved under scripts/ (e.g. scripts/my_helper.py).
• All output files (.json trace dumps, .txt ID lists, etc.) must be saved under data/ (e.g. data/trace_ids.txt, data/<trace_id>.json).
• Inline code snippets shown in this document are for reading and reasoning only — they are not written to disk unless explicitly instructed.

Before doing anything else, confirm the required scripts exist on disk:

scripts/search_traces_adk.py
scripts/export_traces_adk.py
scripts/search_traces_agentops_v3.py
scripts/export_traces_agentops_v3.py

If any are missing, use write_file to create them — the canonical source for each script is in the scripts/ directory of this skill. Write each missing file to its correct path under scripts/ before continuing to Step 2.

Step 2: Determine the trace source

If the user's request does not clearly state where the traces or agent reside, always use ask_followup_question to clarify before doing anything else. Do not assume a source or proceed to Step 3 without a confirmed answer to all of the following that are not already clear from context:

• Is the environment remote (a hosted wxO instance) or local (a locally running server)?
• If remote — what is the environment name (e.g. the name used with orchestrate env activate)?
• If the user mentioned an agent — what is the agent name or ID?
• What time window should be searched (or should it default to the last 20 minutes)?

Once the source is confirmed, classify it as one of:

1. Remote env — a hosted watsonx Orchestrate environment activated with orchestrate env activate. Always try the _agentops_v3.py scripts first; fall back to the _adk.py scripts if the agentops-v3 search returns an error or zero traces. See Step 3A for the full fallback procedure.
2. Local env — IBM telemetry profile — a locally running orchestrate server started with --with-ibm-telemetry. The Traces API is available at http://localhost:4321. Use the _adk.py scripts directly — agentops-v3 is not available on local environments.
3. Trace files on disk — the user already has one or more {trace_id}.json files on disk ready to analyze. Confirm they are in the data/ directory; if not, ask the user for the path before proceeding.

Step 3: Export the trace files

Option A — Remote env (agentops-v3 first, ADK fallback)

For all remote environments, always try _agentops_v3.py first. Fall back to _adk.py only if the agentops-v3 search fails or returns zero traces.

Step A1 — Try agentops-v3:

# 1. Activate the target environment
orchestrate env activate <env-name>

# 2. Search traces via agentops-v3
python scripts/search_traces_agentops_v3.py \
    --start 2025-01-01T09:00:00Z --end 2025-01-01T21:00:00Z \
    --all --save-ids data/trace_ids.txt

If the command exits with a non-zero status or prints Found 0 trace(s), proceed to Step A2. Otherwise, export the discovered traces:

# 3. Export all discovered traces
python scripts/export_traces_agentops_v3.py --ids-file data/trace_ids.txt

Step A2 — Fall back to ADK (only if agentops-v3 failed or returned 0 traces):

# 2. Search traces via ADK — IDs checkpointed to disk after every page
python scripts/search_traces_adk.py --agent-name "My Agent" \
    --start 2025-01-01T09:00:00Z --end 2025-01-01T21:00:00Z \
    --all --save-ids data/trace_ids.txt

# 3. Export all discovered traces
python scripts/export_traces_adk.py --ids-file data/trace_ids.txt

search_traces_adk.py uses cursor-based pagination, retries automatically on 429 rate-limit responses, and handles mid-run 401 token expiry by refreshing the client. Always pass --save-ids so trace IDs are checkpointed to disk page-by-page and are not lost on timeout.

Use --limit N (default 50) for a capped sample, --all to exhaust the full window, or --count for a quick volume figure before committing to a full export.

Option B — Local env (IBM telemetry profile)

# 1. Start the local server with IBM telemetry enabled
orchestrate server start --with-ibm-telemetry --accept-terms-and-conditions

# 2. Activate the local environment
orchestrate env activate local

# 3. Search and export — identical to Option A
python scripts/search_traces_adk.py --agent-name "My Agent" --last 30m --all \
    --save-ids data/trace_ids.txt
python scripts/export_traces_adk.py --ids-file data/trace_ids.txt

The local Traces API endpoint is http://localhost:4321. The scripts detect is_local_dev() automatically and set service_names=["wxo-server"] in the filter, which is required when FORCE_SINGLE_TENANT=true.

Option C — Trace files already on disk

Confirm the directory or file path with the user. Always detect the format before reading spans. Use this helper:

import json

def detect_format(raw: dict) -> str:
    """Return 'otel' or 'langfuse'."""
    if isinstance(raw.get('traceData'), dict) and 'resourceSpans' in raw['traceData']:
        return 'otel'
    if 'resourceSpans' in raw:  # traceData sometimes omitted at root
        return 'otel'
    if 'observations' in raw:   # Langfuse / agentops-v3 REST response
        return 'langfuse'
    return 'langfuse'           # safe default for agentops-v3 output

def get_spans(raw: dict) -> list:
    """Return a flat list of span/observation dicts regardless of format."""
    fmt = detect_format(raw)
    if fmt == 'otel':
        td = raw.get('traceData') or raw
        return [
            s
            for rs in td.get('resourceSpans', [])
            for ss in rs.get('scopeSpans', [])
            for s in ss.get('spans', [])
        ]
    return raw.get('observations', [])

Load a file and get its spans:

with open("data/<trace_id>.json") as f:
    raw = json.load(f)
fmt   = detect_format(raw)
spans = get_spans(raw)
print(f"Format: {fmt}, span/observation count: {len(spans)}")

If the user provides a directory, glob for *.json files and skip any files not matching the 32-character hex trace ID pattern.

Step 4: Analyze raw spans for bugs

Single-trace shortcut: If the user has asked to analyze exactly one trace (by ID or as a single file on disk), skip Steps 4 and 5 entirely. Jump straight to Step 7 to produce the conversational flow report for the thread that trace belongs to. Only return to Step 5 (bug report) if the user explicitly asks for one after seeing the flow report.

Load each trace file using detect_format() + get_spans() from Step 3C, then normalize every span with the adapter below before running any bug-detection logic.

4.1 Format normalization

from datetime import datetime, timezone

def normalize_span(s: dict, fmt: str) -> dict:
    """
    Return a uniform dict with these keys regardless of source format:
      name, start_utc, duration_ms, status_error (bool),
      input, output, attrs (dict: key -> scalar value)
    """
    if fmt == 'otel':
        start_ns  = int(s.get('startTimeUnixNano', 0))
        end_ns    = int(s.get('endTimeUnixNano', 0))
        start_utc = datetime.fromtimestamp(start_ns / 1e9, tz=timezone.utc)
        duration_ms = (end_ns - start_ns) / 1e6
        status_error = s.get('status', {}).get('code') in ('STATUS_CODE_ERROR', 2)
        attrs = {}
        for a in s.get('attributes', []):
            v = a['value']
            attrs[a['key']] = (
                v.get('stringValue') or v.get('intValue') or
                v.get('doubleValue') or v.get('boolValue')
            )
        input_val  = attrs.get('traceloop.entity.input')
        output_val = attrs.get('traceloop.entity.output')
    else:  # langfuse
        start_utc = datetime.fromisoformat(
            s.get('startTime', '1970-01-01T00:00:00Z').replace('Z', '+00:00'))
        end_str   = s.get('endTime') or s.get('startTime', '1970-01-01T00:00:00Z')
        end_dt    = datetime.fromisoformat(end_str.replace('Z', '+00:00'))
        duration_ms = (end_dt - start_utc).total_seconds() * 1000
        status_error = (s.get('level') == 'ERROR' or bool(s.get('statusMessage')))
        meta  = s.get('metadata') or {}
        attrs = dict(meta.get('attributes', {}))
        usage = s.get('usage') or {}
        if usage.get('input'):
            attrs['llm.usage.prompt_tokens'] = usage['input']
        if usage.get('output'):
            attrs['llm.usage.completion_tokens'] = usage['output']
        input_val  = s.get('input')
        output_val = s.get('output')

    return dict(
        name=s.get('name', ''),
        start_utc=start_utc,
        duration_ms=duration_ms,
        status_error=status_error,
        input=input_val,
        output=output_val,
        attrs=attrs,
    )

def get_attr(attrs: dict, key: str):
    """Read a scalar from the normalized attrs dict (works for both formats)."""
    return attrs.get(key)

Build the normalized span list at the start of every analysis:

fmt    = detect_format(raw)
spans  = get_spans(raw)
nspans = [normalize_span(s, fmt) for s in spans]

llm_spans  = [n for n in nspans if n['name'] == 'WatsonxChatModel.chat']
tool_spans  = [n for n in nspans if 'collaborator' in n['name'].lower()
               or n['attrs'].get('openinference.span.kind') == 'TOOL']

Field mapping reference — use this table when reading span data:

For each trace collect: start_utc, span names, duration_ms, status_error, collaborator tool call output, final answer output, thread_id (from attrs), and cache token counts.

Look for the following bug categories. For each issue found, record the span name, start_utc, and relevant attrs values.

4.2 Hard errors

• Any normalized span where status_error == True.
  • OTel source: status.code == "STATUS_CODE_ERROR" or 2.
  • Langfuse source: level == "ERROR" or statusMessage is non-empty.
• Capture span name, duration_ms, and the output field for the error message.

4.3 LLM failures

• get_attr(n['attrs'], 'llm.finish_reason') == "length" — context length exceeded (works for both formats after normalization).
• WatsonxChatModel.chat span with duration_ms == 0 or absent — silent LLM failure.
• Token counts: use llm.token_count.prompt / llm.token_count.completion (OTel attrs) or llm.usage.prompt_tokens / llm.usage.completion_tokens (Langfuse, auto-added by normalize_span). Zero completion tokens with non-zero prompt tokens = failed generation.
• Unexpectedly high prompt token count — possible system prompt bloat.
• Cache token counts: see §4.8. After normalization, try get_attr(attrs, 'gen_ai.usage.cache_read_input_tokens') first; fall back to the raw Langfuse usage dict. If absent on all LLM spans, note as an observation (provider may not support caching).

4.4 Tool / collaborator call failures

• Tool span / SPAN observation whose output field contains an empty or null content (i.e. ToolMessage.content == "").
  • Langfuse note: an empty content on the immediate chat_with_collaborator_* SPAN is the normal async dispatch pattern — the real result arrives via the nested collaborator CHAIN observation. Only flag as a failure if the outer collaborator CHAIN output is also empty.
• Any tool span with duration_ms > 30 000 ms — timeout or hung call.
• Same collaborator tool called twice in a row — retry loop.

4.5 Agent logic bugs

• No span/observation named answer (or answer.task in OTel) — agent never produced a final answer.
• answer span output is empty or whitespace-only.
• agent (or agent.task) span count > 2 — multi-step LLM loop (agent re-planning instead of delegating immediately).
• collaborator (or collaborator.task) span present but answer output is empty — collaborator returned nothing and supervisor silently swallowed it.
• Internal reasoning text present in answer output or output.additional_kwargs.reasoning before the user-facing reply — check both OTel and Langfuse paths.

4.6 Flow / widget issues

• Span/observation named widget_result (Langfuse) or widget_result.task (OTel) with empty output — widget state machine stalled.
• LangGraph GENERATION (Langfuse) or LangGraph.workflow span (OTel) with duration_ms >> sum of child span durations — unexplained gap in the workflow.

4.7 Token / cost anomalies

• Cumulative prompt + completion tokens across all WatsonxChatModel.chat spans exceeds 50 000.
• Prompt tokens >> completion tokens (ratio > 200:1) — system prompt dominates context, little room for conversation history.