Dt Obs Genai

AI Observability (GenAI) Skill

Analyze AI Observability signals from customer GenAI applications using DQL — golden signals, LLM signals, token and cost analytics (with usage attribution and prompt-caching economics), agent signals (including loop/runaway detection and Smartscape topology), conversation/session-level analytics, guardrails, and evaluation quality.

When to Use

Use this skill for observability questions about customer GenAI applications — anything reading OpenTelemetry GenAI spans (gen_ai.*) or LLM evaluation bizevents. Example triggers:

• "LLM latency", "error rate by model"
• "token usage by model", "token throughput / TPM", "am I hitting rate limits", "provider throttling or 429s"
• "cost by model and provider", "who is driving token spend", "do I have prompt caching"
• "cost per conversation", "most expensive sessions"
• "failing agent tool calls", "find runaway agents"
• "responses truncated or blocked", "finish reasons"
• "failed evaluations", "quality scores"

When NOT to Use

Davis CoPilot/MCP telemetry (dt-platform), generic service metrics (dt-obs-services), logs (dt-obs-logs), or non-GenAI distributed tracing (dt-obs-tracing).

Example Questions

When suggesting follow-up questions (e.g., "give me one example question per topic"), use these canonical, ready-to-ask phrasings — one per capability. Keep suggestions single-clause and avoid the literal phrase "content filter" (use "blocked or safety-filtered" instead); overly long, multi-clause questions can be rejected.

• Traffic, errors & latency: What is my LLM error rate and p95 latency by model in the last 24 hours?
• Token usage & cost: Break down token usage and cost by model and provider in the last 24 hours.
• Agent & tool activity: Show me failing agent tool calls in the last 24 hours.
• Conversation analytics: What are my top 10 most expensive conversations by total token usage in the last 24 hours?
• Guardrails: How many responses were blocked or truncated in the last 7 days, and which model was affected most?
• Evaluation quality: Show me failed evaluations from the last 24 hours with the judge's explanation and the question-answer pair.

Core Capabilities

Golden Signals

The four classic observability signals — traffic, errors, latency, and saturation — apply directly to GenAI applications. Traffic is request throughput over time; errors are spans where span.status_code == "error"; latency is the duration field (a Grail duration value — divide by the 1ms literal, duration / 1ms, for a numeric millisecond value); saturation is proxied by total token throughput per minute (input + output tokens combined).

fetch spans, from: now()-24h
| filter isNotNull(gen_ai.request.model)
| summarize total = count(), errors = countIf(span.status_code == "error"), by: {gen_ai.request.model}
| fieldsAdd error_rate_pct = if(total > 0, errors * 100.0 / total, else: 0.0)
| sort error_rate_pct desc

→ Full traffic, latency, and saturation queries: See references/golden-signals.md

LLM Signals

LLM signals describe which model and provider served each request, what operation type was invoked (chat, execute_tool, invoke_agent, create_agent), and how tokens were consumed. Use these to benchmark provider latency, compare model performance, and understand the token distribution across model-provider combinations.

fetch spans, from: now()-24h
| filter isNotNull(gen_ai.request.model)
| summarize p95_ms = percentile(duration, 95) / 1ms, requests = count(), by: {gen_ai.provider.name}
| sort p95_ms desc

→ Slowest models, token usage by model: See references/llm-signals.md

Cost and Tokens

Token consumption is the primary cost driver. Dynatrace stores gen_ai.usage.input_tokens and gen_ai.usage.output_tokens on every span — there is no stored cost field; estimated cost must be derived by multiplying token sums by the per-model price you supply. Use these queries to identify the highest-spend model-provider combinations and detect token-burn spikes.

fetch spans, from: now()-24h
| filter isNotNull(gen_ai.usage.input_tokens) or isNotNull(gen_ai.usage.output_tokens)
| summarize input_tokens = sum(gen_ai.usage.input_tokens), output_tokens = sum(gen_ai.usage.output_tokens), total_tokens = sum(gen_ai.usage.input_tokens) + sum(gen_ai.usage.output_tokens), by: {gen_ai.provider.name, gen_ai.request.model}
| sort total_tokens desc

→ Token spikes, cost estimation, most expensive prompts, usage attribution, prompt-caching economics: See references/cost-and-tokens.md

Agent Signals

GenAI agents emit spans for each tool invocation (execute_tool), agent step (invoke_agent), and agent creation (create_agent). Use agent signals to identify which tools are called most often and which agents are failing. For structural questions — which agents, models, and providers exist and how they connect — query the GenAI Smartscape entities (GENAI_AGENT, GENAI_MODEL, GENAI_PROVIDER, GENAI_SERVICE) instead of scanning spans; this is a feature-flag-gated preview.

fetch spans, from: now()-24h
| filter isNotNull(gen_ai.agent.name)
| summarize total = count(), errors = countIf(span.status_code == "error"), by: {gen_ai.agent.name}
| fieldsAdd error_rate_pct = if(total > 0, errors * 100.0 / total, else: 0.0)
| sort errors desc

→ Tool usage, failing agents, agent step latency, loop/runaway detection, Smartscape topology: See references/agent-signals.md

Conversation Analytics

Per-span and per-trace signals measure one request or one turn. When the application propagates gen_ai.conversation.id, you can roll spans up to the session level — cost per conversation, how deep conversations run, and which sessions are runaway-expensive or error-prone. This is the unit that matters for chargeback and user-perceived reliability.

fetch spans, from: now()-24h
| filter isNotNull(gen_ai.conversation.id)
| summarize turns = countDistinct(trace.id), total_tokens = sum(gen_ai.usage.input_tokens) + sum(gen_ai.usage.output_tokens), errors = countIf(span.status_code == "error"), by: {gen_ai.conversation.id}
| sort total_tokens desc

→ Cost/depth per conversation, session error rate: See references/conversation-analytics.md

Guardrails

Guardrails surface as gen_ai.response.finish_reasons on the span — content_filter means a safety filter blocked or redacted output, length means the response was truncated at the token limit — and as the proactive safety evaluators (prompt-injection, pii-leakage, toxicity, bias) in the evaluation bizevents. Use these to quantify blocked and truncated responses and tie them back to the LLM-judge safety verdicts.

fetch spans, from: now()-24h
| filter isNotNull(gen_ai.response.finish_reasons)
| fieldsAdd finish_reason = gen_ai.response.finish_reasons
| expand finish_reason
| summarize calls = count(), by: {finish_reason, gen_ai.request.model}
| sort calls desc

→ Blocked (safety-filtered), truncated (length), finish-reason breakdown: See references/guardrails.md

Evaluation Quality

Evaluation results are captured as bizevents (not spans) with event.type == "gen_ai.evaluation.result". Each evaluator emits one bizevent per response, carrying the score, pass/fail label, explanation, and the exact Q&A pair. Use evaluation queries to monitor quality dimensions and surface failed responses with the LLM judge's reasoning. Each bizevent also carries the trace.id of the run that produced the evaluated response, so you can pivot from a quality failure to the spans that caused it.

fetch bizevents, from: now()-24h
| filter event.type == "gen_ai.evaluation.result"
| filter gen_ai.evaluation.score.label == "fail"
| fields timestamp, gen_ai.evaluation.name, gen_ai.evaluation.score.value, gen_ai.evaluation.explanation, gen_ai.evaluation.input.question, gen_ai.evaluation.input.answer
| sort timestamp desc

→ Quality scores, failed evaluations, fail rates: See references/evaluations.md

Empty-State Check

When any signal query returns no rows, do not report "no data found" — first confirm whether the application sends GenAI telemetry at all. These two presence checks show which signal families are present:

fetch spans, from: now()-24h
| summarize 
    has_genai = countIf(isNotNull(gen_ai.request.model)), 
    has_tokens = countIf(isNotNull(gen_ai.usage.input_tokens) or isNotNull(gen_ai.usage.output_tokens)), 
    has_agents = countIf(isNotNull(gen_ai.agent.name)), 
    has_tools = countIf(gen_ai.operation.name == "execute_tool"), 
    has_conversation = countIf(isNotNull(gen_ai.conversation.id)), 
    has_finish_reason = countIf(isNotNull(gen_ai.response.finish_reasons)), 
    has_cached_tokens = countIf(isNotNull(gen_ai.usage.cache_read.input_tokens) or isNotNull(gen_ai.usage.cache_creation.input_tokens)),
    total = count()
    

fetch bizevents, from: now()-24h
| filter event.type == "gen_ai.evaluation.result"
| summarize evals = count()

If has_genai is zero, report that the application appears not to be instrumented for AI Observability yet — not "no data found". If has_genai is non-zero but a specific family (has_tokens, has_agents, has_tools, has_conversation, has_finish_reason, has_cached_tokens, evals) is zero, only that signal type is missing — for example has_conversation == 0 means session-level analytics are unavailable because the app does not propagate a conversation id, and has_cached_tokens == 0 means prompt-caching telemetry is not being reported. These optional families may use different attribute names depending on the provider/SDK; verify before reporting them absent.