Otel Queries

OTel Queries

Use this skill to inspect gh-aw OpenTelemetry/OTLP data and answer telemetry questions without re-deriving trace fields, backend filters, and diagnostics.

When To Use

Use this skill for requests such as:

• analyze OTEL or OTLP data
• inspect traces in Grafana, Tempo, Sentry, Honeycomb, or Datadog
• explain why a workflow or agent run is slow or failing
• compare run phases, error clusters, or span attributes
• identify the best observability or performance improvement
• close the loop from telemetry into code or workflow changes

Do not use this skill for instrumentation-only tasks that do not require reading telemetry. For pure emit-side work, start with the existing OTLP code and docs.

Primary Goal

Reduce a broad telemetry task to one tight loop:

1. Find the cheapest trustworthy telemetry source.
2. Run a small fixed set of common queries.
3. Confirm one concrete bottleneck, missing attribute, or broken correlation path.
4. Answer the user's telemetry question directly.
5. Recommend or implement a follow-on optimization only when the evidence supports it.

Telemetry Sources In Priority Order

Prefer sources in this order unless the user says otherwise:

1. Local artifacts or mirrors already in the workspace.
2. /tmp/gh-aw/otel.jsonl for gh-aw spans.
3. Live OTLP backend data through an MCP server or supported tool — Copilot CLI spans are exported directly to the configured OTLP backend (no local file mirror) and must be queried there, filtered by the github.run_id resource attribute.
4. Static code inspection only, when no telemetry is available.

Use the cheapest source that can disconfirm the current hypothesis.

Standard Analysis Loop

Always answer these questions in order before expanding scope.

1. Do spans exist for the run or workflow at all?

Look for:

• traceId
• span name
• service.name
• github.repository
• github.run_id

If these are missing, the problem is likely export, filtering, or trace propagation rather than optimization.

2. Is trace continuity intact?

Check whether spans that should belong together share the same:

• trace ID
• parent span lineage
• run ID
• workflow reference

If setup, agent, and conclusion spans are not connected, fix correlation before interpreting latency.

3. Which phase is actually slow or failing?

Bucket spans into phases:

• setup
• agent execution
• tool or safe-output calls
• conclusion

Prefer wall-clock duration and count by span name prefix before reading code.

4. Do the spans contain enough attributes to explain the slowdown or failure?

Minimum diagnostic attributes to verify:

• service.version
• deployment.environment
• github.repository
• github.run_id
• github.event_name
• github.workflow_ref
• gh-aw.workflow
• gh-aw.engine
• conclusion or failure attributes

If the slow or failing span lacks the attribute needed to group, filter, or explain it, the right next step may be an instrumentation change rather than a runtime change.

5. Is the problem systemic or isolated?

Check whether the pattern repeats across:

• multiple runs of the same workflow
• multiple jobs in the same trace
• one engine only
• one event type only
• one environment only

Do not propose broad architectural changes for a single outlier trace.

Common Queries

Use these backend-agnostic query shapes first. Translate them into the native query language or MCP tool calls for the active backend.

Query 1: Recent gh-aw spans

Filter for the last 24 hours and service.name = gh-aw.

Return:

• timestamp
• trace ID
• span name
• duration
• status
• github.run_id
• github.workflow_ref

Query 2: Slowest spans by name

Group by span name and sort by:

• p95 duration
• max duration
• count

Use this to find whether the bottleneck is setup, agent, tool, or conclusion work.

Query 3: Errors by span name

Filter for error status and group by:

• span name
• status message
• workflow ref
• engine

Use this to separate exporter failures from workflow logic failures.

Query 4: Missing core attributes

Sample recent spans and explicitly record whether each span includes:

• service.version
• github.repository
• github.run_id
• github.event_name
• deployment.environment

If a backend supports has or exists filters, use them. Otherwise inspect a small sample manually.

Query 5: Trace integrity for one failing run

Pick one trace ID and inspect the full trace. Record:

• root span name
• child spans present
• missing expected spans
• parent-child continuity gaps