Fix Datadog Issues

Fix Datadog Error Tracking issues

This skill takes a brand-new agent from "look at Datadog" all the way to a reviewable PR. You will run it repeatedly across separate, context-free sessions, so it is written to be restartable: a fixed error simply stops receiving occurrences, and (when the tooling allows) you leave a comment on the issue so the next agent does not redo your analysis.

The Datadog MCP server is plugin:datadog:mcp. Its tools are deferred — their schemas load on demand. Load them with ToolSearch (e.g. select:mcp__plugin_datadog_mcp__aggregate_spans) before calling, and run the server's skill-discovery first (see §1).

The pipeline (overview)

0. Setup & disambiguate  →  1. Navigate Datadog  →  2. Pick an issue  →  3. Root-cause in code
   →  4. ⛔ CONFIRM PLAN WITH USER  →  5. Reproduce with tests + fix  →  6. Comment on the issue  →  7. PR to development

Hard gate at step 4: you investigate freely, but you do not write a fix, create a branch, or open a PR until the user has seen your plan and approved it (see §4). The only exception is explicit pre-authorization (e.g. "just fix it and PR").

Do not try to fix every issue. Scope a PR by size/risk (see §2.4): one big/critical bug = its own PR; a few small, independent, related bugs may share one PR (cap ~4).

0. Setup and critical disambiguations

Read these first — each one was a real wall that cost time.

• "Incidents" almost always means Error Tracking issues, not Incident Management. Datadog Incident Management (search_datadog_incidents) is typically empty (0) here, and incidents have no "occurrences". When the user says incidents / issues / errors / "most occurrences" / "newest", they mean Error Tracking (errors grouped into issues, with occurrence counts and trends).

• Only v2 services are fixable from this repo. This repo (latitude-v2, trunk development) owns: api, ingest, web, workers, workflows. The latitude-llm-* services (latitude-llm-web, latitude-llm-workers, latitude-llm-gateway, …) are the legacy v1 codebase (branch latitude-v1) — out of scope unless the user says otherwise. Derive the live v2 list from ls apps/ so it never goes stale.

• Production only. Filter env:production. Error Tracking buckets by env; staging issues exist separately (env:staging) but are out of scope unless the user hands you a specific staging issue.

• The Latitude MCP server (mcp__latitude__*) is NOT this. mcp__latitude__listIssues / getIssue / resolveIssues operate on issues detected in Latitude customers' LLM traces — a different product surface. Never use them to triage our own app's runtime errors. Use plugin:datadog:mcp.

• Error Tracking toolset must be enabled. This skill needs the error-tracking toolset (stable, not on by default). If ToolSearch for datadog error tracking issue finds no mcp__plugin_datadog_mcp__*error_tracking* tools, the toolset is off — tell the user to enable it via the /datadog:ddtoolsets skill (add error-tracking), then /reload-plugins + re-auth. You can still do everything except read/write issue objects directly via the span fallback in §1.3.

1. Navigating Datadog

1.1 Always start with skill discovery

The server ships domain guides that are not visible in tool names. In parallel:

• load_datadog_skill(skill_name="datadog/traces") — span query syntax & attributes.
• list_datadog_skills(query="error tracking ...") — find the right guide. Load datadog/logs if you pivot to logs. Skip re-loading a guide you already loaded this session.

1.2 The lay of the land

• Org/site: datadoghq.eu (UI: app.datadoghq.eu, MCP domain mcp.datadoghq.eu).
• Services: search_datadog_services lists everything (apps, DB adapters like *-postgres/*-aws-s3, and external hosts like api.openai.com). Only the bare v2 app names matter here.
• service.version == git commit SHA. Every span tags the deployed SHA. This is gold: it maps an error to a commit and lets you correlate with deploys via get_change_stories.

1.3 Where Error Tracking issues live, and how to read them

An issue is a fingerprinted group of error occurrences. Two ways in — use both:

(a) Error Tracking tools (preferred; needs the toolset). Discover exact names at runtime: ToolSearch(query="datadog error tracking issue"). Expect at least get_datadog_error_tracking_issue (by issue id) and a list/search tool; there may be an update/state and/or comment tool — confirm their real names/schemas before relying on them (see §6).

(b) Span aggregation (always works, even with the toolset off). Issues are stamped onto error spans:

• custom.issue.id — the issue UUID (fetch with custom_attributes:["issue.*"]).
• issue.first_seen (epoch ms), issue.first_seen_version (git SHA of first occurrence), issue.age.
• Plus @error.type, @error.message, error.stack, resource_name, service, env.

Find the heavy hitters (the workhorse query):

aggregate_spans(
  query   = "status:error env:production",
  from    = "now-7d", to = "now",
  computes= [{field:"*", aggregation:"COUNT", output:"count", sort:"desc"}],
  group_by= {fields:["service","@error.type"], limit:40}
)

Then narrow into messages/resources for the candidates you care about:

aggregate_spans(query="service:workers status:error env:production @error.type:(TypeError OR RepositoryError)",
  group_by={fields:["@error.message","resource_name"], limit:25}, computes=[COUNT desc])

Read raw detail (stack, http, issue id) for a specific group:

search_datadog_spans(
  query = "service:web status:error env:production @error.type:Error resource_name:GET",
  custom_attributes = ["error.*","http.*","issue.*"], max_tokens = 7000)

Link to an issue (needed in the §4 report and the §7 PR). Prefer the canonical URL the error-tracking tool returns. Otherwise build it from the org base + issue id and open it to confirm it resolves: https://app.datadoghq.eu/error-tracking/issues/<issue.id>. (Span search responses also return a base_url and a traces_explorer_url you can fall back to.)

1.4 Query pitfalls (these bit us)

• @error.message is not reliably wildcard/full-text searchable. @error.message:"foo*" may return 0. Instead group by @error.message in aggregate_spans, or filter by @error.type + resource_name and read messages from raw spans.
• Reserved attrs take no @: service, resource_name, status, type, trace_id. Span attrs take @: @error.type, @http.status_code, @duration (nanoseconds!).
• Group multi-values: @error.type:(A OR B), not @error.type:A OR @error.type:B.

2. Pick an issue

2.1 If the user named an issue

Target it directly (by issue id/slug/url, or a quoted error message → resolve via the queries above). Skip the ranking; go to §3.

2.2 Otherwise: the funnel (classification before ranking)

Occurrence count alone is a trap. Most high-count "errors" are not fixable code bugs. Filter first.

Stage A — scope gate (cheap, 1–2 aggregate calls): v2 service, env:production, status:error, now-7d. Inspect the top ~15 by count. Ignore issues with < 5 occurrences in the window unless they are new+rising or user-specified (sub-5 are usually non-reproducible one-offs).

Stage B — classify each candidate (read its message + stack). Bucket it:

Only Class A proceeds. For B–E: leave a one-line verdict comment on the issue if you can (§6), then skip.

Stage C — rank the Class-A bugs on three axes, then pick the top:

1. Impact — customer-facing (web/api) > background (workers/workflows); data corruption/loss > transient failure; silent-wrong > loud-fail; does it block a user flow?
2. Volume × trend — occurrences and direction. Use the 14-day trend; rising/new beats flat/decaying.
3. Fix confidence × blast radius — clear, bounded root cause + small change = high ROI; sprawling or unknown = defer.

Pick = highest (impact × trend) among the confidently fixable.

2.3 The recency premium (catch regressions early)

Give extra weight to new + rising issues even at lower absolute count:

• "New" = issue.first_seen within ~72h, OR first_seen_version is one of the last 1–2 deploys.
• A recent first-seen usually means a fresh regression — first_seen_version + get_change_stories often hand you the culprit commit, making the fix faster and higher-confidence, and catching it early prevents pile-up. An ancient, flat, high-count issue is a yellow flag, not an automatic top pick.

2.4 How many to fix / how to batch

• Solo PR if: critical or large; touches core/shared code; or the root cause is non-trivial.
• Group 2–4 into one PR only if all are: small, independent, low-risk, and thematically related (same subsystem → one reviewer context). Never mix a risky fix with trivial ones. Cap ~4.

3. Root-cause in code

Use the analyze-problem skill's method. Then, specific to this workflow:

• Generalize past the observed symptom. One issue is often one instance of a broader bug. Example: a Voyage embeddings 400 invalid UTF-8 and a ClickHouse missing second part of surrogate pair were one root cause — unsanitized lone UTF-16 surrogates hitting two sinks. Fix the source, not each sink, and look for sibling call sites with the same flaw.
• Map the error to code from the span: service → apps/<svc>; resource_name (e.g. bullmq process <queue>, or GET /…) → the handler/job; error.stack frames → the throwing module. Confirm the deployed SHA (service.version) matches what you're reading.
• Find the commit that introduced it. issue.first_seen_version is the git SHA of the first occurrence — the deploy that introduced the regression. Once you've located the faulty line(s), run git blame/git log -S '<symbol>' -- <file> to name the culprit commit, and get_change_stories to see that deploy in context. Capture both the file:line and the commit SHA — §4 asks for them.
• Decide fixability honestly. If it's Class B/C/D/E in disguise, or the fix needs product/infra decisions beyond code, say so and record it (§6) instead of forcing a fake fix.
• Respect the architecture (architecture-boundaries): fix at the right layer (domain use-case vs platform adapter vs app boundary). Prefer the layer the codebase already uses for that concern — search for an existing helper before writing a new one.