Datadog Insights

Gram Production Health Digest

You are producing a health report for Gram's production services. The report must be actionable and visually structured — critical issues must stand out immediately, tabular data must use code blocks, and every section must be separated by a divider.

Before starting: activate the datadog skill for Gram service names, MCP tools, and query guidelines.

⚠️ MANDATORY FORMAT RULES — READ BEFORE COMPOSING THE MESSAGE:

1. Every major section MUST be preceded by a Unicode divider line: ────────────────────────────────────── on its own line, with a blank line above and below.
2. Top endpoints, error type breakdowns, and latency tables MUST use triple-backtick code blocks — never bullet points for tabular data.
3. Code block tables must have aligned columns using spaces. Minimum widths: endpoint 38 chars, count 8 chars, 4xx% 6 chars, 5xx% 6 chars, p95 8 chars.
4. Each monitor in alert MUST get its own paragraph — never combine multiple monitors into one block.
5. Do NOT collapse or omit data to save space. If there are 8 monitors, show all 8.

Step 1: Check for critical issues first

These take priority over everything else. If any exist, they become the top of the digest.

1. Open incidents — search_datadog_incidents for state:(active OR stable) in the last 24h
2. Monitors in alert — search_datadog_monitors with query status:alert (notification:slack-Speakeasy-gram-oncall OR notification:slack-oncall-gram). This filters to Gram-only monitors. Never include monitors that don't notify one of these two channels.
3. Error spikes — Use analyze_datadog_logs with SQL:

   SELECT service, status, count(*) FROM logs GROUP BY service, status ORDER BY count(*) DESC

   Filter: env:prod status:(error OR critical OR alert OR emergency), last 24h. Compare the last 6h vs. the previous 18h to detect spikes.

If there are critical issues, investigate each one:

• Get a sample of the actual error logs (search_datadog_logs)
• Follow trace IDs with get_datadog_trace to find root causes
• Grep in server/internal/ for the error message to find the source code location

For top error message breakdown, use analyze_datadog_logs:

SELECT message, count(*) as cnt
FROM logs
WHERE service = 'gram-server' AND status IN ('error', 'critical')
GROUP BY message
ORDER BY cnt DESC
LIMIT 10

Step 2: Top endpoints by traffic

Use search_datadog_spans for service:gram-server env:prod over the last 24h, or:

sum:trace.http.server.request.hits{service:gram-server,env:prod} by {resource_name}.rollup(sum, 86400)

Collect the top 10 endpoints with:

• Request count
• 4xx count and rate (% of requests returning 4xx)
• 5xx count and rate (% of requests returning 5xx)
• p95 latency

Keep 4xx and 5xx separate — never fold them into a single error rate. 4xx is mostly client behaviour (bad auth, missing resources) and is expected on public endpoints like /mcp/{mcpSlug}, while 5xx indicates a server fault. Use get_datadog_metric grouped by status code and bucket the series by leading digit:

sum:trace.http.server.request.hits{service:gram-server,env:prod} by {resource_name,http.status_code}.rollup(sum, 86400)

If the metric is missing the http.status_code tag, fall back to aggregate_spans over service:gram-server env:prod grouped by resource_name, once with @http.status_code:[400 TO 499] and once with @http.status_code:[500 TO 599].

Step 3: Traffic volume and trends

Compare traffic between two 12h windows:

1. Current 12h: from: now-12h, to: now
2. Previous 12h: from: now-24h, to: now-12h

Use get_datadog_metric with:

sum:trace.http.server.request.hits{service:gram-server,env:prod}.rollup(sum, 43200)

Report:

• Total requests in the last 24h
• % change between the two 12h periods (flag if > 30% change)
• Per-service breakdown (gram-server, gram-worker, gram, fly)

Step 4: Latency analysis

p50:trace.http.server.request{service:gram-server,env:prod} by {resource_name}
p95:trace.http.server.request{service:gram-server,env:prod} by {resource_name}
p99:trace.http.server.request{service:gram-server,env:prod} by {resource_name}

Over the last 24h with .rollup(avg, 86400).

Report:

• Global latency: p50, p95, p99 across all endpoints
• Slowest 5 endpoints by p95 latency (with their p50 for comparison)
• Flag any endpoint where p95 > 2s or p99 > 5s

Step 5: Create a Datadog Notebook

Call create_datadog_notebook with name "Gram Health Digest — <DAY> <DATE>" (e.g. "Gram Health Digest — Fri 2026-03-27"). Use absolute_time: true with start_time = 24h ago and end_time = now. One notebook is created per run — old ones accumulate and can be manually deleted periodically.

The notebook cells must be wrapped in {"cells": [...]}. Include:

1. Summary markdown cell:

   {
     "type": "notebook_cells",
     "attributes": {
       "definition": {
         "type": "markdown",
         "text": "One paragraph verdict with key numbers."
       }
     }
   }

2. Error rate timeseries cell:

   {
     "type": "notebook_cells",
     "attributes": {
       "definition": {
         "type": "timeseries",
         "title": "gram-server Error Rate (1h buckets)",
         "requests": [
           {
             "q": "sum:trace.http.server.request.errors{service:gram-server,env:prod}.rollup(sum, 3600)",
             "display_type": "bars",
             "style": { "palette": "warm" }
           }
         ],
         "show_legend": true,
         "yaxis": { "scale": "linear" },
         "markers": [
           {
             "value": "y = 500",
             "display_type": "warning dashed",
             "label": "Elevated"
           }
         ]
       }
     }
   }

3. Traffic volume timeseries cell:

   {
     "type": "notebook_cells",
     "attributes": {
       "definition": {
         "type": "timeseries",
         "title": "gram-server Traffic Volume (1h buckets)",
         "requests": [
           {
             "q": "sum:trace.http.server.request.hits{service:gram-server,env:prod}.rollup(sum, 3600)",
             "display_type": "area",
             "style": { "palette": "dog_classic" }
           }
         ],
         "show_legend": true,
         "yaxis": { "scale": "linear" }
       }
     }
   }

4. p95 latency by endpoint timeseries cell:

   {
     "type": "notebook_cells",
     "attributes": {
       "definition": {
         "type": "timeseries",
         "title": "Top Endpoint p95 Latency",
         "requests": [
           {
             "q": "p95:trace.http.server.request{service:gram-server,env:prod} by {resource_name}.rollup(avg, 3600)",
             "display_type": "line",
             "style": { "palette": "dog_classic" }
           }
         ],
         "show_legend": true,
         "yaxis": { "scale": "linear" },
         "markers": [
           {
             "value": "y = 2",
             "display_type": "error dashed",
             "label": "2s threshold"
           }
         ]
       }
     }
   }

5. gram-worker error rate timeseries cell:

   {
     "type": "notebook_cells",
     "attributes": {
       "definition": {
         "type": "timeseries",
         "title": "gram-worker Error Rate (1h buckets)",
         "requests": [
           {
             "q": "sum:trace.http.server.request.errors{service:gram-worker,env:prod}.rollup(sum, 3600)",
             "display_type": "bars",
             "style": { "palette": "warm" }
           }
         ],
         "show_legend": true,
         "yaxis": { "scale": "linear" }
       }
     }
   }

6. gram (frontend) trace errors timeseries cell — gram is an APM service, so use trace metrics:

   {
     "type": "notebook_cells",
     "attributes": {
       "definition": {
         "type": "timeseries",
         "title": "gram (frontend) Trace Errors (1h buckets)",
         "requests": [
           {
             "q": "sum:trace.http.server.request.errors{service:gram,env:prod}.rollup(sum, 3600)",
             "display_type": "bars",
             "style": { "palette": "warm" }
           }
         ],
         "show_legend": true,
         "yaxis": { "scale": "linear" }
       }
     }
   }

7. fly (functions) error log stream cell — fly is a log source (not an APM service), so use a log stream, not a trace metric:

   {
     "type": "notebook_cells",
     "attributes": {
       "definition": {
         "type": "log_stream",
         "title": "fly (functions) Error Logs (24h)",
         "query": "source:fly env:prod status:error",
         "columns": ["timestamp", "host", "message"],
         "message_display": "inline",
         "show_date_column": true,
         "show_message_column": true,
         "sort": { "column": "timestamp", "order": "desc" }
       }
     }
   }

8. Slow endpoints + top errors markdown table cell with the real data from Steps 1–4.
9. All Gram services error log stream cell — includes source:fly for Gram Functions logs:

   {
     "type": "notebook_cells",
     "attributes": {
       "definition": {
         "type": "log_stream",
         "query": "(service:(gram-server OR gram-worker OR gram) OR source:fly) env:prod status:error",
         "columns": ["timestamp", "host", "service", "message"],
         "message_display": "inline",
         "show_date_column": true,
         "show_message_column": true,
         "sort": { "column": "timestamp", "order": "desc" }
       }
     }
   }

Save the notebook URL — you will link it in the Slack message footer.