Observability Triage

Observability triage

Query Cloudflare Workers Observability for prod errors, count them correctly, and skip the noise that has already been investigated to a dead end. Apply the known-noise list below without re-investigating those entries.

Access

• Resolve the account at runtime — never hardcode it: the cloudflare-api MCP server pre-binds accountId in mcp__cloudflare-api__execute, and npx wrangler whoami prints it. The workers to triage are the ones this repo deploys (see deploy/alchemy/alchemy.run.ts): the main app worker plus the aux workers (audit engine, landing, self-host).
• Query via the cloudflare-api MCP server (mcp__cloudflare-api__execute). If its tools are absent, run its authenticate flow and give the user the URL — wrangler's OAuth token gets a 403 on the observability API (missing scope), so don't burn time on curl-with-wrangler-token.
• PostHog is the second error source but cannot see exceededMemory / canceled / responseStreamDisconnected outcomes — worker-outcome questions are answerable only here.

Query recipes (verified shapes)

POST /accounts/{account_id}/workers/observability/telemetry/query. All of these are load-bearing; the API's 400s are opaque:

• queryId: "adhoc" is required. Timeframe is epoch milliseconds: timeframe: { from, to }.

• Count invocations by outcome:

  {
    "queryId": "adhoc",
    "timeframe": { "from": 0, "to": 0 },
    "parameters": {
      "datasets": ["cloudflare-workers"],
      "filters": [
        { "key": "$metadata.type", "operation": "eq", "value": "cf-worker-event", "type": "string" }
      ],
      "calculations": [{ "operator": "count", "alias": "count" }],
      "groupBys": [
        { "value": "$workers.scriptName", "type": "string" },
        { "value": "$workers.outcome", "type": "string" }
      ],
      "limit": 100
    },
    "view": "calculations",
    "limit": 100
  }

• The cf-worker-event filter is what makes counts mean invocations; without it you count every log line.

• Raw samples: view: "events", empty calculations/groupBys, small limit. Events carry $metadata (service, trigger, level, message, fingerprint, requestId) and $workers (outcome, scriptName, scriptVersion, wallTimeMs, cpuTimeMs, eventType).

• Error-level app logs grouped by message: filter $metadata.level eq error, groupBy $metadata.message.

• Verified filter operations: eq, neq. Percentiles: the API wants "median", not "p50". Filter for substrings client-side on fetched events.

• Useful drill-downs: groupBy $metadata.trigger (route), $workers.scriptVersion (did a deploy change the rate mid-window), $workers.event.request.headers.user-agent.

Counting gotchas

• Grouped results are unsorted and effectively capped (~10 rows returned regardless of limit). A missing group ≠ zero. To see error outcomes, add $workers.outcome neq ok instead of hoping the error rows make the cut; sort client-side.
• One isolate death fans out. An OOM kills every request pinned to the isolate at the same instant — cluster raw OOM events by timestamp before reading the count as user impact.
• Message prefixes fragment groups. Logs with leading timestamps (better-auth's format) split one error into N single-count groups; grouped-by-message counts badly understate them. Sample events and merge client-side.
• Prod deploys are manual — main being fixed doesn't mean prod runs the fix. Check $workers.scriptVersion and the scripts' modified_on before concluding a fix didn't work.

Known noise — filter these out, do not re-investigate

Entries land here only after an investigation proved there is no first-party emit site to fix or demote. Each keeps the one condition that would make it real signal again.