Debugging MCP Analytics

Debugging MCP analytics

Product analytics for MCP servers. A team ships an MCP server; the @posthog/mcp SDK wraps it in one line; every tool call, agent intent, and failure lands in PostHog as a $mcp_* event you can query, chart, alert on, and cluster — plus a dedicated dashboard. The MCP-layer sibling of @posthog/ai.

The differentiator is intent: not "ran query_run 14 times" but "was trying to find a churn cohort". Explicit non-goal: this does not replace LLM analytics / AI observability — generation traces, prompt/response, and token cost belong there.

Status: beta, TypeScript and Python SDKs shipped, available to every project. PostHog dogfoods it — its own MCP server instruments itself, and that data drives the dashboard. Public tracking: mega-issue PostHog/posthog#64016, which is the live source for roadmap and customer wishlist.

Repos

GitHub is the source of truth for where the code lives. Paths below are in-repo; for the repos outside this monorepo, resolve a local checkout via references/local-repos.md rather than assuming a location.

Don't conflate:

• PostHog/mcp-analytics is the archived prototype of this SDK — stuck at 0.0.9 with an old track(server, {...}) API. It published under the same @posthog/mcp name, so grepping that name can land you there. npm @posthog/mcp now resolves to PostHog/posthog-js.
• products/mcp_store/ is the MCP server marketplace / team gateway, not this product. (Older notes also mention a products/mcp/ build-tooling directory; it no longer exists — the server and its generation tooling live in services/mcp/.)
• wizard mcp add installs the PostHog MCP server into a coding agent. That is NOT wizard mcp-analytics, which instruments the user's own server.

Line numbers drift and this area moves fast — grep for the symbol, never trust a remembered line number. Confirm a checkout is on a sane branch before quoting its code.

Hard rules (break these and the numbers are silently wrong)

These are the failure modes that produce a plausible-looking answer rather than an error.

1. Always resolve the effective tool name through EFFECTIVE_TOOL_SQL. The expression lives once, in products/mcp_analytics/backend/hogql_queries/base.py: coalesce(nullIf(toString(properties.$mcp_exec_tool_call_name), ''), toString(properties.$mcp_tool_name)). It exists because a single-exec server can report the tool two different ways, and the two eras of data coexist. Today services/mcp resolves the inner tool itself and passes it straight in as the tool name (execToolName() in src/hono/tool-executor.ts, which falls back to the literal exec when the inner command isn't recognized), so $mcp_tool_name usually already holds the real tool. $mcp_exec_tool_call_name is registered in posthog/taxonomy/taxonomy.py and coalesced defensively here, but nothing on master emits it — treat it as historical rows plus in-flight work, not current producer behaviour. Either way, aggregate through the coalesce: hand-rolling properties.$mcp_tool_name alone silently buckets unrecognized exec calls under exec, and misses any data that does carry the dedicated property.
2. Failures come from $mcp_is_error / $mcp_error_type / $mcp_error_status, never $exception. $exception can be disabled, isn't emitted when no error value is passed, and never matched new-SDK events — so querying it returns nothing rather than failing.
3. Dash the in-progress bucket. Every time-bucketed chart zero-fills and marks the final incomplete interval via products/mcp_analytics/frontend/timeBuckets.ts (resolveWindow, normalizeBucket, buildBucketKeys, lastBucketIsInProgress). Omit it and a partial period reads as a real decline.
4. harness is derived, and its logic exists in three places that must move in lockstep: products/mcp_analytics/backend/mcp_harness.py (source of truth — see its module docstring), products/mcp_analytics/frontend/dashboard/harnessRegistry.ts, and models-mcp.md.
5. Check which SDK version the dogfood server is on before trusting dogfood data. services/mcp consumes the SDK through an alias in its package.json and has historically lagged the published version, so version-dependent properties (typed error types, $lib identity, payload redaction) can be absent from PostHog's own data even when documented as current. A query filtering on $lib = 'posthog-node-mcp' silently excludes all dogfood traffic if that pin predates SDK 0.7.0. Note too that services/mcp uses the custom-dispatcher (PostHogMCP) path rather than instrument(), so behaviour living only in the instrument() path — stable sessions, $identify deduplication, _meta-based client identity — has never applied to it at any version.
6. Know which session model produced the data. Under the stateless spec there is no transport session, so $session_id is only stable if the server opted into conversation anchoring — enableConversationId, which is off by default. With it off, a stateless client's sessions fragment (often one per request); with it on, $session_id is derived from an agent-echoed handle and survives reconnects, restarts, and pods. Check the flag before diagnosing "fragmented sessions" as an ingestion problem. See references/stateless-and-sessions.md.
7. There are no SQL template files. Every dashboard and tool-quality query is a typed query runner behind the generic /query/ endpoint. A backend/templates/*.sql referenced by older notes no longer exists.

Event vocabulary

All data lives on the shared ClickHouse events table — there is no dedicated table. Every metric is an aggregation over $mcp_tool_call, usually grouped by $session_id.

Source of truth for the SDK-emitted names is packages/mcp/src/extensions/constants.ts in PostHog/posthog-js, exported as PostHogMCPAnalyticsEvent / PostHogMCPAnalyticsProperty (import them for typesafe queries). PostHog-side descriptions — including the server-stamped and exec-mode properties the SDK does not define — live in posthog/taxonomy/taxonomy.py.

Events (all $-prefixed; non-$ names would be treated as customer events): $mcp_tool_call (primary), $mcp_tools_list, $mcp_initialize, $mcp_missing_capability, $mcp_resource_read / $mcp_resources_list, $mcp_prompt_get / $mcp_prompts_list, $identify, $exception.

$mcp_initialize is not a reliable session anchor — but check whose server you're looking at. The 2026-07-28 revision removes the initialize handshake, so a customer server on the SDK's instrument() path emits nothing for a stateless client. PostHog's own server is the exception: services/mcp fires the same $mcp_initialize event from server/discover as from initialize (dispatcher.ts::recordDiscoveryRequest covers both entry points), so the event is present in dogfood data either way. Treat its absence as meaningful only for customer servers. The real anchor is now the conversation handle when the server enables it — references/stateless-and-sessions.md covers the resolution order and the delivery protocol. Live consequence, for customer servers only: frontend/mcpAnalyticsOnboardingLogic.ts derives has_initialize from this event, so a stateless customer server reads as not-instrumented until its first tool call. Onboarding still completes — hasToolCall is checked first, in both that selector and statusFromProbeDefinitions. Projects on services/mcp are unaffected, since it emits the event from server/discover.

Full property tables — split by provenance (SDK-emitted vs stamped by PostHog's own server vs exec-mode only), the identifier distinctions, per-version SDK behaviour, and TypeScript/Python parity — are in references/event-vocabulary.md. Read that before writing queries or changing what gets captured.

Reading the data

Governed metric first

When debugging an MCP failure-rate headline, call posthog:metric-list before the dedicated analysis skills, typed tools, or hand-written HogQL and look for mcp_tool_call_fail_pct. Run an approved, non-drifted match with posthog:data-catalog-metric-run as the canonical headline. Use the paths below only for requested tool, harness, or time breakdowns after that run, and label those breakdowns noncanonical. If no governed metric matches, state that the catalog has no match and label the derived rate noncanonical.

Prefer the dedicated analysis skills over hand-written HogQL; they already encode the exec-mode and harness handling that Hard rules 1 and 4 describe:

• exploring-mcp-tool-usage — front door / router: takes a broad "how is my MCP doing" question and dispatches to the right typed tool or focused skill. Start here.
• exploring-mcp-tool-quality — error rates, latency, reach, failing and slow tools.
• exploring-mcp-sessions — session list, per-session tool calls, intent.
• exploring-mcp-intent-clusters — "what are people trying to do" clusters.
• improving-mcp-tools — eval-scored campaign loop: measure, make one bounded fix, re-measure.

Typed tools exist for most questions and are preferable to raw SQL: posthog:query-mcp-tool-stats, -daily-stats, -failures, -failure-occurrences, -descriptions, -neighbors, -sample-intents, -top-users, and posthog:query-mcp-harness-breakdown, plus session tools (posthog:mcp-analytics-sessions-list / -tool-calls / -generate-intent) and the intent-cluster tools. They are declared in products/mcp_analytics/mcp/tools.yaml.

Harness is the friendly label for the calling client (Claude Code, Cursor, ChatGPT, Windsurf, and ~30 other buckets). It is resolved at query time only, with no stored column: mcp_harness.py::HARNESS_TOKEN_SQL picks the strongest available signal in priority order, over exactly three properties — the ones the SDK schemas can emit ($mcp_vendor_client, with the legacy non-$ mcp_vendor_client coalesced for historical rows -> Claude Code user-agent surface -> Grok and Kimchi user-agents -> $mcp_client_name -> generic user-agent token, both from $mcp_client_user_agent), then harness_label_sql() buckets it (or harness_label_or_token_sql(), which names an unrecognized client verbatim instead of collapsing it into "Other" — use it for ranked top-N lists, never where labels feed an array or unbounded GROUP BY).

$mcp_client_name is one mid-priority input, not a synonym for harness — grouping by it directly gives a different, messier answer: on old SDK versions it rode only on the session's initialize, and Anthropic's pooled surfaces self-report a generic Anthropic/ClaudeAI that only the vendor header can disambiguate. The dogfood-only mcp_session_client_name and $mcp_oauth_client_name are no longer read by harness resolution — the server folds the session-pinned name into per-event $mcp_client_name, and neither property ever resolved an event alone.

For hand-written SQL, models-mcp.md carries the property reference and worked query examples.