Gram Telemetry Query Dimensions

What this covers

telemetry.query (POST /rpc/telemetry.query) is a generic, org-scoped analytics endpoint. It groups pre-aggregated usage metrics by an allowlisted dimension and filters on those same dimensions. This skill explains how to add a new dimension (a new attribute value to group/filter by).

Dimensions vs. measures. A dimension is a breakdown/filter axis (department_name, model, role). A measure is a number being aggregated (total_cost, total_tokens). This skill is about dimensions only. Adding a new measure is a different (parallel) path through the same files — queryMeasures, attributeMeasureSelects, AttributeMetricsMeasures, QueryMeasures.

Everything is backed by one ClickHouse AggregatingMergeTree, attribute_metrics_summaries, fed by attribute_metrics_summaries_mv. Each dimension is one column on that table. The query layer never sees raw JSON paths or SQL from clients — dimensions are a closed allowlist validated at three layers that must stay in sync.

Activate the clickhouse skill (schema/MV work) and golang skill (repo/service edits) alongside this one.

The four layers (keep them in sync)

A dimension key like department_name must appear in all four places. Adding a new one means touching each:

dimension_values (the per-group distinct-value lists on each QueryRow) is automatic — it iterates attributeDimensionRegistry, so a new dimension shows up there with no extra code.

Pick the dimension kind

The repo classifies every dimension as one of three attributeDimensionKinds. This drives how it is grouped and filtered:

• attributeDimScalar — one string value per row (e.g. department_name, model). Group: plain column. Filter: IN.
• attributeDimArray — Array(String), multiple values per user/row (e.g. roles, groups). Group: arrayJoin() (attributes spend to each element). Filter: hasAny(). Stored intact in the sorting key (one array per row) so it does not multiply row count.
• attributeDimProject — the gram_project_id UUID key column; grouped via toString().

Most new identity/request attributes are scalar. Use array only when the attribute is genuinely list-valued per user.

Cardinality. The MV stays cheap because every scalar dimension is functionally determined by the user (one department, one job title per user), so adding one does not multiply rows. Do not add a high-cardinality, per-event scalar (e.g. a raw request id) as a dimension — it would explode the aggregate.

Step-by-step

1. ClickHouse schema + migration

Edit server/clickhouse/schema.sql. For a scalar dimension foo derived from a log attribute attributes.user.attributes.foo:

1. Add the column to the attribute_metrics_summaries table:

   foo String,

2. Add the derived SELECT to attribute_metrics_summaries_mv and the column to its GROUP BY:

   -- in the SELECT list
   toString(attributes.user.attributes.foo) AS foo,
   -- ...and in GROUP BY
   GROUP BY gram_project_id, time_bucket, department_name, ..., foo;

   (Array dimensions use CAST(attributes.user.foo AS Array(String)) AS foo. If a materialized column already exists on telemetry_logs — e.g. user_email, hook_source — reference it directly instead of re-deriving from JSON.)
3. Add the column to the table's ORDER BY sorting key (scalar columns slot in with the other identity dimensions; array columns go at the end alongside roles, groups). The sorting key keeps cardinality bounded.

Then generate the migration (this is a ClickHouse migration, governed by the same expand/contract rules as Postgres — see the migration rules in CLAUDE.md):

mise clickhouse:diff add-foo-attribute-dimension

Atlas diffs schema.sql and emits the migration into server/clickhouse/migrations/ (and the golang-migrate copy), then auto-runs clickhouse:gen-materialized-cols. Never hand-edit the generated migration or atlas.sum.

MV recreation + backfill. Changing the MV's SELECT/GROUP BY and the table sorting key makes Atlas drop and recreate the MV. The MV only transforms rows ingested after it exists — historic rows are not re-aggregated, so the new column reads empty ('') for old buckets. That is acceptable (data ages out at the 30-day TTL); call it out in the PR. Run migrations against local DBs only.