OpenTelemetry Pipeline Metrics Spec

Metrics Instrumentation

Normative spec for the backend half of operational observability: per-stage OpenTelemetry metrics in apps/opik-backend. The metrics are designed so a flow-ordered Grafana dashboard can read top-to-bottom — the failing stage is the one whose numbers break — with a per-workspace drill. Building that dashboard (layout, query contracts, per-customer name resolution, dependency panels, validation) is a separate concern, specified by the dashboard-authoring skill in comet monitoring tooling; this skill covers only what to emit.

Patterns applied in this implementation (online scoring is the worked example):

• Producer metrics — every stage that emits work is counted at the source: sampler decisions (sampler_decisions_total{decision}) and enqueue-to-Redis (enqueue_total{result}). A producer that emits nothing makes downstream starvation explainable rather than mysterious.
• Consumer metrics — throughput and per-stage timing on the side that drains the queue (processing_time, plus per-Redis-op read/claim/ack_and_remove/list_pending_time).
• Entrypoint RED — the workflow's front door (the HTTP ingest route) is measured Rate / Errors / Duration from http_server_request_duration_seconds, with 5xx broken down by endpoint × error_type × workspace, so an ingestion problem is never mistaken for a scoring problem.
• Errors — a dedicated error counter per stage, dimensioned by error_type (the exception class) and, for shared async plumbing, by component (listener/subscriber): processing_errors_total, unexpected_errors_total, and enqueue_total{result="error"} (a push failure = real loss).
• Success — success is derived, never double-counted: throughput − errors, surfaced as one "success rate" headline tile.
• Queue time & end-to-end latency — queue_delay (enqueue→pickup) is kept separate from processing_time (scorer/LLM work) so a backlog is distinguishable from a slow scorer; end-to-end = queue_delay + processing_time.
• Backpressure — poll-tick skips are counted but are benign (consumer busy), never lost work.
• Saturation & resource levels — gauges for in-flight work (max per pod) and JVM heap used-vs-limit per pod expose the pipeline approaching a ceiling before it starts failing (the USE method — utilization / saturation / errors — complementing RED).
• Volume & payload size — byte/char counters (bandwidth, total bytes) and payload-size distributions, broken down by content type and workspace, for cost and impact attribution.
• Per-workspace dimensioning — the customer drill (§1.3) is a first-class label, not an afterthought.
• Infrastructure dependencies — the datastores the flow leans on (Redis streams, ClickHouse, locks, MySQL) are surfaced on the dashboard from their exporters and system.query_log, so a "slow pipeline" resolves to the dependency causing it.

To see these conventions already in practice, grep apps/opik-backend for the existing metric families rather than specific classes (metric names are the stable contract; class locations move): the online-scoring *_sampler_decisions_total, *_enqueue_total{result}, *_processing_time_milliseconds, *_queue_delay_milliseconds, *_processing_errors_total{error_type} / *_unexpected_errors_total, the per-Redis-op *_{read,claim,ack_and_remove,list_pending}_time_milliseconds, and the attachment upload byte/size families.

Apply alongside the opik-backend skill (general conventions, logging rule).

1. Model

1.1 Decompose the workflow into ordered stages, each defined by: input, output, failure mode, existing instrumentation.

1.2 Choose the instrument by the question it answers. Most stages need a counter (did it happen, and how often?) and a histogram (how long did it take?); add a gauge only for a level you cannot derive from those.

• Counter — a monotonically increasing total, read as a rate. Use for events: throughput, decisions, results, errors, and cumulative volume (messages, bytes). Anything you phrase "per second" or "how many since start". Surfaces with a _total suffix. MUST NOT be used for a value that can decrease.
• Native histogram — a latency/size distribution, read with histogram_quantile. Use whenever a p95/p99 matters, not just an average: processing time, queue delay, per-dependency op time, end-to-end latency, and payload size (bytes/chars). Prefer native (no explicit buckets, §2.1) over classic le-bucketed histograms; reach for classic buckets only when a downstream consumer (e.g. an existing exporter) forces them. Do NOT use a histogram where a counter suffices — a plain success/error tally needs no distribution.
• Gauge — an instantaneous level, read as-is. Use for a quantity that rises and falls and can't be reconstructed from a counter: current queue depth / backlog, in-flight / in-progress count (saturation), batch/read/claim size, lock waiters, heap used. If you actually want a trend, count events with a counter rather than sampling a level with a gauge.
• UpDownCounter (OTel) — when the level is naturally maintained as signed +1/−1 deltas (in-flight = increment on start, decrement on finish) rather than sampled. Exports like a gauge but is cheaper and less racy than reading a size on every observation.

Rule of thumb: "how many happened" → counter; "how long / how big" → histogram; "how much is there right now" → gauge (or UpDownCounter). Cover each stage with RED (Rate, Errors, Duration) for the work flowing through it and USE (Utilization, Saturation, Errors) for the resource it runs on.

1.3 Define identity dimensions, each bounded in cardinality:

• workspace_id and workspace_name — the customer drill. Always paired; workspace_name falls back to workspace_id when the name is absent (§2.3).
• a stage/type label — what kind of work this is (evaluator_type, decision, the Redis op, content/mime type). Lets the dashboard sum by(...) per stage (§2.2).
• result ∈ {success, error} on outcome counters.
• error_type on every error counter — the exception class / failure category — plus a component label (listener/subscriber/endpoint) where one counter serves many call sites, so the outcome panel breaks errors down by both cause and origin.

Cardinality MUST stay bounded by #workspaces × #types × #error_types; an unbounded value (trace id, user input, raw message, full URL) MUST NOT be placed on a label.