Telemetry

Telemetry Operations

Use this skill to operate the telemetry stack — Prometheus, the OpenTelemetry Collector, and Loki — as the one deployment unit it ships as: collection and scraping, ingestion, retention, and the rules that turn raw signals into alerts. This is a tool skill for one stack of named tools. Observability strategy — SLIs, SLOs, error budgets, and what to instrument — belongs to platform-engineering; dashboards, panels, and Grafana-side alert rules, contact points, and notification policies belong to grafana. This skill owns the collection/ingest/retention layer and the Prometheus rules files that both of those skills consume.

Operating contract

1. Read-only discovery before any mutation. Inspect scrape configs, rules files, collector pipelines, and retention settings first. The bundled telemetry-check script runs rule sanity and scrape-target reachability checks without changing anything.
2. Confirm the target, scope, and rollback path before acting. Read-only discovery may proceed without confirmation. Mutations — a config reload, a promtool rules push, a collector restart, a retention-policy change — require an explicit human directive naming the instance.
3. A config that parses is not a config that works. Rule sanity catches structure; it does not prove the expression is meaningful or that the target is scrapable. Verify at the delivery boundary (scrape succeeded, rule evaluated, alert fired) before claiming health.
4. Keep evidence bounded. Summarize config diffs and query results; never dump full prometheus.yml, collector pipelines, or credentials into chat.
5. Own the retention decision. Retention is a capacity and compliance decision made deliberately per component — Prometheus block retention, OTel exporter buffering, Loki retention per tenant — and reviewed on a schedule, not left at defaults.

The telemetry-check script

scripts/telemetry-check is an agent-first, read-only checker. It parses Prometheus rules files with a bundled stdlib YAML reader and runs dependency-free structural sanity checks; it extracts static targets from scrape configs and probes TCP reachability; and it emits bounded JSON. It never writes files and never sends data anywhere.

scripts/telemetry-check --help                      # no server needed
scripts/telemetry-check --rules rules.yml --json    # rule sanity, machine-readable
scripts/telemetry-check --scrape prometheus.yml --json   # probe static targets
scripts/telemetry-check --targets targets.txt --timeout 5

Exit codes: 0 all checks passed, 1 issues found or a fatal error, 2 usage error. telemetry-check --rules checks structure only: exactly one of record/alert per rule, a non-empty expression with balanced delimiters, valid durations, recording-rule and label names, and string-only label values. Use promtool check rules separately for full PromQL parsing.

Operating loop

1. Identify the deployment: which components are in scope (Prometheus, OTel Collector, Loki), how they are deployed (binary, container, operator), where configs live, and who owns them.
2. Collect evidence: run telemetry-check --rules and --scrape on the configs, then check the live status endpoints (/-/healthy, /api/v1/targets, collector health, Loki ready) where access exists.
3. For a query investigation, load references/05-query-workflows.md. Define the signal, selector, UTC time window, step/limit, and expected unit; validate syntax separately from semantics; execute read-only instant then bounded range/log queries; and capture status, scope, time, limits, warnings, and cardinality evidence.
4. Triage against the symptom: map the reported problem to the evidence (missing series → scrape or relabeling; alert not firing → rule or retention; logs missing → ingest or label cardinality). Treat an empty result as unknown, never as numeric zero, and distinguish stale, partial, expired, absent-label, and query-error states.
5. Act with confirmation: bounded, scoped mutations after a human directive, with a rollback path named first.
6. Verify: re-run the relevant check and confirm the observable at the delivery boundary.

Prometheus: scrape, rules, relabeling, retention, HA

• Scrape config (scrape_configs): one job per scrape group with a deliberate scrape_interval, scrape_timeout below it, and metrics_path. Prefer static_configs for known endpoints and service discovery (*_sd_configs) for dynamic ones. Verify the running config with /api/v1/status/config and targets with /api/v1/targets?state=active.
• Recording and alerting rules: rules files are groups of record or alert rules with a PromQL expr, optional for/keep_firing_for durations, and labels/annotations. Validate every change with promtool check rules for full PromQL parsing and with the bundled telemetry-check --rules for dependency-free structural sanity before reload. Rules must be small, well-named, and reviewable — a 100-line expression is a debugging liability, not a rule.
• Relabeling: relabel_configs and metric_relabel_configs rewrite labels before ingestion. Use them to enforce label naming, drop high-cardinality or internal labels, and attach scrape metadata. Relabeling mistakes silently change series identity — verify with a targeted curl of /metrics and the target's scrapeUrl in /api/v1/targets.
• Retention: --storage.tsdb.retention.time and --storage.tsdb.retention.size bound local block retention; blocks are 2h by default. Retention is a capacity decision (see references/04-stack-integration-and-retention.md), not a default to leave alone. Watch prometheus_tsdb_head_series and prometheus_tsdb_compaction for cardinality and compaction pressure.
• High availability (HA): two identically configured Prometheus instances with --query.max-concurrency headroom and consistent external labels let you shard or deduplicate at the query layer (Thanos, Mimir, or Grafana data sources). Alerting rules must not double-fire: HA pairs need a dedup layer or consistent labeling, and rule evaluation must stay consistent across replicas. Rule evaluation state (for counters) is local to each instance.

OpenTelemetry Collector: pipeline, sampling, correlation

• Collector pipeline: a pipeline is a directed acyclic chain of receivers → processors → exporters per signal type (metrics, logs, traces). Keep pipelines narrow and per-signal; a pipeline that mixes signals becomes un-debuggable. Each pipeline must have at least one exporter; unused receivers/exporters are dead configuration.
• Receivers, processors, exporters: receivers accept data (OTLP, Prometheus, filelog, hostmetrics); processors transform, batch, filter, sample, and attach resource attributes; exporters send data onward (OTLP, Prometheus remote write, Loki, logging). Order matters — batching and the memory_limiter processor belong before exporters; tail_sampling belongs on trace pipelines only.
• Sampling: tail_sampling on traces decides at the batch level; probabilistic_sampler is stateless and cheaper. Sample deliberately: full traces for errors and slow paths, tail sampling for high-volume success traffic, and never sample away the error signal. Sampling must be coordinated with retention — a sampled trace is gone forever, so the decision belongs in the pipeline design, not in an emergency.
• Trace/span correlation: carry trace_id and span_id in log lines and metric exemplars so LogQL and PromQL can pivot back to the trace. The collector's spanmetrics processor derives RED metrics from spans, and OTLP logs with trace context land in Loki with trace_id as a structured label for correlation. Trace context propagation is an application-level concern that backend-engineering owns; the collector side is here.