Observability Monitoring

Observability Monitoring

Use this skill to turn vague "is it working?" questions into evidence-backed monitoring, alerting, and incident workflows. Start from user or business impact, then move down through the system layers and choose the signal that can prove the current hypothesis.

Operating rule

Do not treat a green dashboard as proof of health. A monitoring claim is complete only when it names:

1. the observed scope and time window;
2. the user, business, or operator outcome being protected;
3. the signal and exact query/probe that supports the claim;
4. the threshold or SLO that defines bad;
5. the next human action and its runbook/evidence link.

Keep code review, live runtime proof, UI/render proof, and release readiness as separate verdicts.

Investigation is read-only by default. A restart, alert suppression, metric-schema/label change, sampling or retention change, or vendor reconfiguration is a production mutation: involve the responsible owner or incident authority, preserve the relevant evidence first, capture the exact config/command diff, state the rollback condition, and verify the user probe plus SLI after the change.

Workflow

1. Freeze scope and collect live facts

Before changing a monitor, alert, host, or service:

• Read the repository AGENTS.md, relevant rules, runbooks, and deployment docs.
• Establish the actual checkout/branch, deployment, process, host, port, proxy/tunnel, and data source.
• Record the observation time, environment, query/probe, and whether the result is current or historical.
• Inspect the source code/config that emits or consumes the signal before changing it.
• Never infer health from an old screenshot, a stale handoff, a single PID, or a dashboard with no successful user probe.

For infrastructure fixes, document the traffic path (DNS, proxy, tunnel, ingress, service, backend) before touching a surprising value such as 127.0.0.1, a non-default port, or a disabled check.

2. Define the outcome before the metric

Write the protected outcome in concrete terms:

• User: can open the page, submit the request, download the result, or complete the workflow.
• Business: orders, successful jobs, conversions, revenue, or another domain event continue to occur.
• Operator: an on-call engineer receives a page early enough to act and can identify the next step.

Add at least one black-box or synthetic check for the user-facing path. Add real-user telemetry when the experience can vary by browser, geography, device, or network. Technical resource health is not a substitute for a user or business signal.

3. Map the monitored layers

Inspect the layers from bottom to top and state which ones are in scope:

1. Hardware/infrastructure: CPU, memory, disk capacity and latency, network, temperature, power, GPU, host availability.
2. Host/OS: load, processes, file descriptors, swap, service state, kernel/resource pressure.
3. Network: reachability, packet loss, latency, port state, DNS, TLS, ingress/proxy health.
4. Application/APM: request rate, errors, latency, saturation, dependency calls, queue depth.
5. Databases and queues: throughput, slow queries, connection pools, replication lag, backlog, consumer health.
6. Containers/orchestration: desired versus ready replicas, restarts, scheduling, evictions, resource limits, ephemeral identity.
7. Business: successful workflows, jobs, orders, revenue, conversion, or domain-specific zero-activity checks.

Do not stop at the infrastructure layer when the business outcome is failing. A system with green CPU and memory can still have a broken payment, form, queue consumer, or model job.

4. Choose the right signal

Use the signal that answers the question instead of collecting everything indiscriminately:

Metrics answer aggregate how much/how often. Logs answer what happened. Traces answer where in the path. Profiles answer which code or process consumed the resource. Correlate them with stable resource identity, timestamps, and trace/request context.

5. Design metrics without cardinality accidents

Treat every unique metric-name plus label set as a time series. Before adding a label, estimate its possible values and lifetime.

• Good metric dimensions are bounded and useful for aggregation: method, status class/code, service, region, queue, job type, or environment.
• Keep high-cardinality or per-event values in logs/traces: user ID, request ID, email, raw URL with parameters, exception text, or arbitrary object IDs.
• Prefer route templates over raw URLs.
• Set explicit cardinality limits or views where the telemetry SDK supports them, and monitor overflow/dropped-attribute indicators.
• Prefer histograms for fleet-wide aggregatable latency SLOs. Client-side summaries/quantiles generally cannot be aggregated across instances without changing their meaning; use them only when that limitation is understood. Never hide tail latency behind an average.

If a query needs a unique ID to be useful, that ID belongs in a trace or structured log field, not in a metric label.

6. Define SLI, SLO, SLA, and error budget

Use the terms precisely:

• SLI: a measured indicator, such as successful requests divided by valid requests or p95 latency for a workflow.
• SLO: the internal target and time window for that indicator.
• SLA: an external agreement with consequences if the target is missed.
• Error budget: the allowed unreliability implied by the SLO during the window.

Choose the SLI from the user-facing outcome, not from whichever metric is easiest to collect. Define the valid-request population, exclusions, time window, aggregation, and owner. Treat an SLO as a control loop: remaining budget informs release pace, risk, testing, and reliability work.

7. Build actionable alerts

Alert on symptoms that indicate user or service pain. Use dashboards and drill-downs to investigate causes.

Every paging alert must include:

• symptom and affected scope;
• exact query, threshold, duration, and evaluation window;
• severity and owner;
• a link to the dashboard/query and runbook;
• concrete first actions, rollback criteria, and escalation path;
• a test or smoke procedure proving the alert route works.

If no immediate action exists, make it a dashboard annotation, ticket, or recording rule instead of a page. Use a pending duration (for) to suppress short blips. If using hysteresis, define separate fire and clear thresholds; if using multi-window evaluation, define each window and its condition. Do not confuse either control with Slack/chat routing. Treat "more than two incidents per shift" as a review heuristic, not a universal law: frequent pages mean the system or alert policy needs repair.

Prefer multi-window or burn-rate alerts for SLOs when the backend supports them. A burn rate near 1 spends the budget at the planned rate; a materially higher rate requires faster response. Verify the math against the actual SLO window and alert implementation.

Portable examples

Adapt metric and label names to the live schema; these are patterns, not copy-paste production queries:

# RED error ratio for one service over five minutes.
sum(rate(http_requests_total{service="checkout",code=~"5.."}[5m]))
/
sum(rate(http_requests_total{service="checkout"}[5m]))

# Aggregatable fleet-wide p95 from a histogram.
histogram_quantile(0.95,
  sum by (le, service) (
    rate(http_request_duration_seconds_bucket{service="checkout"}[5m])
  )
)

Use a synthetic/domain probe that asserts the real outcome, not only HTTP reachability. Prefer a read-only endpoint or a sandbox/test tenant. If a write path is unavoidable, require a dry-run or idempotency key, a bounded fixture, explicit cleanup, and an owner-approved change window: submit safe test request -> assert expected status and domain result/job ID -> record latency and trace ID -> clean up. Never send an arbitrary representative request into production.

A burn rate is observed error ratio / allowed error ratio, where allowed error ratio = 1 - SLO. Worked example: for an SLO of 99.9% (allowed = 0.001), a 5-minute observed error ratio of 1.0% (0.010) gives burn rate 10; if the local paging policy is the illustrative >=10 for 5m AND >=5 for 1h, page, otherwise route it according to the lower-severity policy. The query pattern is:

(
  sum(rate(http_request_errors_total{service="checkout"}[5m]))
  /
  sum(rate(http_requests_total{service="checkout"}[5m]))
) / 0.001

The thresholds and windows are examples only; derive and test them from the real SLO, traffic population, exclusions, and paging budget.