Observability Service Reliability

Service Reliability

Design reliability targets that people will actually act on, then operate them. This skill covers the judgment before the API call — which service-level indicator fits the data you have, what target is achievable rather than aspirational, whether an SLO is even the right instrument — and then the mechanics of creating, alerting on, resetting, and retiring SLOs through the Kibana API.

Reliability instruments are not interchangeable. An SLO measures a user-visible outcome against a spendable budget; a threshold rule fires on a raw condition; an anomaly job finds deviations where no fixed threshold exists; a synthetics monitor is the only one of the four that can see a service that has stopped emitting telemetry entirely. Choosing wrong produces alerts that are technically correct and operationally useless. For diagnosing a service that is already degraded, and for the incident workflow itself, use the observability-sre-triage skill; for general rule lifecycle mechanics use the kibana-alerting-rules skill.

<!-- begin-partial: preamble -->

Environment Configuration

This skill executes Elasticsearch operations through the elastic CLI. If the elastic CLI is not installed, tell the user what it is needed for. Do not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g., GET /, GET /_cat/indices, GET /{index}/_mapping, GET /{index}/_settings/index.mode, POST /_query). The Operations table at the end of this document maps each shorthand to the equivalent elastic CLI command — always use the CLI rather than calling the HTTP API directly.

<!-- end-partial: preamble -->

Analysis without cluster access

The CLI check above gates querying the cluster — it does not gate analysis. When the user has already supplied the evidence in their question (metric values, counts, status reasons, log lines, alert payloads, configuration), reason from that evidence and deliver the conclusion.

When you genuinely do need data the user has not provided, still say what you would check and how — name the specific query, index, and field that would settle the question — and then ask for CLI setup. An answer that names the check is useful without a cluster; one that only asks for setup is not.

SLO and alerting operations run against Kibana and use the kbn: prefix (for example, POST kbn:/api/observability/slos); data validation runs against Elasticsearch with a bare path (for example, POST /_query). Kibana SLO commands are space-scoped — pass the space explicitly. For non-default spaces the HTTP path becomes kbn:/s/<space_id>/api/observability/slos.

Full request-body schemas for every SLI type live in references/slo-api-schemas.md, the burn-rate rule schema in references/burn-rate-rules.md, and curated official documentation in references/documentation.md.

Jobs to be done

• Translate a reliability concern into the right instrument: SLO, burn-rate rule, threshold rule, anomaly job, or synthetics monitor
• Choose an SLI type for a given service and data shape, and validate the underlying fields before committing
• Set a target, time window, and budgeting method that are defensible against measured history
• Decide whether to group an SLO, and refuse high-cardinality grouping
• Create, verify, update, reset, disable, and delete SLOs
• Attach burn-rate alert rules with fast-burn and slow-burn windows routed to different severities
• Keep the alerting surface signal-dense: remove duplicate coverage, snooze instead of disable, delete unactionable rules
• Monitor availability of user-facing endpoints with synthetics and feed that into an availability SLI

Output discipline

Applies to every response produced under this skill.

• Commit to the best-supported conclusion. Recommend one SLI design and defend it. Do not enumerate every SLI type with equal weight and leave the choice to the user — that is not a recommendation, it is a menu.
• State the target and window you chose, and why, once. Do not restate the justification per bullet.
• Do not invent field names. Verify every field against the mapping before writing it into an indicator. If a field cannot be confirmed, say so and stop; a plausible-looking field name is worse than an admitted gap.
• Do not speculate past the evidence. If the measured history does not support a target, say what it does support.
• Report absence as absence. Zero events means the data is missing or the service is not emitting; it never means the service is healthy. An SLI ratio computed over zero total events is undefined, not 100%.
• Do not pad. No restating the question, no narrating which queries were run unless the result mattered.
• End on the finding. No trailing offers such as "want me to set this up?". Actionable follow-ups belong in a recommendations list, phrased as recommendations, not as questions.

Process: route the reliability concern to an instrument

Do this before designing anything. Most bad SLOs are threshold rules wearing a costume.

1. Name the user-visible symptom. Ask what a user or downstream consumer would notice and complain about. If the answer is a resource number rather than an experience — disk at 90%, heap climbing, replica lag — the concern is a capacity ceiling, not a reliability target. Route it to a threshold rule.

2. Check whether coverage already exists. List SLOs with GET kbn:/api/observability/slos, alerting rules with GET kbn:/api/alerting/rules/_find, and anomaly jobs with GET /_ml/anomaly_detectors. Adding a second instrument on a signal that is already covered is the most common source of duplicate pages.

3. Pick the instrument.

   Concern	Instrument	Why, and what it costs
   A user-visible success or latency outcome you want to budget over weeks	SLO + burn-rate rule	Gives a spendable budget and history, and paces change velocity. Costs a transform, needs steady traffic, and is deliberately slow — even fast-burn uses a one-hour window.
   A hard bound with a known safe value and an immediate operator action	Threshold or custom rule	Fires within one schedule interval, no transform. But it has no budget and no memory, so it re-fires for as long as the condition holds.
   A signal with no fixed threshold, strong seasonality, or many entities	Anomaly detection job	Finds unknown-unknowns and adapts to seasonality. Needs weeks of history to be trustworthy and emits scores, not outcomes, so it is a poor pager.
   Reachability of a user-facing endpoint from outside your own telemetry	Synthetics monitor	The only instrument that detects a total outage. Costs a check budget (see the synthetics section) and cannot explain an internal partial failure.
   An internal job with no consumer contract, or a service still changing daily	None	An SLO with no owner and no stable baseline becomes permanently red and is then ignored, which is worse than no SLO.

4. Do not skip the outage case. An SLI built from a service's own logs or traces cannot see the service disappear: with zero events there is no total, so the ratio is undefined and the SLO neither burns nor recovers. Pair every request-based SLO on a user-facing service with either a synthetics availability monitor or a no-data threshold rule. This is the one place where two instruments on one signal is correct rather than duplicative.

Process: design the SLO

1. Locate the data and confirm the fields exist. Resolve the index pattern with GET /_resolve/index/<pattern>, then confirm every field the indicator will reference with GET /<index>/_mapping or GET /_field_caps. OTel-native data lives in traces-*.otel-*, metrics-*.otel-*, and logs-*.otel-*, with service.name populated on all three; APM aggregate metrics live in metrics-service_summary.1m.otel-*, metrics-service_transaction.1m.otel-*, metrics-transaction.1m.otel-*, and metrics-service_destination.1m.otel-*. Do not assume a status-code, duration, or outcome field exists because it is conventional — confirm it. A wrong field name produces an SLO that computes cleanly and means nothing; How a missing field fails shows exactly how.

   Two field placements are worth knowing because they are commonly guessed wrong: http.response.status_code is on traces-*.otel-* and not on logs-*.otel-*, and transaction.duration.us is on traces-*.otel-* and not on the metrics-*.1m.otel-* rollups, which carry transaction.duration.histogram and transaction.duration.summary instead. Confirm both against the deployment in front of you rather than trusting this list.

2. Validate the SLI with ES|QL before creating anything. Run the good/total ratio over recent history with POST /_query so you know the measured baseline. This query is exploratory and is written in ES|QL, not KQL:

   FROM traces-*.otel-*
   | WHERE service.name == "cart" AND kind == "Server" AND @timestamp >= NOW() - 30 day
   | STATS total = COUNT(*), good = COUNT(*) WHERE http.response.status_code < 500
   | EVAL achieved = good::DOUBLE / total

   Against an OTel demo cluster this returns total: 1393887, good: 1393887, achieved: 1.0. That is the measured input to step 4, not a target to copy — a clean 30 days argues for a target below 100%, not at it.

   HTTP status lives on traces, not on logs. http.response.status_code is populated on traces-*.otel-*. OTel log records do not carry an HTTP status, so the field does not exist in logs-*.otel-* at all, and its coverage on traces is per-service: only spans emitted by an HTTP server carry it. Filtering kind == "Server" keeps the service's own inbound requests and drops the client spans that report the status of calls it made to others. Confirm coverage for the specific service before building an indicator on it:

   FROM traces-*.otel-*
   | WHERE service.name == "cart" AND kind == "Server" AND @timestamp >= NOW() - 24 hour
   | STATS spans = COUNT(*), with_status = COUNT(http.response.status_code)

   On the same cluster cart returns spans: 46420, with_status: 46420, while checkout — a gRPC service — returns spans: 4887, with_status: 0. When with_status is 0 there is no HTTP status to budget. Use sli.apm.transactionErrorRate, or build the ratio on event.outcome, which is populated on every OTel span regardless of protocol:

   FROM traces-*.otel-*
   | WHERE service.name == "checkout" AND kind == "Server" AND @timestamp >= NOW() - 30 day
   | STATS total = COUNT(*), good = COUNT(*) WHERE event.outcome == "success"
   | EVAL achieved = good::DOUBLE / total

   That returns total: 147009, good: 146996, achieved: 0.9999. Write good in the positive form (== "success"), not as a negation of "failure" — a null outcome must not count as good.

   This split is not specific to Serverless or to one demo application. The same measurement across every service on a Stack 9.4.4 cluster, over traces-*.otel-* with kind == "Server":

   Service	Server spans	http.response.status_code coverage	event.outcome coverage
   catalog	49,271	0%	100%
   gateway	37,773	100%	100%
   orders	37,754	100%	100%
   payments	34,710	100%	100%
   recommendations	34,692	0%	100%
   shipping	34,531	100%	100%

   Two of six services have no HTTP status at all while every one of the six has event.outcome on every span. An availability SLO built on http.response.status_code < 500 for catalog would compute good: 0 against total: 49271 — an achieved SLI of 0%, a fully consumed error budget, and a burn-rate rule that pages continuously against a healthy service. Nothing in the API response or the SLO UI flags this; the numerator is simply always zero. Run the coverage check above for the specific service every time, and prefer event.outcome when you are writing one indicator to cover several services.

   Never substitute a plausible-looking status field for a missing one — see How a missing field fails for what that costs.

   Also check that traffic is thick enough for a ratio to be meaningful. If the thinnest buckets carry only a handful of events, a single failure swings the SLI by whole percentage points and the SLO will be noise:

   FROM traces-*.otel-*
   | WHERE service.name == "cart" AND kind == "Server" AND @timestamp >= NOW() - 7 day
   | STATS events = COUNT(*) BY bucket = BUCKET(@timestamp, 1 hour)
   | SORT events ASC
   | LIMIT 10

   Bound this one explicitly. Without a @timestamp predicate it buckets the entire retention of the trace data streams, which is cheap on a demo cluster and expensive on a customer's.

3. Choose the SLI type from the data shape, not from preference.

   SLI type	Use when
   sli.kql.custom	Raw logs or documents where good and total are expressible as filters over events
   sli.metric.custom	Pre-aggregated metric fields where good and total are equations over sums or counts
   sli.metric.timeslice	A metric compared against a threshold per slice, such as a p95 latency ceiling
   sli.histogram.custom	Histogram fields, using a range for good and a value count for total
   sli.apm.transactionDuration	APM transaction latency against a millisecond threshold
   sli.apm.transactionErrorRate	APM transaction success rate
   sli.synthetics.availability	Synthetics monitor uptime for a user-facing endpoint

   Prefer an APM or synthetics type when it fits: they encode the service, environment, and transaction dimensions for you and stay correct when the underlying index layout changes. Reach for sli.kql.custom when the outcome is only visible in raw events, and for sli.metric.custom when the service already emits its own counters.

4. Set a target you can meet. Take the measured baseline from step 2 and set the target at or just below it, then ratchet upward once the service earns it. A target above the measured baseline burns the entire budget on day one, the SLO stays red permanently, and the team stops looking at it. objective.target is a decimal between 0 and 1 — 0.995, not 99.5. Sanity-check the target against the budget it implies over 30 days:

   Target	Error budget over 30 days
   99%	7h 12m
   99.5%	3h 36m
   99.9%	43m 12s
   99.95%	21m 36s
   99.99%	4m 19s

   If a single rolling deploy, a node restart, or one dependency blip costs more than the whole budget, the target is unachievable and should be rejected, not accepted with a caveat.

5. Choose the time window. timeWindow.type is rolling (7d, 30d, 90d) or calendarAligned (1w, 1M). Rolling windows move continuously, so budget recovers gradually and burn-rate alerting stays meaningful — this is the default for anything operational. Calendar-aligned windows reset at the period boundary, which matches contractual or monthly-reporting language but produces a budget cliff on the first of the month. Use rolling for paging, and add a calendar-aligned SLO alongside it only when someone genuinely reports on calendar periods.

6. Choose the budgeting method. occurrences divides good events by total events across the whole window, so a high-traffic hour dominates and a quiet overnight outage barely registers. timeslices chops the window into slices, marks each slice good or bad against objective.timesliceTarget, and divides good slices by total slices, so every period counts equally. Choose timeslices when low-traffic periods matter or when the indicator is a threshold on an aggregate rather than a countable good/total. sli.metric.timeslice requires budgetingMethod: "timeslices" — the pairing is not optional, and objective.timesliceTarget and objective.timesliceWindow become required.

7. Decide grouping deliberately. groupBy creates one independent SLO instance per unique value, each with its own transform buckets and its own alerts. Measure the cardinality before setting it:

   FROM traces-*.otel-*
   | WHERE @timestamp >= NOW() - 24 hour
   | STATS instances = COUNT_DISTINCT(service.name)

   Keep the @timestamp bound. A cardinality aggregation with no time predicate runs over the whole retention of the trace data streams; a recent window answers the same question at a fraction of the cost.

   Cardinality is not the only check — the dimension also has to be populated. groupBy on a field that is mapped but null produces one degenerate instance covering everything, which looks like a working grouped SLO and is not. Confirm with COUNT(<field>) alongside COUNT(*) before setting it.

   Group on stable, bounded dimensions such as service.name, service.environment, or k8s.namespace.name. Refuse high-cardinality fields — trace.id, url.full, user.id, and churning identifiers like k8s.pod.name — and say why rather than creating the SLO and warning afterward. If per-entity visibility is genuinely needed on a wide dimension, filter to the entities that matter instead of grouping across all of them. Synthetics SLOs are auto-grouped by monitor and location; do not set groupBy manually.

8. Create and verify. Build the body per references/slo-api-schemas.md and POST kbn:/api/observability/slos. Then read it back with GET kbn:/api/observability/slos/{id} and confirm it is computing before reporting success — a created SLO whose transform has not started yet returns no summary data.

KQL is an API contract here, not a style choice

ES|QL is the query language for Observability, and every exploratory query you run to validate an SLI — baselines, cardinality checks, traffic distribution — must be ES|QL against POST /_query.

The exception is inside the SLO request body. The good, total, and filter fields of sli.kql.custom (and the filter fields of the other indicator types, plus the --kql-query parameter on SLO search) are KQL strings, because the SLO API defines them that way. There is no ES|QL form of those fields. Write KQL there, and only there:

{
  "type": "sli.kql.custom",
  "params": {
    "index": "traces-*.otel-*",
    "filter": "service.name : \"cart\" and kind : \"Server\"",
    "good": "http.response.status_code < 500",
    "total": "*",
    "timestampField": "@timestamp"
  }
}

Do not translate these fields to ES|QL — the API will reject or silently mis-parse them. Do not translate exploratory queries to KQL either.