Observability K8s Investigation

Kubernetes Investigation

Diagnose Kubernetes issues using OTel telemetry collected via EDOT (Elastic Distribution of OpenTelemetry) and the kube-stack collector. Correlate cluster state, pod runtime metrics, K8s events, application logs, and APM to identify root cause across the workload, node, and control-plane layers.

<!-- begin-partial: preamble -->

Environment Configuration

This skill executes Elasticsearch operations through the elastic CLI. If the elastic CLI is not installed, tell the user what it is needed for. Do not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g., GET /, GET /_cat/indices, GET /{index}/_mapping, GET /{index}/_settings/index.mode, POST /_query). The Operations table at the end of this document maps each shorthand to the equivalent elastic CLI command — always use the CLI rather than calling the HTTP API directly.

<!-- end-partial: preamble -->

Analysis without cluster access

The CLI check above gates querying the cluster — it does not gate analysis. When the user has already supplied the evidence in their question (metric values, counts, status reasons, log lines, alert payloads, configuration), reason from that evidence and deliver the conclusion.

When you genuinely do need data the user has not provided, still say what you would check and how — name the specific query, index, and field that would settle the question — and then ask for CLI setup. An answer that names the check is useful without a cluster; one that only asks for setup is not.

Every ES|QL query in this skill and in references/query-recipes.md runs via POST /_query. Alert state is read with GET kbn:/api/alerting/rules/_find. Field-presence checks use GET /<index>/_mapping or GET /_field_caps. The Operations table maps each to its elastic CLI equivalent.

Scope

In scope: OTel-receiver-namespaced indices (metrics-kubeletstatsreceiver.otel-*, metrics-k8sclusterreceiver.otel-*, logs-k8seventsreceiver.otel-*, logs-k8sobjectsreceiver.otel-*) and OTel semantic conventions (k8s.pod.name, k8s.namespace.name, k8s.container.restarts).

Out of scope:

• The legacy Elastic Agent Kubernetes integration (metrics-kubernetes.*, logs-kubernetes.*, kubernetes.* fields). Being deprecated — do not author queries against these paths.
• APM-layer analysis (service SLO breaches, transaction error rates, upstream dependency health). Different domain — once a K8s root cause is ruled in or out, hand off to the observability-sre-triage skill, which owns SLO status and burn rate, active alerting rules, throughput, latency, error rate, dependency health, and log funnelling. That is also the skill to use when the workload turns out not to be Kubernetes-hosted at all.
• Cluster provisioning, capacity planning, cost optimization. Different domain.

Guidelines

These apply to every investigation. When in doubt, re-read them before writing the synthesis.

Absence of evidence is not evidence. Do not confabulate from empty results. If log queries return 0 rows, logs are likely not collected or the pod has no recent lines — this does not mean "dependency unavailable" or any other specific failure mode. Report no_logs_available and weight remaining signals accordingly.

Empty dependency data ≠ upstream healthy. Services without APM instrumentation (load generators, workers) emit no destination metrics. Report insufficient_dependency_data, not "upstreams OK."

Co-symptoms are not causes. Two services degrading simultaneously usually share an upstream, not a causal link. Only attribute causation when (a) one service's degradation clearly precedes the other's, and (b) the delta is large (>5× error rate, >3× latency).

OOMKilled ≠ memory leak by default. The limit might simply be undersized for the workload's working set. Tell them apart by the shape of the memory curve: a monotonic climb to the limit that resets on each restart, with load flat against the prior week and no recent deploy, is the leak signature — commit to it at high confidence. Reach for a 7-day same-hour baseline when the shape is ambiguous — spiky, diurnal, or load-correlated — not as a precondition for every OOMKilled finding.

Error-termination ≠ application bug by default. Check k8s.container.cpu_limit_utilization first. CFS throttling driving liveness probe timeouts is the most common misdiagnosis in this space.

Average CPU hides throttling. A pod can look healthy at 40–60% average cpu_limit_utilization while being throttled severely at p99. Linux enforces CPU limits in 100ms periods; bursty workloads reach quota mid-period and stall. Look at max and p95, not only the average.

Restart count is boolean, not a counter. k8s.container.restarts is pulled directly from the K8s API and can be pruned by the kubelet at any time, so the absolute value is unreliable. Treat it as == 0 (no recent restarts) versus > 0 (recently restarting); do not derive backoff timing or "linear versus exponential" patterns from it. Confirm the restart pattern via K8s Killing / BackOff events instead.

Prefer to report uncertainty over manufacturing confidence. If the evidence is ambiguous, the synthesis should say so. Competing hypotheses are a valid output.

Equally, do not manufacture uncertainty. The rule above is about ambiguous evidence, not about tone. When the pivotal signal is present and corroborated, commit to it at high confidence. Hedging an unambiguous finding down to "medium" is as much a defect as overclaiming.

Deliver the synthesis and stop. State confidence once, in the HYPOTHESIS line — not again per bullet. Do not narrate which queries were run unless a result changed the conclusion, and do not restate the alert back to the reader. End on RECOMMENDED NEXT STEPS or DOWNSTREAM IMPACT; never close with an offer such as "want me to look further?". Follow-up work belongs in the recommendations list, phrased as a recommendation.

Indices and fields

Where to look

Key fields

Flat OTel paths work in ES|QL. Prefer the flat form for readability; the nested resource.attributes.* form is for raw log documents only.

Container-level against pod-level limit utilization

Read limit utilization at the container level. k8s.container.cpu_limit_utilization and k8s.container.memory_limit_utilization are the default; the pod-level pair is a different measurement, not a synonym.

The receiver emits the two families on separate documents in the same data stream: pod-level fields appear on documents that carry no k8s.container.name, and container-level fields appear only on documents that do. A STATS ... BY k8s.container.name therefore returns null for every pod-level field, and the reverse holds too. Measured over one hour on a live 9.6.0 cluster: of 42,240 documents without a container name, 360 carried k8s.pod.cpu_limit_utilization and none carried the container field; of 18,240 documents with a container name, 900 carried the container field and none carried the pod field.

Availability differs too. Container-level utilization is emitted for each container that declares the limit, while the pod-level aggregate requires every container in the pod to declare it. Across two live clusters over three hours, no pod carried the pod-level field without also carrying the container-level one, while 27 pods carried the container-level field with the pod-level field absent — every one of them a multi-container pod in which only some containers declared limits. A pod-level throttling check on a sidecar-injected pod silently returns null.

Use the pod-level fields only when the question is genuinely about the pod as a whole — total consumption against the sum of its containers' limits — and only after confirming they are populated. observability-sre-triage applies the same rule, so the two skills return the same answer for the same pod.

Field availability

Several fields above are off by default in stock kube-stack collectors and require explicit configuration. Verify presence with GET /<index>/_mapping or GET /_field_caps before relying on them; if absent, fall back as noted and call out the substitution in the synthesis.

If a fall-back is used, note it in the synthesis (for example, (via memory.usage; limit_utilization not collected)) so the reader knows the signal is indirect.

ES|QL gotchas

Before writing queries, know these. Each of them silently produces wrong answers rather than failing loudly.

VALUES() returns scalar for single distinct value, array for multiple. Templating that assumes array shape (for example, | first) extracts the first character of the string when scalar. Use MV_FIRST(VALUES(...)) or handle both.

VALUES() is newer than this skill's base floor. It is GA on Serverless, but on Stack it is preview from 8.14.0 and GA only in 9.4.0, and it does not exist at all below 8.14. Check GET / before using it: build_flavor: "serverless" means it is available, otherwise read version.number. Where it is not available, move the field into the BY clause instead of aggregating it — one row per distinct value carries the same information:

| STATS restarts = MAX(k8s.container.restarts), phase = MAX(k8s.pod.phase)
    BY term_reason = k8s.container.status.last_terminated_reason
| SORT restarts DESC
| LIMIT 10

PERCENTILE does not work on OTel histogram type (as of 8.15). For APM duration percentiles, use AVG on the aggregate_metric_double summary field (AVG(transaction.duration.summary) divides sum by value_count). For true percentiles, fall back to Kibana Query DSL.

COUNT(agg_metric_double) returns value_count (events), not doc count. SUM(field) gives the sum component; AVG(field) gives sum/value_count. Do not use SUM(transaction.duration.summary) as an event-count proxy — it returns total duration.

K8s metrics use flat OTel field paths in ES|QL. k8s.pod.name, not resource.attributes.k8s.pod.name. The nested form is for raw log documents.

Failure-mode taxonomy

The classification vocabulary — pivotal signal and corroborating checks for each mode across the workload, node, control plane, autoscaling and networking layers, plus what to do when two modes fit — lives in references/failure-modes.md. Read it before classifying.

Signal interpretation

Memory

• Monotonic rise over 30–60 min → leak. Check GC metrics for the language: JVM jvm.gc.duration, Go process.runtime.go.gc.pause_ns, Node v8js_gc_duration. Rising GC frequency/pause with stable live-set is the canonical leak signature.
• Diurnal / load-correlated spikes → load-driven, not leak. Consider HPA tuning or limit increase.
• Hits 1.0, then restart → OOMKilled confirmed. Exit code 137 (SIGKILL) in app logs consistent.

CPU

• cpu_limit_utilization > 1.0 sustained → CFS throttling. Node has spare CPU; the pod is quota-blocked.
• Symptoms of throttling (not the throttle metric itself): liveness probe timeouts, p99 latency 4–16× p50, queue backpressure upstream, Error-reason container terminations.
• Average can look healthy while p95 is throttled. Do not trust average alone.