K8s Incident Triage

Kubernetes Incident Triage

Performs read-only triage of Kubernetes workloads and namespaces using kubectl.

Supports:

• Natural language input (human mode)
• Structured JSON input (automation mode)

This skill NEVER mutates cluster state.

---

Tool Usage

This skill uses cli_execute with kubectl commands exclusively. NEVER use http_request or web_search to interact with Kubernetes. All cluster operations MUST go through kubectl via the cli_execute tool.

---

Tool: k8s_triage

Diagnose unhealthy Kubernetes workloads, pods, or namespaces.

---

Input Modes

1) Human Mode (Natural Language)

Input is a plain string.

Examples:

• triage payments-prod
• triage deployment payments-api in payments-prod
• why are pods pending in checkout-prod?
• investigate crashloop in payments-prod
• triage pod api-7c9f6d7f86-abcde in payments-prod
• check rollout of deployment payments-api in prod

Behavior:

• Parse namespace, workload, pod, or selector intent.
• If namespace omitted, use $DEFAULT_NAMESPACE if set.
• If ambiguity exists, default to namespace-level triage.
• Never require the user to remember JSON fields.

---

2) Automation Mode (Structured JSON)

Input JSON schema:

{ "namespace": "payments-prod", "workload_kind": "deployment", "workload_name": "payments-api", "pod_name": null, "label_selector": null, "include_logs": true, "logs_tail_lines": 200, "include_previous_logs": true, "events_limit": 50, "include_node_diagnostics": true, "include_metrics": false, "output_format": "markdown" }

Rules:

• namespace is required.
• If pod_name provided → pod-level triage.
• If workload fields provided → workload-level triage.
• Else → namespace scan.

---

Triage Process

Step 0 — Preconditions

Verify cluster access:

kubectl version --client kubectl cluster-info

If RBAC denies access:

• Continue with allowed operations.
• Explicitly report denied commands in output.

---

Step 1 — Fast Health Snapshot

Namespace scope:

kubectl get pods -n <ns> -o wide kubectl get deploy,sts,ds,job,cronjob -n <ns>

Workload scope:

kubectl get <kind> <name> -n <ns> kubectl rollout status <kind>/<name> -n <ns> --timeout=10s

Select pods in states:

• CrashLoopBackOff
• ImagePullBackOff
• ErrImagePull
• Pending / Unschedulable
• Error
• NotReady
• OOMKilled
• High restart count

Limit deep triage to $TRIAGE_MAX_PODS (default 5).

---

Step 2 — Events Timeline

kubectl get events -n <ns> --sort-by=.lastTimestamp | tail -n <events_limit>

Look for:

• FailedScheduling
• FailedMount
• Unhealthy (probe failures)
• Back-off pulling image
• Evicted
• Node pressure signals

---

Step 3 — Describe Pods & Workloads

For each selected pod:

kubectl describe pod <pod> -n <ns>

Capture:

• Container state and reason
• Restart count
• Last termination reason
• Probe failures
• Volume mount issues
• Node assignment
• Taints / tolerations
• Affinity constraints

If workload-level triage:

kubectl describe <kind> <name> -n <ns>

---