Gke Node Notready

GKE Node NotReady Troubleshooting Skill

Use this skill to systematically diagnose why one or more GKE nodes report a NotReady (or Ready: Unknown) status and to propose safe remediations. A NotReady status means the node's kubelet is not reporting to the control plane correctly, so Kubernetes stops scheduling new Pods on the node, which can reduce application capacity and cause downtime.

This skill operates non-interactively and enforces a read-only diagnostics boundary: gather evidence first, then propose a fix (a kubectl/gcloud command or a GitOps manifest change) for a human to apply. Never mutate the cluster, drain, delete, or recreate nodes automatically.

[!IMPORTANT] First rule out an expected NotReady: a node that is newly provisioning, upgrading, being repaired, cordoned, or scaling down will transiently report NotReady. Only treat it as a fault if it persists beyond the expected window.

🔍 Diagnostic Workflow

Step 0: Context discovery & time window

1. Parameter extraction — obtain project_id, cluster_name, cluster_location, and node_name non-interactively from the user prompt, active SETTINGS.md, or environment defaults (kubectl config current-context, gcloud config get-value project).
2. Credentials & fallback — attempt gcloud container clusters get-credentials {cluster_name} --location {cluster_location} --project {project_id}. If the cluster is unreachable or commands fail (sandbox/dry-run/offline), present the exact diagnostic commands for a human to run and continue the analysis from the reported symptoms.
3. Time window — determine {issue_time} (explicit, relative, or now) and center a 1-hour window around it (start = issue_time - 30m, end = issue_time + 30m) for all log/metric queries.

Step 1: Identify NotReady nodes and gather initial status

# List nodes and spot NotReady status, node IPs, and container-runtime version.
kubectl get nodes -o wide

# Inspect the affected node's Conditions and Events (the primary clues).
kubectl describe node "{node_name}"

Equivalent via Cloud Logging (preferred when kubectl access is limited or for historical events). Open it as a Logs Explorer deep link — URL-encode the query and append the project and Step 0 time window: https://console.cloud.google.com/logs/query;query={URL_ENCODED_QUERY};timeRange={start}%2F{end}?project={project_id} (encode / as %2F, or use ;duration=PT1H for a rolling hour):

resource.type="k8s_node"
log_id("events")
resource.labels.node_name="{node_name}"
resource.labels.cluster_name="{cluster_name}"
resource.labels.location="{cluster_location}"

Interpret the Conditions table:

• Ready: False / Ready: Unknown with reason KubeletNotReady / NodeStatusUnknown ("Kubelet stopped posting node status") → kubelet or runtime problem; continue to Step 2.
• MemoryPressure: True, DiskPressure: True, PIDPressure: True → resource exhaustion; go to Step 4b.
• NetworkUnavailable: True → networking/CNI problem; go to Step 4d.

Step 2: Scan kubelet logs for error signatures

Open these kubelet logs as a Logs Explorer deep link using the same logs/query;query={URL_ENCODED_QUERY};timeRange=...?project=... pattern as Step 1.

resource.type="k8s_node"
resource.labels.node_name="{node_name}"
resource.labels.cluster_name="{cluster_name}"
resource.labels.location="{cluster_location}"
log_id("kubelet")
severity>=WARNING

Also review the node's serial-console logs (log_id("serialconsole.googleapis.com/serial_port_1_output") or the resource.type="gce_instance" serial logs) for kernel TaskHung, OOM-killer, or disk I/O errors that correlate with the kubelet failures.

Step 3: Map the signature to a root cause (decision table)

Step 4: Branch investigations

4a. Container runtime (containerd) down

Confirm the kubelet cannot talk to containerd (socket errors above). Check for containerd restarts/crashes in serial logs. Remediation (propose, don't run): recreate/repair the node (kubectl drain then let the node pool recreate it, or gcloud container clusters upgrade/node auto-repair); if it recurs across nodes, suspect a node image or custom DaemonSet interfering with containerd.

4b. Resource pressure & OOM

# Node allocatable vs. usage.
kubectl describe node "{node_name}" | sed -n '/Allocated resources/,/Events/p'

Cloud Monitoring metrics to inspect (read-only): kubernetes.io/node/memory/used_bytes, kubernetes.io/node/cpu/core_usage_time, kubernetes.io/node/ephemeral_storage/used_bytes.

• DiskPressure / disk throttling: full boot disk or slow PD → increase disk size / use a faster PD type; reduce image/log churn.
• System OOM: node memory exhausted → set/raise Pod memory requests/limits, reduce over-commit, or use larger machine types. Distinguish system OOM (node-wide, kills kubelet/runtime) from cgroup OOM (single container).
• PIDPressure: too many processes → cap Pod PIDs / reduce workload density.