Gke AI Troubleshooting Node Unresponsive Timeout

Troubleshoot unresponsive GKE TPU and GPU nodes (NodeStatusUnknown)

When the Compute Engine host of a TPU or GPU node has a fatal hardware error, kernel panic, or non-maskable interrupt (NMI) lockup, the guest OS stops responding. The kubelet can no longer send heartbeats, so the node Ready condition becomes Unknown with Reason: NodeStatusUnknown (Kubelet stopped posting node status.). If node auto-repair is disabled on the node pool, GKE doesn't repair the node. The node can stay NotReady, and pods on it can stay in Terminating, which blocks multi-host JobSet workloads from recovering.

Prerequisites

• Tools: Install the Google Cloud SDK (gcloud) and kubectl.
• Cloud Billing & Project Configuration: Verify an active billing account is linked (gcloud billing projects describe {project_id}), authenticate (gcloud auth login), set the target project (gcloud config set project {project_id}), and ensure container.googleapis.com, compute.googleapis.com, logging.googleapis.com, and monitoring.googleapis.com are enabled.
• Required IAM Roles:
  • Kubernetes Engine Viewer (roles/container.viewer)
  • Compute Viewer (roles/compute.viewer)
  • Logs Viewer (roles/logging.viewer)
  • Monitoring Viewer (roles/monitoring.viewer)
  • For remediation ([High Risk] steps): Kubernetes Engine Cluster Admin (roles/container.clusterAdmin)
• Documentation:
  • Troubleshoot nodes with the NotReady status in GKE (Sections: "Check the node's status and conditions", "Confirm node preemption", "Verify that the node has recovered")
  • Deploy TPU workloads in GKE Standard (Sections: "Monitor health metrics for TPU nodes and node pools", "Configure auto repair for TPU slice nodes")
  • Troubleshoot OOM events
  • View GKE logs (Sections: "System logs")
  • Viewing serial port output
  • Troubleshoot Linux VM boot issues due to kernel panic
  • Auto-repair nodes (Sections: "Settings for Autopilot and Standard", "Verify node auto-repair is enabled for a Standard node pool", "Get information about recent automated repair events", "Enable auto-repair for an existing Standard node pool", "Repair criteria", "Node repair process", "Node auto repair in TPU slice nodes")
  • Troubleshooting VM shutdowns and reboots (Sections: "Querying Cloud Audit Logs", "Reviewing Cloud Audit Logs")

Read-only rule: Run read-only diagnostic commands only. Never drain, delete, or re-create nodes, or run any other command that changes the cluster. Give the user any fix to apply themselves.

When you recommend a fix, link the doc section that describes it.

Diagnostic workflow

Step 0: Collect context and set the investigation window [Low Risk]

Collect the target parameters. By default, query a 60-minute window `[T - 30m, T

• 30m]around{issue_time}`:

• {project_id}: Google Cloud project ID
• {cluster_name}: GKE cluster name
• {location}: Cluster region or zone
• {nodepool_name}: Target TPU or GPU node pool name
• {node_name}: Unresponsive GKE node name (and its Compute Engine {zone})
• {issue_time}: Incident timestamp in RFC3339 UTC
• {start_time}: {issue_time} - 30m
• {end_time}: {issue_time} + 30m

Step 1: Verify the NodeStatusUnknown heartbeat timeout [Low Risk]

1. Check Kubernetes node conditions: To inspect the node status and verify whether the Ready condition is Unknown with Reason: NodeStatusUnknown (Kubelet stopped posting node status.), follow the instructions in the section Check the node's status and conditions.
2. Query Cloud Logging (read-only LQL): Query k8s_node and k8s_cluster logs across [{start_time}, {end_time}] to confirm when the control plane lost heartbeat contact with {node_name}:

(resource.type="k8s_node" OR resource.type="k8s_cluster")
resource.labels.cluster_name="{cluster_name}"
("{node_name}" AND ("NodeNotReady" OR "NodeStatusUnknown" OR "Kubelet stopped posting node status"))
timestamp >= "{start_time}" AND timestamp <= "{end_time}"

3. Query Cloud Monitoring (read-only PromQL): Correlate the duration of the Unknown state using the GKE system metric kubernetes.io/node/status_condition (kubernetes_io:node_status_condition, GKE 1.32.1-gke.1357001+) documented in Monitor health metrics for TPU nodes and node pools, filtered by condition="Ready" and status="Unknown":

kubernetes_io:node_status_condition{
  monitored_resource="k8s_node",
  cluster_name="{cluster_name}",
  node_name="{node_name}",
  condition="Ready",
  status="Unknown"
}

• Decision logic:
  • If the node Ready condition is True and NodeStatusUnknown is absent, rule out an unresponsive node timeout and pivot to workload-level troubleshooting (for example, Troubleshoot OOM events) rather than repairing or draining the node.
  • If Ready is Unknown (NodeStatusUnknown), proceed to Step 2.

Step 2: Inspect serial port output for a kernel panic [Low Risk]

The guest OS can't send logs after a fatal kernel freeze, so check the serial port output of the node's VM:

• Cloud Logging: If serial port logging is enabled (i.e. VM metadata serial-port-logging-enable is set to true), GKE system logs in Cloud Logging include the node's serial port output. See the section System logs. Use Cloud Logging when the VM is stopped or has already been replaced by auto-repair, or when you need more than the most recent output.
• Running VM: Follow Viewing serial port output to retrieve the serial port 1 output (gcloud compute instances get-serial-port-output with --port=1) for {node_name} in {zone}. This method returns only the most recent 1 MB of output per port.

Consult Troubleshoot Linux VM boot issues due to kernel panic to identify documented kernel panic and hardware crash patterns (such as Fatal Machine check, hung_task: blocked tasks, or NMI: Not continuing) in the serial port output.