Gke AI Troubleshooting Jobset Interruption

GKE JobSet Interruption Troubleshooting

Use this skill to systematically diagnose and resolve JobSet interruptions, restarts, and preemptions on GKE clusters hosting large-scale AI/ML workloads.

⚠️ Prerequisites & Sandbox Rules

• JobSet metrics package must be enabled in kube-state-metrics for your cluster.
• Cloud Logging and Cloud Monitoring enabled for the Google Cloud Project.
• Autonomous & Sandbox Execution Rule: If API queries, tools, or commands encounter 403 Permission Denied, authentication errors, or network isolation, do NOT enter authentication or credential troubleshooting loops. Populate the query templates with the acquired variables ({project_id}, {cluster_name}, {workload_name}, {start_time}, {end_time}), inspect any locally staged telemetry or mock data files if available, and complete the diagnostic workflow and resolution recommendations autonomously.

🔍 Diagnostic Workflow

Step 0: Context Acquisition & Time Window Definition

Independently gather context using tools, workspace files, environment details, or user prompt context:

• Project ID ({project_id})
• Cluster Name ({cluster_name})
• Workload Name (JobSet Name) ({workload_name})
• Workload Namespace ({namespace})
• Issue Time ({issue_time})

If specific variables are not explicitly provided by the user, inspect cluster resources or logs to determine them, or use the {variable} placeholders provided.

Time Handling Rules

1. Autonomous Time Window: If a relative time (e.g., "X minutes ago") or no exact timestamp is provided, calculate the query window based on current time or available log timestamps.
2. Window Calculation: If a timestamp {issue_time} is available (or calculated as T), set {start_time} = T - 30m and {end_time} = T + 30m.

---

Step 1: Identify JobSet Restarts and Attempts [Low Risk]

Verify if the JobSet is experiencing restart loops and determine the frequency of restarts.

Visual Chart / MQL Query - restarts

• MQL Query Specification:

  fetch prometheus_target
  | metric 'prometheus.googleapis.com/kube_jobset_restarts/gauge'
  | filter resource.cluster_name == '{cluster_name}' && metric.jobset_name == '{workload_name}'
  | align next_older(1m)
  | every 1m
  | group_by [metric.jobset_name], [val: max(value)]

PromQL Metric Query - restarts

• PromQL Query Specification:

  kube_jobset_restarts{jobset_name="{workload_name}", cluster="{cluster_name}"}

• Diagnostic Logic: A non-zero or increasing value for restarts indicates that the JobSet is being actively restarted by the controller due to worker failure or interruption.

• Automation: Proceed to Step 2 automatically after reporting findings.

---

Step 2: Inspect Nodepool Interruptions [Low Risk]

Determine if the JobSet restarts were triggered by physical nodepool-level events (such as spot preemptions, maintenance, or host terminations).

A. Metrics Query (Nodepool Interruption Counts)

Visual Chart / MQL Query - interruptions

• MQL Query Specification:

  fetch k8s_node_pool
  | metric 'kubernetes.io/node_pool/interruption_count'
  | filter cluster_name == '{cluster_name}'
  | align next_older(10m)
  | every 10m
  | group_by [metric.interruption_type, metric.interruption_reason, metadata.system.node_pool_name], [val: sum(value)]

PromQL Query - interruptions

• PromQL Query Specification:

  sum by (interruption_type, interruption_reason, node_pool_name, cluster_name) (
    avg_over_time(kubernetes_io:node_pool_interruption_count{cluster_name="{cluster_name}"}[10m])
  )

B. Log Query (Nodepool Life Events)

• LQL Log Filter Specification:

  resource.type="gke_nodepool"
  AND resource.labels.cluster_name="{cluster_name}"
  AND timestamp >= "{start_time}"
  AND timestamp <= "{end_time}"

• Diagnostic Logic:

  • PreemptionEvent: Spot VMs were preempted, or node was scale-down.
  • MaintenanceEvent: Node pool updated or Google scheduled maintenance.
  • TerminationEvent: Serious host failures. Check interruption_reason or logs for host issues.
  • See Failure Signatures for examples of node termination logs and preemption events.

• Automation: Proceed to Step 3 automatically.

---

Step 3: Inspect Nodes and Underlying Host VMs [Low Risk]

Correlate node readiness failures with physical host VMs to see if a single faulty host repeatedly fails coordinator pods.

A. Metrics Query (Node Ready Status Check)

Visual Chart / MQL Query - node status

• MQL Query Specification:

  fetch k8s_node
  | metric 'kubernetes.io/node/status_condition'
  | filter cluster_name == '{cluster_name}' && metric.condition == 'Ready' && metric.status == 'False'
  | align next_older(1m)
  | every 1m
  | group_by [node_name, metadata.user.gke_nodepool], [val: max(value)]

PromQL Query - node status

• PromQL Query Specification:

  sum by (status, condition, node_pool_name) (
    kubernetes_io:node_status_condition{cluster_name="{cluster_name}", condition="Ready", status="False"}
  )

B. Metrics Query (Node-to-Host Metadata Topology Correlation)

• MQL Query Specification:

  fetch k8s_node
  | metric 'kubernetes.io/node/cpu/total_cores'
  | filter cluster_name == '{cluster_name}'
  | align next_older(1m)
  | every 1m
  | group_by [node_name, metadata.user.gce_topology_host, metadata.user.gke_nodepool], [val: max(value)]