Gke Workload Troubleshooting

GKE Workload Troubleshooting Skill

Use this skill to systematically diagnose and resolve failures in application workloads deployed in GKE clusters. This skill operates non-interactively and enforces a read-only diagnostics boundary: it only proposes fixes — whether Kubernetes manifest/config patches or Google Cloud changes (for example gcloud IAM bindings or node-pool recreation) — and never executes live mutations itself.

🔍 Diagnostic Workflow

Step 0: Non-Interactive Context Discovery & Time Window Definition

1. Parameter Extraction: Extract required context (project_id, cluster_name, cluster_location, workload_name, workload_namespace) non-interactively from the user prompt, active SETTINGS.md, or active environment defaults:

   • Default workload_namespace to default if omitted.
   • Infer missing cluster parameters from active environment (kubectl config current-context or gcloud config get-value project).
   • Prioritize non-interactive context discovery from prompts and environment defaults to ensure autonomous execution flow.

2. Cluster Credentials & Fallback Mode:

   • Attempt credential fetch: gcloud container clusters get-credentials {cluster_name} --region/--zone {cluster_location}
   • Fallback / Dry-Run Mode: If the cluster is unreachable, non-existent, or live command execution fails (such as in sandboxed evaluations, dry-run mode, or offline analysis):
     • Limit retry attempts to avoid resource exhaustion and context overflow in unreachable cluster scenarios.
     • Immediately present the exact sequence of kubectl diagnostic commands for the human operator to run.
     • Synthesize the root cause analysis and output the proposed GitOps manifest fix based on the reported symptoms.

3. Time Handling & Fallbacks:

   • Determine Issue Timestamp ({issue_time}):
     • Specific Time Provided: If the user provides a specific timestamp, use it as {issue_time}.
     • Relative Time Provided (e.g., "5 minutes ago"): Dynamically calculate the corresponding UTC timestamp based on current system time, and use it as {issue_time}.
     • No Time Provided (Default): Use current system time as {issue_time}.
   • Window Calculation: Center a 1-hour query window around {issue_time} (start_time = {issue_time} - 30m, end_time = {issue_time} + 30m).

Step 1: Analyze Pod Status and Conditions

Inspect the workload's active pod states and controller status.

Diagnostic Commands:

# 1. Inspect the deployment's actual selector labels:
kubectl get deployment {workload_name} -n {workload_namespace} -o jsonpath='{.spec.selector.matchLabels}'
# 2. Query the pods using the returned labels, for example:
kubectl get pods -l {selector_labels} -n {workload_namespace}
kubectl get deploy/{workload_name} -n {workload_namespace} -o yaml

Diagnostic Decision Tree:

• Phase: Pending:

  • The Pod cannot schedule on any node. Proceed directly to Step 2 (Query Namespace Events).

• State: CrashLoopBackOff / Error:

  • The container boots but exits repeatedly; the kubelet restarts it with an increasing back-off delay of up to five minutes. First read the terminated reason and exit code:

  kubectl describe pod {pod_name} -n {workload_namespace}
  kubectl get pod {pod_name} -n {workload_namespace} -o jsonpath='{.status.containerStatuses[*].lastState.terminated}'

  • Reason: OOMKilled (Exit Code 137): The container's memory limit was reached. Proceed to Step 3 (Inspect Logs) → OOM Analysis to classify container-level vs node-level, then Step 5 to propose fixes.
  • Exit Code 0 (successful exit): Unexpected for a long-running Deployment/StatefulSet — restartPolicy: Always restarts the finished process, creating the loop. Common causes: the command/entrypoint does not start a persistent process, a worker exits on an empty queue, or a missing/invalid config (e.g., an unattached or mis-keyed ConfigMap volume) makes the app exit cleanly. Proceed to Step 3 (Inspect Logs).
  • Exit Code 128: Invalid command/entrypoint — the executable path is wrong or absent in the image. Verify the container command in the manifest.
  • Exit Code 1 or other non-zero: The application crashed — configuration errors, missing/invalid env vars or config files, unreachable dependencies, or auth failures (401/403) on Google Cloud calls (check the Pod's IAM / Workload Identity Federation). Proceed directly to Step 3 (Inspect Logs).
  • If the exit code looks healthy but the container keeps restarting, suspect a liveness probe failure (see Step 3).

• State: ImagePullBackOff / ErrImagePull:

  • The kubelet cannot pull the container image. ImagePullBackOff means it keeps retrying with back-off; ErrImagePull is a general, non-recoverable pull error. Related statuses: InvalidImageName, RegistryUnavailable, SignatureValidationFailed, ImageInspectError. Proceed to Step 2 (Query Namespace Events) to read the exact pull error message.

• State: ContainerCreating:

  • The container is blocked during volume mount, networking setup, or image pulling. Proceed directly to Step 2 (Query Namespace Events).

Step 2: Query Namespace Events

Look for infrastructure, volume, image, or scheduling alerts in GKE.

Diagnostic Command:

kubectl get events -n {workload_namespace} --sort-by='.metadata.creationTimestamp'
# Or query Cloud Logging for historical GKE events within the time window:
gcloud logging read "resource.type=\"k8s_cluster\" AND logName=\"projects/{project_id}/logs/events\" AND jsonPayload.involvedObject.namespace=\"{workload_namespace}\"" --start-time="{start_time}" --end-time="{end_time}" --project="{project_id}"
# Or query specifically for image pull failures within the time window:
gcloud logging read 'log_id("events") AND resource.type="k8s_pod" AND resource.labels.cluster_name="{cluster_name}" AND jsonPayload.message=~"Failed to pull image"' --project="{project_id}"

Note: Retrieve the sorted events list and manually inspect the event timestamps (CreationTimestamp/LastSeen) to identify failures occurring within the {start_time} and {end_time} window.

Signature Identifiers:

• FailedScheduling: Node resource exhaustion. Look for messages like 0/3 nodes are available: 3 Insufficient memory. or missing node affinity tolerations (e.g. Spot VM taints).

• FailedMount:

  • Missing PersistentVolumeClaim (PVC).
  • Missing Secret (Secret "{secret_name}" not found).
  • Missing ConfigMap (ConfigMap "{configmap_name}" not found).

• Failed / BackOff (Image Pull): First read the exact event message (Failed to pull image "IMAGE": ...) and triage by what it actually says. Do not jump to IAM / node service-account investigation unless the message is genuinely a permission or authentication error.

  • Wrong image name/tag — start here (not found, manifest unknown, InvalidImageName): the most common cause — the tag or path is wrong, or the image was deleted, frequently introduced by a recent deployment change.

    • Identify the failing container image name and the invalid tag.
    • Check the Git history for the last known working image tag: git log -p -S "{image_name}" -- {manifest_file_path} (or run git log on the folder containing manifests).
    • Propose reverting the image tag to the last working version (or correcting the tag) in the manifest patch.

  • Permission / authentication errors only (the message contains 403 Forbidden / denied, or 401 Unauthorized / unauthorized): the node cannot authorize or authenticate to the registry. Pursue the checks below only when the message matches.

    • 403 Forbidden (authorization) — the node pool service account (or the imagePullSecret's service account) is missing registry read access. Suggest granting it by presenting the following command for the user to review and run; do not execute it. For Artifact Registry:

      gcloud artifacts repositories add-iam-policy-binding {repository} \
        --location={repo_location} \
        --member="serviceAccount:{node_service_account_email}" \
        --role="roles/artifactregistry.reader"

      For Container Registry (gcr.io), grant roles/storage.objectViewer on the backing bucket (or the Artifact Registry role if gcr.io was migrated). Also check that any VPC Service Controls perimeter allows Artifact Registry.

    • 401 Unauthorized (authentication) — the node service account is disabled or the node lacks the required OAuth scope:

      gcloud container clusters describe {cluster_name} --location={cluster_location} \
        --format="table(nodePools.name,nodePools.config.serviceAccount)"
      gcloud iam service-accounts list \
        --filter="email:{node_service_account_email} AND disabled:true" --project={project_id}
      gcloud compute instances describe {node_name} --zone={node_zone} \
        --format="flattened(serviceAccounts[].scopes)"

      Scopes must include devstorage.read_only or cloud-platform (provided by gke-default). Nodes are immutable, so suggest recreating the node pool with --scopes="gke-default" if the scope is missing — present it as a proposed command for the user to run, do not execute it.

    • Private / self-hosted registry: ensure a valid imagePullSecret exists and is referenced by the Deployment.

  • Other statuses: RegistryUnavailable / i/o timeout / DNS server misbehaving → registry network path (DNS, firewall egress, Google API connectivity); exec format error or a deprecated schema-1 image → architecture/schema mismatch.