Gke Storage Troubleshooting

GKE Storage Troubleshooting Skill

Use this skill to systematically diagnose and resolve persistent-storage failures for workloads running on GKE — volume attach/mount errors, disk performance and node storage pressure, volume expansion, storage-related cluster/node-pool creation errors, and Cloud Storage FUSE memory issues. This skill operates non-interactively and enforces a read-only diagnostics boundary before proposing manifest or configuration corrections.

For routine storage provisioning and StorageClass/PVC authoring, use the gke-storage skill instead. This skill focuses on failure diagnosis.

🔍 Diagnosis & Resolution Workflow

Step 0: Non-Interactive Context Discovery & Dry-Run Fallback

1. Parameter Extraction: Extract required context (project_id, cluster_name, cluster_location, workload_name, workload_namespace, pod_name, and the relevant pvc_name / pv_name / node_name) non-interactively from the user prompt, active SETTINGS.md, or environment defaults:

   • Default workload_namespace to default if omitted.
   • Infer missing cluster parameters from the active environment (kubectl config current-context or gcloud config get-value project).

2. Cluster Credentials & Fallback Mode:

   • Attempt credential fetch: gcloud container clusters get-credentials {cluster_name} --location {cluster_location} --project {project_id}.
   • Fallback / Dry-Run Mode: If the cluster is unreachable, non-existent, or live command execution fails (such as in sandboxed evaluations, dry-run mode, or offline analysis):
     • Limit retry attempts to avoid resource exhaustion and context overflow.
     • Immediately present the exact kubectl / gcloud diagnostic commands for the human operator to run.
     • Synthesize the root-cause analysis and output the proposed GitOps correction based on the reported symptoms.

Step 1: Classify the Storage Symptom

Gather the primary signals, then jump to the matching branch under Step 2 (Resolution) — you normally perform only the one branch that matches your diagnosis, not all of them.

Diagnostic Commands:

kubectl describe pod {pod_name} -n {workload_namespace}
kubectl get pvc,pv -n {workload_namespace}
kubectl get events -n {workload_namespace} --sort-by='.metadata.creationTimestamp'
kubectl describe node {node_name}

• Pod stuck in ContainerCreating with an attach/mount event → Volume Attach & Mount Failures.
• Node-level slowness, PLEG is not healthy, or StoragePressureDetected events → Disk Performance & Node Storage Pressure.
• Cluster / node-pool creation or provisioning error → Storage Provisioning & Creation Failures.
• A resized volume is not reflected inside the container → Volume Expansion Not Reflecting in the Container.
• Cloud Storage FUSE Pod / sidecar OOM → Cloud Storage FUSE Out-Of-Memory (OOM) Events.

Step 2: Resolution

Perform only the branch that matches your Step 1 diagnosis. These branches are mutually exclusive alternatives, not sequential steps.

Volume Attach & Mount Failures

• Error 400: Cannot attach RePD to an optimized VM: Regional persistent disks are restricted from being used with memory-optimized or compute-optimized machine types.

  • If a regional PD is not a hard requirement, switch the workload to a non-regional persistent disk StorageClass.
  • If a regional PD is required, use taints and tolerations so that Pods needing regional PDs are scheduled onto a node pool that does not use optimized machine types.

• Pods stay Pending / FailedScheduling after a node pool is moved to a 4th-generation (N4, N4A, N4D) machine series while the workload uses a Persistent Disk StorageClass: N4/N4A/N4D machines do not support Persistent Disk (they support Hyperdisk only), so a PVC bound to a pd-* StorageClass cannot bind or schedule on those nodes. Events typically show FailedScheduling with a volume node-affinity / topology conflict.

  • Switch the workload to a Hyperdisk StorageClass (for example type: hyperdisk-balanced) for the Gen4 node pool.
  • For existing Persistent Disk volumes, migrate the data to a Hyperdisk volume; the original PD cannot be attached to a Gen4 node.
  • If the workload must keep Persistent Disk, keep it on a PD-capable machine series (for example N2) via node selection. This is a machine-type/disk-type incompatibility, not a capacity problem, so increasing disk size or quota does not help.

• Hyperdisk Pods become unschedulable when a compute class falls back across VM generations (for example N4 priority, N2 fallback), or one StorageClass must serve mixed generations: a single static disk type in the StorageClass is not compatible with every machine series in the fallback list, so Pods cannot bind their volume on the fallback nodes.

  • Use automated disk type selection: set the StorageClass parameters.type to dynamic with hyperdisk-type, pd-type, and disk-type-preference, plus use-allowed-disk-topology: "true", so GKE selects a compatible disk type per node and schedules Pods only onto nodes that support it. One dynamic StorageClass can then span multiple VM generations (requires the GKE versions noted in the docs).

    Example dynamic StorageClass (GKE 1.35.3-gke.1290000+):

    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: dynamic-volume
    provisioner: pd.csi.storage.gke.io
    volumeBindingMode: WaitForFirstConsumer
    allowVolumeExpansion: true
    parameters:
      type: dynamic
      pd-type: pd-balanced
      hyperdisk-type: hyperdisk-balanced
      # Preferred storage on nodes that support both PD and Hyperdisk;
      # defaults to hyperdisk-type when omitted.
      disk-type-preference: hyperdisk-type
      # Best practice: schedule Pods only onto nodes that support the disk type.
      use-allowed-disk-topology: "true"

• Mount stops responding due to the fsGroup setting: A Pod configured with a securityContext.fsGroup on a volume that contains a large number of files makes the kubelet recursively change ownership on every file, which can time out the mount. The symptom is:

  Unable to attach or mount volumes for pod; skipping pod ... timed out waiting for the condition

  Confirm by checking the Pod logs for a Setting volume ownership for ... and fsGroup set entry, then apply one of:

  • Reduce the number of files in the volume.
  • Set securityContext.fsGroupChangePolicy: OnRootMismatch so ownership is only changed when the top-level permissions do not match.
  • Stop using the fsGroup setting if it is not required.