Gke Workload Scaling Troubleshooting

GKE Workload Scaling Troubleshooting Skill

Use this skill to systematically diagnose and resolve HorizontalPodAutoscaler (HPA) failures on GKE — metrics reported as <unknown>, FailedGet* events, missing resource requests, custom/external metrics-pipeline breakage, HPA that refuses to scale up or down, scale-to/from-zero issues, and slow HPA reaction on large clusters. This skill operates non-interactively and enforces a read-only diagnostics boundary before proposing manifest or configuration corrections.

For configuring HPA/VPA objects and scaling best practices, use the gke-workload-scaling skill instead. This skill focuses on failure diagnosis.

🔍 Diagnosis & Resolution Workflow

Step 0: Non-Interactive Context Discovery & Dry-Run Fallback

1. Parameter Extraction: Extract required context (project_id, cluster_name, cluster_location, hpa_name, workload_name, workload_namespace) non-interactively from the user prompt, active SETTINGS.md, or environment defaults:

   • Default workload_namespace to default if omitted.
   • Infer missing cluster parameters from the active environment (kubectl config current-context or gcloud config get-value project).

2. Cluster Credentials & Fallback Mode:

   • Attempt credential fetch: gcloud container clusters get-credentials {cluster_name} --location {cluster_location} --project {project_id}.
   • Fallback / Dry-Run Mode: If the cluster is unreachable, non-existent, or live command execution fails (such as in sandboxed evaluations, dry-run mode, or offline analysis):
     • Limit retry attempts to avoid resource exhaustion and context overflow.
     • Immediately present the exact kubectl / gcloud diagnostic commands for the human operator to run.
     • Synthesize the root-cause analysis and output the proposed GitOps correction based on the reported symptoms.

Step 1: Inspect the HPA and Classify the Symptom

Start every investigation with kubectl describe hpa, then route to the matching branch. The three key sections are Metrics (an <unknown> value means the HPA hasn't fetched the metric or the pipeline is broken), Conditions (AbleToScale, ScalingActive, ScalingLimited — a False status marks a failure), and Events (specific reasons such as FailedGetScale or FailedGetResourceMetric).

Diagnostic Commands:

kubectl describe hpa {hpa_name} -n {workload_namespace}
kubectl get hpa {hpa_name} -n {workload_namespace} -o yaml

For historical events, query Cloud Logging (the HPA events survive after the live Events list rolls over):

resource.type="k8s_cluster"
resource.labels.cluster_name="{cluster_name}"
resource.labels.location="{cluster_location}"
logName="projects/{project_id}/logs/events"
jsonPayload.involvedObject.kind="HorizontalPodAutoscaler"

Route by signal:

• FailedGetScale, FailedComputeMetricsReplicas, Error 400 ... label is not allowed, or fluctuating replicas from competing HPAs → Branch A (Configuration Errors).
• FailedGetResourceMetric, unable to fetch pod metrics, or multiple services selecting the same target → Branch B (Workload & Service Errors).
• <unknown> custom/external metric, FailedGetExternalMetric / FailedGetCustomMetric, or no known available metric versions found → Branch C (Metrics API & Data Availability).
• Conditions all True / no errors but the workload won't scale up or down → Branch D (Healthy but Unexpected Scaling).
• Workload configured with minReplicas: 0 won't scale to or from zero → Branch E (Scale To / From Zero).
• Correct HPA but slow reaction on a cluster with many HPA objects → Branch F (Slow Recalculation on Large Clusters).

Step 2: Resolution — Route to the Matching Branch

Based on the signal you classified in Step 1, jump to one of the mutually-exclusive branches below (A–F). These are alternatives — you do not run them in sequence. After applying the branch's fix, go to Step 3 to present it as a reviewable GitOps change.

Branch A: HorizontalPodAutoscaler Configuration Errors

• FailedGetScale — unable to get the target's current scale: ... "TARGET" not found: the scaleTargetRef doesn't resolve to an existing scalable workload.

  • Verify the scaleTargetRef name, kind, and apiVersion exactly match the target workload's metadata.
  • Confirm the target workload exists in the same namespace as the HPA (a missing -n puts objects in default, causing a mismatch).
  • The target must be a scalable kind (Deployment, StatefulSet, ReplicaSet) — you cannot autoscale a DaemonSet.

• FailedComputeMetricsReplicas — invalid metrics (1 invalid out of 1): the metric type and target don't match.

  • If type: Utilization, the target must be averageUtilization.
  • If type: AverageValue, the target must be averageValue.

• unable to fetch metrics from external metrics API: googleapi: Error 400: Metric label: 'LABEL' is not allowed: an invalid key in metric.selector.matchLabels.

  • Remove or correct the disallowed label; find valid filterable labels in the Cloud Monitoring metric documentation.

• Replica count fluctuates / contradictory SuccessfulRescale events from different HPAs: more than one HPA targets the same workload via spec.scaleTargetRef, and they compete. There is no dedicated condition for this — confirm with kubectl get hpa -n {workload_namespace} -o yaml and look for duplicate scaleTargetRef values.

  • Consolidate all metrics into one HPA object (it takes the highest of its spec.metrics) and delete the duplicates.

Branch B: Workload & Service Errors

• ScalingActive: False, reason FailedGetResourceMetric, message unable to compute the replica count (or a persistent unable to fetch pod metrics): the HPA computes utilization as a percentage of the container resource request, but at least one container in the Pod is missing a resources.requests entry for the scaled resource (cpu or memory).

  • Add resources.requests for the scaled resource to every container in the Pod spec (including sidecars). A brief unable to fetch pod metrics right after the metrics server starts is normal and self-heals.

• multiple services selecting the same target of HPA_NAME: SERVICE: traffic-based autoscaling requires a one-to-one Service↔workload relationship, but more than one Service's selector matches the workload's Pods.

  • Make the intended Service's selector unique (add a distinct label to the workload and to that one Service), or tighten the other Services' selectors so they no longer match the workload's Pods.

Branch C: Metrics API & Data Availability (Custom / External Metrics)

The custom/external pipeline is: HPA controller → Kubernetes metrics API server → metrics adapter (for example custom-metrics-stackdriver-adapter) → metric source (Cloud Monitoring / Prometheus). Symptoms are <unknown> metric values or FailedGetExternalMetric / FailedGetCustomMetric events.

1. Is the adapter registered and available?

   kubectl get apiservice | grep -E 'NAME|metrics.k8s.io'

   Expect v1beta1.custom.metrics.k8s.io and/or v1beta1.external.metrics.k8s.io with AVAILABLE: True. If False/missing, the adapter is crashed or misconfigured — inspect its Pod logs in the custom-metrics or kube-system namespace for permission, connectivity, or "metric not found" errors.

2. Query the metrics API directly (bypasses the HPA to test the whole pipeline; jq optional):

   kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/{workload_namespace}/{metric_name}" | jq .
   kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/{workload_namespace}/pods/*/{metric_name}" | jq .

3. Interpret the result:

   • Valid JSON with a value → the pipeline works; the fault is in the HPA manifest (metric-name typo or wrong matchLabels).
   • Error from server (Service Unavailable) → network isolation is blocking the control plane from reaching the adapter. Add the adapter's targetPort to the control-plane firewall rule (in addition to the existing tcp:443 and tcp:10250). Identify the rule with gcloud compute firewall-rules list --filter="name~gke-{cluster_name}-[0-9a-z]*-master", and also confirm no NetworkPolicy blocks ingress to the adapter Pods.
   • Empty list [] → the adapter runs but can't retrieve the metric. Inspect the adapter Pod logs, and confirm in Metrics Explorer that the metric actually exists in the source with the expected name and labels.

• unable to fetch metrics from custom metrics API: no known available metric versions found: a communication breakdown (control plane briefly unavailable during an upgrade/repair, or the adapter Pods are unhealthy or not registered) — not a problem at the metric source. Check control-plane health/notifications, confirm the adapter Pods are Running with no restarts (kubectl get pods -n custom-metrics,kube-system -o wide), and re-verify the APIServices are AVAILABLE: True. Often transient.

• googleapi: Error 400: The supplied filter ... will not return any time series: the query is valid but no data matched (different from a value of 0) — the application wasn't writing the metric during the window. Verify the metric name/labels match what the app emits, confirm the app had permission and was active, and check the app logs for metric-emission errors.