Gke Workload Identity

GKE Workload Identity Federation Troubleshooting Skill

Use this skill to systematically diagnose why a Pod using Workload Identity Federation for GKE cannot authenticate to Google Cloud APIs. Typical symptoms:

• HTTP/403 ... Permission 'iam.serviceAccounts.getAccessToken' denied on resource
• google.auth.exceptions ... could not find default credentials / ComputeEngineCredentials cannot find the metadata server
• API calls that unexpectedly use the node's default Compute Engine service account instead of the workload's identity.

Read-only boundary

This skill is diagnostic and non-interactive. It only reads cluster, IAM, and logging state and proposes fixes as commands or GitOps manifest changes for a human to apply. It must never create or modify IAM bindings, KSA annotations, node pools, or clusters automatically. When evidence is missing or the fix requires a privileged change, summarize findings and hand off to a human (see Step 7).

Output discipline (apply to every conclusion)

When you report a diagnosis, always:

1. Name the single most-likely root cause (not an open-ended list of possibilities).
2. Explicitly rule out the other plausible causes, citing the evidence that excludes them. In particular, when the cause is node-pool configuration, state plainly that it is not a KSA annotation or IAM binding problem; when the cause is a missing role on a target resource, state that Workload Identity itself is not misconfigured.
3. Give the exact remediation as a proposed change for a human — a concrete gcloud command (or GitOps manifest edit) with the real identifiers filled in — and never apply it automatically.

Diagnostic Workflow

Step 0: Context discovery & time window

Collect (from the user or the failing resource): PROJECT_ID, PROJECT_NUMBER, CLUSTER, cluster LOCATION, NAMESPACE, the KSA the Pod runs as, the node and node pool the Pod is scheduled on (used in Step 2), the target resource / API being called, and the exact error string. Define a time window around the first observed failure for log queries.

# Resolve the project number (used in the direct-binding principal identifier).
gcloud projects describe "{PROJECT_ID}" --format="value(projectNumber)"

# Confirm which KSA the workload runs as.
kubectl get pod "{pod_name}" -n "{namespace}" \
  -o jsonpath='{.spec.serviceAccountName}'

# Identify the node the Pod runs on, then the node pool that node belongs to
# (Step 2 checks the node pool's Workload Identity mode).
NODE=$(kubectl get pod "{pod_name}" -n "{namespace}" -o jsonpath='{.spec.nodeName}')
kubectl get node "$NODE" \
  -o jsonpath='{.metadata.labels.cloud\.google\.com/gke-nodepool}'

Step 1: Capture the exact error signature

Read the workload's own logs and the gke-metadata-server logs to classify the failure.

kubectl logs "{pod_name}" -n "{namespace}" --all-containers --prefix
kubectl describe pod "{pod_name}" -n "{namespace}"

Equivalent via Cloud Logging (preferred for historical events). Open it as a Logs Explorer deep link — URL-encode the query and append the project and time window: https://console.cloud.google.com/logs/query;query={URL_ENCODED_QUERY};timeRange={start}%2F{end}?project={project_id} (encode / as %2F, or use ;duration=PT1H for a rolling hour):

resource.type="k8s_container"
resource.labels.namespace_name="{namespace}"
resource.labels.pod_name="{pod_name}"
severity>=WARNING

Classify the signature:

• iam.serviceAccounts.getAccessToken denied / HTTP/403 → the workload is using the GSA impersonation path. This is expected for the legacy setup, but if you intend to use direct binding, it means the KSA still carries a leftover iam.gke.io/gcp-service-account annotation (or the client SDK is configured to impersonate) and is unintentionally impersonating a GSA; go to Step 3.
• 403 PERMISSION_DENIED on the target API/resource (no getAccessToken in the error) → the resolved identity lacks the required IAM role on that resource; go to Step 4.
• could not find default credentials / cannot find the metadata server → metadata-server connectivity or a startup race; go to Step 5.
• Calls succeed but as the node default service account → Workload Identity is not in effect for this node pool; go to Step 2.

Step 2: Verify Workload Identity is enabled (cluster + node pool)

Both the cluster and the node pool the Pod runs on must have Workload Identity enabled. A node pool with GCE_METADATA (instead of GKE_METADATA) causes Pods to fall back to the node's default Compute Engine service account.

# Cluster must have a workload identity pool (PROJECT_ID.svc.id.goog).
gcloud container clusters describe "{cluster}" --location "{location}" \
  --format="value(workloadIdentityConfig.workloadPool)"

# Node pool must have workloadMetadataConfig.mode = GKE_METADATA.
gcloud container node-pools describe "{node_pool}" --cluster "{cluster}" \
  --location "{location}" \
  --format="value(config.workloadMetadataConfig.mode)"

• Empty workload pool → Workload Identity is not enabled on the cluster.
• Node-pool mode is GCE_METADATA (or empty) → the node pool is not using the GKE metadata server; this is the usual cause of "runs as the node default service account". Remediation: enable --workload-metadata=GKE_METADATA on the node pool (propose to a human; recreates nodes). This is a node-pool configuration problem — not a KSA annotation or IAM binding problem — so do not change KSA annotations or IAM bindings to fix it. When the symptom is "runs as the node default service account", say so explicitly: the root cause is the node pool's workloadMetadataConfig.mode, and the KSA annotation and IAM bindings are ruled out as the cause. Propose the exact fix, e.g.:

gcloud container node-pools update "{node_pool}" --cluster "{cluster}" \
  --location "{location}" --workload-metadata=GKE_METADATA

Step 3: Verify the KSA → identity binding

There are two supported models. Prefer direct binding (current default); treat GSA impersonation as the legacy path.

How the two models fail differently: with direct binding the KSA principal accesses resources directly, so failures show up as a plain 403 PERMISSION_DENIED on the target API (fix in Step 4). A 403 iam.serviceAccounts.getAccessToken instead means an impersonation attempt — intended under the legacy path, or unintended if a leftover iam.gke.io/gcp-service-account annotation remains on a KSA that was meant to use direct binding.

(a) Direct KSA binding (no GSA impersonation). The KSA principal is granted roles directly. Construct the principal identifier and search for its bindings:

principal://iam.googleapis.com/projects/{PROJECT_NUMBER}/locations/global/workloadIdentityPools/{PROJECT_ID}.svc.id.goog/subject/ns/{NAMESPACE}/sa/{KSA_NAME}

(b) Legacy: KSA + GSA impersonation. The KSA must be annotated to point at a GSA, and the KSA must hold roles/iam.workloadIdentityUser on that GSA.

# The KSA annotation must reference the intended GSA.
kubectl get serviceaccount "{ksa_name}" -n "{namespace}" \
  -o jsonpath='{.metadata.annotations.iam\.gke\.io/gcp-service-account}'

# The GSA's IAM policy must bind the KSA member to workloadIdentityUser.
gcloud iam service-accounts get-iam-policy \
  "{gsa_name}@{project_id}.iam.gserviceaccount.com" \
  --format=json
# Expect a binding: role roles/iam.workloadIdentityUser,
# member serviceAccount:{PROJECT_ID}.svc.id.goog[{NAMESPACE}/{KSA_NAME}]

If the annotation is present but the binding is missing, the binding was likely removed — check the setIamPolicy audit logs around the failure time to find the responsible principal.

Step 4: Verify IAM permissions on the target resource

Even with a correct binding, the identity (the KSA principal for direct binding, or the GSA for legacy) must hold the role required by the API call (for example roles/storage.objectViewer). The standard IAM Policy Troubleshooter has limited support for Workload Identity principals; use Cloud Asset Inventory to search all IAM policies for the principal instead.

# Direct binding: search for the KSA principal's bindings across the project.
gcloud asset search-all-iam-policies \
  --scope="projects/{PROJECT_ID}" \
  --query='policy:"{PROJECT_ID}.svc.id.goog"'

# Legacy: search for the GSA's bindings on the target resource's project.
gcloud asset search-all-iam-policies \
  --scope="projects/{TARGET_PROJECT_ID}" \
  --query='policy:"{gsa_name}@{project_id}.iam.gserviceaccount.com"'

If no binding grants the required role on the target resource, that missing role is the root cause (and Workload Identity itself is not misconfigured). Present the fix as a proposed change for a human to apply — the exact add-iam-policy-binding with the real principal and role — never applying it automatically. For example, for direct binding on a project-level resource:

gcloud projects add-iam-policy-binding "{TARGET_PROJECT_ID}" \
  --member="principal://iam.googleapis.com/projects/{PROJECT_NUMBER}/locations/global/workloadIdentityPools/{PROJECT_ID}.svc.id.goog/subject/ns/{NAMESPACE}/sa/{KSA_NAME}" \
  --role="{REQUIRED_ROLE}"   # e.g. roles/storage.objectViewer