Eks Upgrade Readiness

EKS Upgrade Readiness

Assess and plan Amazon EKS cluster upgrades with comprehensive pre-upgrade validation aligned with the EKS Best Practices Guide.

When to Use

Activate this skill when the user asks to:

• Check if an EKS cluster is ready to upgrade
• Plan an EKS version upgrade (control plane, node groups, or both)
• Identify deprecated Kubernetes APIs before upgrading
• Validate addon compatibility with a target version
• Assess node group upgrade strategy and capacity requirements
• Review Pod Disruption Budgets or topology spread for upgrade safety
• Understand EKS end-of-support, extended support, or auto-upgrade implications
• Evaluate Karpenter Drift or node expiry upgrade behavior
• Compare in-place vs blue-green upgrade strategies
• Create an upgrade runbook or checklist
• Detect GitOps/IaC version ownership before upgrading

Safety First

Before doing anything, load references/safety-invariants.md. It defines the knowledge hierarchy, hard rules, operation classification, and uncertainty handling. Keep it in context for the entire assessment.

Critical Warnings

• This skill is read-only. All commands are describe*, list*, get*. The agent does NOT execute mutating APIs. Mutations are in Step 14 and require explicit operator approval.
• One minor version at a time. EKS control plane upgrades proceed one minor version per operation (e.g., 1.30 → 1.31).
• Version skew policy. Before planning an upgrade, no kubelet may be newer than the current control plane. For the target version, kubelet may be no more than N-3 on 1.28+ (N-2 below 1.28).
• Addons must be upgraded AFTER the control plane (exceptions in Step 8).
• Auto-upgrade policy. Clusters past the 26-month lifecycle will be auto-upgraded. Proactive upgrade avoids disruption.
• Control plane rollback (July 2026+). 7-day rollback window after upgrade. Conditional, not guaranteed — skill checks eligibility.
• UNKNOWN ≠ PASS. Any gate that cannot be assessed MUST be UNKNOWN, never PASS. Overall verdict cannot be READY while any gate is UNKNOWN.

Evidence Completeness

Uses references/required-check-registry.yaml to track checks performed, skipped, or blocked. EC = checks_performed / total_applicable × 100%. EC < 50% produces a mandatory warning.

Grading and Confidence

False-positive guards:

• Empty query result ≠ PASS (mark UNKNOWN)
• No kubectl ≠ N/A for everything (AWS APIs still work)
• EKS Insights PASSING ≠ skip other checks (covers a subset only)
• Addon "compatible" ≠ "recommended"
• Pagination not exhausted → confidence LOW

Verdict rules (evaluate applicable gates only; N/A gates are excluded):

1. NOT READY: one or more applicable gates are FAIL. A known blocker wins over uncertainty because proceeding is unsafe.
2. CANNOT DETERMINE: no gate is FAIL, but one or more applicable gates are UNKNOWN (including inaccessible, incomplete, stale, or unpaginated data).
3. READY WITH WARNINGS: all applicable gates are assessed, none FAIL or UNKNOWN, and one or more are WARN.
4. READY: every applicable gate is PASS.

Format: [PASS|FAIL|WARN|UNKNOWN|N/A] (confidence: HIGH) — <evidence>

Cost Awareness

• EKS Insights API (Step 3) is free — always use first.
• CloudWatch Logs Insights cost ~$0.0076/GB scanned. Default to 60-min windows.
• Extended support costs $0.60/cluster/hour — upgrading saves money.
• Surge nodes incur temporary EC2 cost during overlap period.

Required Permissions

AWS IAM — see README.md "Prerequisites → IAM Permissions" for the full read-only action list (eks:Describe*, eks:List*, ec2:Describe*, autoscaling:Describe*, iam:GetRole, servicequotas:GetServiceQuota).

Kubernetes RBAC (only if kubectl access is available — the assessment still runs on AWS APIs alone without it, at lower confidence for CRD/Helm/PDB checks). Read-only ClusterRole covering every kubectl get/describe used in this skill:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: eks-upgrade-readiness-readonly
rules:
  - apiGroups: [""]
    resources:
      - nodes
      - pods
      - configmaps
      - secrets
      - events
      - persistentvolumeclaims
      - certificatesigningrequests
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "statefulsets", "daemonsets", "replicasets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["policy"]
    resources: ["poddisruptionbudgets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apiextensions.k8s.io"]
    resources: ["customresourcedefinitions"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["admissionregistration.k8s.io"]
    resources:
      - validatingwebhookconfigurations
      - mutatingwebhookconfigurations
    verbs: ["get", "list", "watch"]
  - apiGroups: ["karpenter.sh"]
    resources: ["nodepools", "nodeclaims"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["karpenter.k8s.aws"]
    resources: ["ec2nodeclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["crd.k8s.amazonaws.com"]
    resources: ["eniconfigs"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses", "csinodes"]
    verbs: ["get", "list", "watch"]

Bind with a ClusterRoleBinding to the identity the agent assumes (e.g. via IRSA/Pod Identity or an EKS access entry). secrets read access is required only for the Helm stored-manifest scan (Step 4) — omit that rule and accept UNKNOWN on Helm checks if a customer's security policy disallows it.

Step 1: Gather Cluster Context

aws eks describe-cluster --name <cluster-name> --region <region>

Extract: cluster.version, platformVersion, status (must be ACTIVE), kubernetesNetworkConfig, logging.clusterLogging (audit log must be enabled), resourcesVpcConfig.subnetIds, tags (IaC ownership detection).

Determine target version: ask user or default to current + 1 minor. Confirm target is in standard support via the EKS release calendar.

Step 2: Verify Infrastructure Prerequisites

Check these — failures are BLOCKERs:

1. Subnet IP availability — need ≥5 IPs per cluster subnet. Mode-aware: standard IPv4, prefix delegation, custom networking, IPv6, SGP. Use aws ec2 describe-subnets with cluster subnet IDs.
2. EKS IAM role — verify role exists with eks.amazonaws.com trust.
3. KMS key (if encryption enabled) — verify cluster role has key access.
4. Service quota headroom — EC2 vCPU (L-1216C47A) and EBS gp3 (L-7A658000) must have room for surge nodes. Use aws service-quotas get-service-quota.

VPC CNI mode and capacity-input detection:

kubectl get ds aws-node -n kube-system -o json | jq '
  .spec.template.spec.containers[0].env[]
  | select(.name | test("ENABLE_PREFIX_DELEGATION|AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG|ENABLE_POD_ENI|WARM_IP_TARGET|MINIMUM_IP_TARGET|WARM_ENI_TARGET|WARM_PREFIX_TARGET"))
  | {name, value}'

VPC CNI Surge-Capacity Gate

Do not treat "mode detected" as capacity validated. First calculate the managed-node-group surge using references/capacity-planning.md, distribute it by the node group's AZ placement, then verify the relevant subnet/ENI resource for the selected mode. Record the inputs and calculations as evidence; a missing mode-specific input is UNKNOWN, not PASS.

# Custom networking: inspect every AZ-to-pod-subnet mapping
kubectl get eniconfig -o json | jq '.items[] | {name: .metadata.name, subnet: .spec.subnet, securityGroups: .spec.securityGroups}'

# Security Groups for Pods: inspect trunk/branch interfaces after mode detection
aws ec2 describe-network-interfaces \
  --filters Name=interface-type,Values=trunk,branch \
  --query 'NetworkInterfaces[].{type:InterfaceType,subnet:SubnetId,instance:Attachment.InstanceId,status:Status}'

Client and CI Tooling Skew (WARN-level, not a blocker)

Operator/CI tooling that is too far behind the target version causes confusing failures during and after the upgrade. Check installed versions where available:

kubectl version --client -o json   # client minor version
eksctl version                      # if eksctl-managed
helm version --short                # if Helm-managed workloads

Do not hardcode exact version floors here — they shift every EKS release. Report the installed version, the rule, and a WARN if it cannot be confirmed current; never treat "tool not detected" as PASS.

Step 3: Check EKS Upgrade Insights

Primary authoritative signal. Always query first.

aws eks list-insights --cluster-name <cluster> \
  --filter '{"categories":["UPGRADE_READINESS"],"kubernetesVersions":["<target>"]}'
aws eks describe-insight --cluster-name <cluster> --id <insight-id>

Freshness gate: For every returned summary, capture lastRefreshTime and lastTransitionTime, then call DescribeInsight to collect the status, affected resources, and recommendation. Treat data as stale when lastRefreshTime is more than 24 hours old at assessment time, or predates a known relevant workload/addon change. A stale, missing, inaccessible, or unpaginated insight set is UNKNOWN; never reuse it as PASS. The assessment must not call StartInsightsRefresh because this skill is read-only. Instead, ask the operator to refresh insights through an approved workflow and rerun the assessment after the refresh completes.

Critical: Insights does NOT cover Helm stored manifests, CRD deprecations, StatefulSets, Karpenter, service quotas, PDBs, or capacity planning.

Step 4: API Deprecation Analysis

Check version-specific removal gates relevant to user's target:

• ≥1.33: AL2 AMI unavailable (Critical)
• ≥1.35: kube-proxy IPVS deprecated; ≥1.36: removed
• ≥1.25: Dockershim and PodSecurityPolicy removed
• ≥1.23: In-tree EBS provisioner deprecated

Helm stored manifests — the #1 missed blocker. Scan latest deployed release secrets for deprecated apiVersion lines:

kubectl get secrets -A -l owner=helm,status=deployed
# Decode: base64 -d | base64 -d | gunzip | jq -r '.manifest'

Third-party CRD deprecations — check Istio, cert-manager, Karpenter, Flux, Argo, Prometheus Operator versions against known deprecation timelines.

Tools: kubent, pluto detect-all-in-cluster, helm mapkubeapis --dry-run. See references/api-deprecations.md for full removal schedule.

Step 5: Addon Compatibility Check

Managed addons: Use live API to build compatibility matrix:

aws eks list-addons --cluster-name <cluster>
aws eks describe-addon --cluster-name <cluster> --addon-name <name>
aws eks describe-addon-versions --addon-name <name> --kubernetes-version <target>

Self-managed addons: First compare ListAddons with the actual kube-system workloads. Explicitly detect the three core addons: aws-node (VPC CNI), coredns, and kube-proxy. If a core component is absent from the managed-addon inventory but present in-cluster, mark it self-managed/custom and inspect its image, args, and configuration before assessing target support:

kubectl -n kube-system get daemonset aws-node kube-proxy -o json | \
  jq '.items[] | {name: .metadata.name, images: [.spec.template.spec.containers[].image], args: [.spec.template.spec.containers[].args]}'
kubectl -n kube-system get deployment coredns -o json | \
  jq '.items[] | {name: .metadata.name, images: [.spec.template.spec.containers[].image], args: [.spec.template.spec.containers[].args]}'
kubectl -n kube-system get configmap coredns aws-node -o yaml

For a custom CoreDNS Corefile, run the Corefile migration check. For VPC CNI, validate custom environment/config-map values and the mode-specific capacity gate in Step 2. For kube-proxy, validate its deployed mode and version against its upstream support policy. Then scan other self-managed components (aws-load-balancer, external-dns, metrics-server, cluster-autoscaler, cert-manager, ingress-nginx, argocd, flux); extract image tags and validate against their K8s support matrices.

Pod Identity Agent (eks-pod-identity-agent): Check like any other managed addon via DescribeAddon / DescribeAddonVersions — its version gates which association features are available (e.g. multiple associations per pod, target IAM role sessions). If not installed but kubectl get pods -A -o json shows service accounts with eks.amazonaws.com/role-arn annotations instead, the cluster is on IRSA, not Pod Identity — note this for blue-green planning (see references/upgrade-troubleshooting.md → Identity Migration Considerations).

Upgrade order: Pre-CP: Karpenter, Cluster Autoscaler, incompatible webhooks. Post-CP: kube-proxy → vpc-cni → coredns → CSI drivers → others → self-managed.

See references/addon-version-matrix.md for static fallback reference.