Troubleshoot Platform

Troubleshoot Platform

Overview

Systematic troubleshooting methodology for the workshop environment. Prioritizes investigation over immediate fixes and enforces safety gates on destructive operations.

Parameters

• symptom (required): Description of what's broken or unexpected
• component (optional): "eks", "terraform", "ingress", "argocd", "mcp", or "general"

Workflow

1. Verify the Problem Exists

Constraints:

• You MUST verify the actual problem exists before starting troubleshooting because symptoms may be transient
• You MUST validate YAML syntax after modifying any YAML file: yq eval '.' <file> > /dev/null for config files, kubectl apply --dry-run=client -f <file> for K8s manifests
• You MUST search EKS troubleshooting guide and documentation with 3-4 different query variations using EKS MCP tools before attempting fixes because the answer is usually documented

2. Investigate Systematically

For ArgoCD issues (ALWAYS check first):

Before any other investigation, check for stuck ArgoCD state:

# Check for apps stuck in Deleting state
kubectl get applications.argoproj.io -n argocd -o jsonpath='{range .items[*]}{.metadata.name}{" del="}{.metadata.deletionTimestamp}{"\n"}{end}' | grep "del=2"

# Check for stuck operations (Running > 5 min)
kubectl get applications.argoproj.io -n argocd -o json | jq -r --arg now "$(date -u +%Y-%m-%dT%H:%M:%SZ)" '.items[] | select(.status.operationState.phase == "Running" and ((.status.operationState.startedAt // .metadata.creationTimestamp) | strptime("%Y-%m-%dT%H:%M:%SZ") | mktime) < (($now | strptime("%Y-%m-%dT%H:%M:%SZ") | mktime) - 300)) | .metadata.name'

# Check for ComparisonError (git cache stale)
kubectl get applications.argoproj.io -n argocd -o json | jq -r '.items[] | select(.status.operationState.message // "" | contains("ComparisonError")) | .metadata.name + ": " + .status.operationState.message'

Recovery procedures:

1. Stuck operations (Running > 5 min): Terminate and re-trigger

   kubectl patch applications.argoproj.io <app> -n argocd --type merge -p '{"status":{"operationState":null}}'
   kubectl annotate applications.argoproj.io <app> -n argocd argocd.argoproj.io/refresh=hard --overwrite

2. Apps stuck Deleting: The EKS ArgoCD Capability controller re-adds finalizers. You cannot force-delete while the controller is running. Options:

   • Wait for the controller to complete cleanup (may take 5-10 min)
   • If the target cluster no longer exists, delete the cluster secret so the controller stops trying:

     kubectl delete secret <cluster-name> -n argocd

   • For apps targeting the hub itself: the controller will eventually succeed once it confirms resources are gone

3. Git cache stale ("app path does not exist"): The EKS ArgoCD Capability caches git repos. Force refresh:

   kubectl annotate applications.argoproj.io <app> -n argocd argocd.argoproj.io/refresh=hard --overwrite

   If that doesn't work, delete and let the ApplicationSet recreate:

   kubectl patch applications.argoproj.io <app> -n argocd --type merge -p '{"metadata":{"finalizers":[]}}'
   kubectl delete applications.argoproj.io <app> -n argocd --wait=false

4. All apps disappeared (ExternalSecret flicker): The hub cluster secret was momentarily reset, causing ArgoCD to delete all apps. Recovery:

   # Re-apply root-appset
   kubectl apply -f gitops/bootstrap/root-appset.yaml
   # Wait for bootstrap to regenerate child appsets
   # Then run argocd-sync to recover stuck apps
   ./scripts/argocd-sync.sh

5. Use the recovery script:

   ./scripts/argocd-sync.sh          # refresh all + terminate stuck ops
   ./scripts/argocd-sync.sh <app>    # refresh specific app

Key insight: With EKS ArgoCD Capability, you CANNOT force-remove finalizers — the managed controller re-adds them. The only way to unstick a deleting app is to fix the underlying issue (remove stale cluster secrets, wait for resource cleanup) or wait for the controller timeout.

CRITICAL: You MUST ALWAYS ask the user before deleting any ArgoCD Application or ApplicationSet. Deleting an app with preserveResourcesOnDeletion: false (the default) will CASCADE-DELETE all Kubernetes resources it manages. This can destroy entire namespaces, deployments, secrets, and PVCs. Even deleting a single ApplicationSet can wipe out dozens of apps and their resources across multiple clusters. NEVER delete ArgoCD apps without explicit user confirmation.

For infrastructure issues:

• Use terraform state list then terraform state show <resource> before checking AWS CLI because Terraform state is the source of truth
• For networking issues, examine Terraform config files first
• You MUST NOT use get_eks_vpc_config MCP tool because it returns incomplete data — use Terraform state instead