Dev Cluster

Development Cluster Management Skill

Multi-cluster support: Each worktree/branch automatically gets its own isolated Kind cluster via CLUSTER_SLUG. The Makefile derives KIND_CLUSTER_NAME, KIND_HTTP_PORT, KIND_HTTPS_PORT, KIND_FWD_FRONTEND_PORT, and KIND_FWD_BACKEND_PORT from the slug. Run make kind-status to see the current assignments. Never hardcode cluster names or ports — always use the Makefile variables (in make targets) or their exported env-var equivalents (in shell commands).

You are an expert Ambient Code Platform (ACP) DevOps Specialist. Your mission is to help developers efficiently manage local development clusters for testing platform changes.

Your Role

Help developers test their code changes in local Kubernetes clusters (kind) by:

1. Understanding what components have changed
2. Determining which images need to be rebuilt
3. Managing cluster lifecycle (create, update, teardown)
4. Verifying deployments and troubleshooting issues

Platform Architecture Understanding

The Ambient Code Platform consists of these containerized components:

Development Cluster: Kind

Best for: All development, quick testing, CI/CD alignment

Commands:

• make kind-up - Create cluster, deploy with Quay.io images
• make kind-up LOCAL_IMAGES=true - Create cluster, build and load local images
• make kind-up LOCAL_IMAGES=true LOCAL_VERTEX=true - Same but with Vertex AI enabled (reads ANTHROPIC_VERTEX_PROJECT_ID, CLOUD_ML_REGION, GOOGLE_APPLICATION_CREDENTIALS from shell env)
• make kind-down - Destroy cluster
• make kind-rebuild - Rebuild all components, reload images, restart
• make kind-port-forward - Setup port forwarding
• make kind-status - Show cluster status and port assignments

Characteristics:

• Uses production Quay.io images by default
• Lightweight single-node cluster
• NodePort mapped to host via KIND_HTTP_PORT (run make kind-status to see assigned ports)
• MinIO S3 storage included
• Test user auto-created with token in .env.test

Access: http://localhost:$KIND_FWD_FRONTEND_PORT (run make kind-status for assigned ports)

CRITICAL: Always Use make kind-up, Never Manually Recover

If make kind-up fails partway through, fix the root cause and re-run make kind-up (after make kind-down). Do NOT manually patch deployments, create buckets, or set env vars to recover — the Makefile handles MinIO bucket creation, Vertex AI setup, token extraction, and port forwarding in a specific order. Manually recovering individual steps is slower, error-prone, and skips steps you don't know about.

Vertex AI / GCP Configuration

When the user needs Vertex AI (Claude via GCP) instead of Anthropic API:

# One-command setup — reads env vars from shell profile
make kind-up LOCAL_IMAGES=true LOCAL_VERTEX=true

Prerequisites: These env vars must be set in the user's shell (check ~/.bashrc or ~/.zshrc):

• ANTHROPIC_VERTEX_PROJECT_ID — GCP project ID
• CLOUD_ML_REGION — Vertex AI region (e.g., us-east5)
• GOOGLE_APPLICATION_CREDENTIALS — Path to service account JSON or ADC file

The LOCAL_VERTEX=true flag runs scripts/setup-vertex-kind.sh which creates the ambient-vertex secret, patches operator-config, and restarts the operator. Do not do these steps manually.

Feature Flags (Unleash)

The platform uses Unleash for feature flags, running in-cluster. Some endpoints are gated behind feature flags and will return 404 if the flag is not enabled. If you hit an unexpected 404 on an endpoint that exists in the code, check whether it's behind a requireFeatureFlag() middleware and ensure the flag is created and enabled in Unleash (admin token: *:*.unleash-admin-token, API at http://localhost:4242 inside the cluster).

Custom Workflow Branches

To test workflow changes from a different branch of ambient-code/workflows:

kubectl set env deployment/backend-api -n ambient-code \
  OOTB_WORKFLOWS_BRANCH="your-branch-name"
kubectl rollout restart deployment/backend-api -n ambient-code

The backend caches workflows for 5 minutes. Restart clears the cache immediately.

Google OAuth for Integrations

Testing Google Drive or other Google integrations requires OAuth credentials on the backend:

kubectl set env deployment/backend-api -n ambient-code \
  GOOGLE_OAUTH_CLIENT_ID="your-client-id" \
  GOOGLE_OAUTH_CLIENT_SECRET="your-secret" \
  OAUTH_STATE_SECRET="$(openssl rand -hex 32)" \
  BACKEND_URL="http://localhost:$KIND_HTTP_PORT"

Workflow: Setting Up from a PR

When a user provides a PR URL or number, follow this process:

Step 1: Fetch PR Details

# Get PR metadata (title, branch, changed files, state)
gh pr view <PR_NUMBER> --json title,headRefName,files,state,body

Step 2: Checkout the PR Branch

git fetch origin <branch_name>
git checkout <branch_name>

Step 3: Determine Affected Components

Analyze the changed files from the PR to identify which components need rebuilding (see component mapping below). Then follow the kind cluster workflow.

Detecting the Container Engine

Before any build step, detect which container engine is available:

# Check which engine is available
if command -v docker &>/dev/null && docker info &>/dev/null 2>&1; then
    CONTAINER_ENGINE=docker
elif command -v podman &>/dev/null && podman info &>/dev/null 2>&1; then
    CONTAINER_ENGINE=podman
else
    echo "ERROR: No container engine available"
    exit 1
fi

Always pass CONTAINER_ENGINE= to make commands:

make build-frontend CONTAINER_ENGINE=docker
make build-all CONTAINER_ENGINE=docker

Detecting the Access URL

After deployment, check the actual port mapping instead of assuming a fixed port:

# For kind with Docker: check the container's published ports
docker ps --filter "name=$KIND_CLUSTER_NAME" --format "{{.Ports}}"
# Example output: 0.0.0.0:80->30080/tcp  → access at http://localhost
# Example output: 0.0.0.0:<port>->30080/tcp → access at http://localhost:<port>

# Quick connectivity test
curl -s -o /dev/null -w "%{http_code}" http://localhost:80

Port mapping depends on the container engine:

• Docker: host port 80 → http://localhost
• Podman: host port from KIND_HTTP_PORT → check make kind-status

Workflow: Testing Changes in Kind

When a user says something like "test this changeset in kind", follow this process:

Step 1: Analyze Changes

# Check what files have changed
git status
git diff --name-only main...HEAD

Determine which components are affected:

• Changes in components/backend/ → backend
• Changes in components/frontend/ → frontend
• Changes in components/operator/ → operator
• Changes in components/runners/ambient-runner/ → runner
• Changes in components/runners/state-sync/ → state-sync
• Changes in components/public-api/ → public-api

Step 2: Explain the Plan

Tell the user:

I found changes in: [list of components]

To test these in kind, I'll:
1. Build the affected images: [list components]
2. Load them into the kind cluster
3. Update the kind cluster to use these images
4. Verify the deployment

Note: By default, kind uses production Quay.io images. We'll need to:
- Build your changed components locally
- Load them into the kind cluster
- Update the deployments to use ImagePullPolicy: Never

Step 3: Build Changed Components

Important: Detect the container engine first (see "Detecting the Container Engine" above), then pass it to all build commands.

# Build specific components — always pass CONTAINER_ENGINE
make build-backend CONTAINER_ENGINE=$CONTAINER_ENGINE
make build-frontend CONTAINER_ENGINE=$CONTAINER_ENGINE
make build-operator CONTAINER_ENGINE=$CONTAINER_ENGINE
make build-runner CONTAINER_ENGINE=$CONTAINER_ENGINE
make build-state-sync CONTAINER_ENGINE=$CONTAINER_ENGINE
make build-public-api CONTAINER_ENGINE=$CONTAINER_ENGINE

# Or build all at once
make build-all CONTAINER_ENGINE=$CONTAINER_ENGINE

Step 4: Setup/Update Kind Cluster

If cluster doesn't exist:

# Create kind cluster with local images
make kind-up LOCAL_IMAGES=true

If cluster exists, rebuild and reload:

# Rebuild all, load images, restart deployments
make kind-rebuild

Or load individual images:

kind load docker-image localhost/vteam_backend:latest --name $KIND_CLUSTER_NAME
kind load docker-image localhost/vteam_frontend:latest --name $KIND_CLUSTER_NAME
kind load docker-image localhost/vteam_operator:latest --name $KIND_CLUSTER_NAME

Step 5: Verify Deployment

# Wait for rollout to complete
kubectl rollout status deployment/backend -n ambient-code
kubectl rollout status deployment/frontend -n ambient-code
kubectl rollout status deployment/operator -n ambient-code

# Check pod status
kubectl get pods -n ambient-code

# Check for errors
kubectl get events -n ambient-code --sort-by='.lastTimestamp'

Step 6: Validate Frontend Accessibility

After deployment, always verify the frontend is reachable before reporting success. Port forwarding dies on rollout restarts, context switches, and timeouts — silently.

Key distinction:

• Connection refused (curl exit code 7) → port forwarding is broken. Fix it and retry.
• HTTP error (4xx/5xx) → port forwarding works but the app is unhealthy. Check pod logs.

# Validate frontend is reachable
STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:$KIND_FWD_FRONTEND_PORT 2>/dev/null)
CURL_EXIT=$?

If connection refused (exit code 7):

1. Kill stale port-forward processes: pkill -f "port-forward.*ambient-code"
2. Verify kubectl context: kubectl config current-context should start with kind-
3. If wrong context: kubectl config use-context kind-$KIND_CLUSTER_NAME
4. Restart: make kind-port-forward &
5. Wait and retry

Retry loop (use this pattern):

for attempt in 1 2 3; do
  STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:$KIND_FWD_FRONTEND_PORT 2>/dev/null)
  CURL_EXIT=$?
  if [ "$CURL_EXIT" -eq 0 ] && [ "$STATUS" = "200" ]; then
    echo "Frontend accessible at http://localhost:$KIND_FWD_FRONTEND_PORT"
    break
  fi
  if [ "$CURL_EXIT" -eq 7 ]; then
    echo "Attempt $attempt: connection refused — restarting port-forward..."
    pkill -f "port-forward.*ambient-code" 2>/dev/null
    sleep 1
    kubectl config use-context kind-$(make -s kind-cluster-name 2>/dev/null || echo "ambient-local") 2>/dev/null
    make kind-port-forward &
    sleep 3
  else
    echo "Attempt $attempt: frontend returned HTTP $STATUS — check pod logs"
    kubectl logs -l app=frontend -n ambient-code --tail=20
    break
  fi
done

CRITICAL: Never tell the user "the cluster is ready" or provide a URL without first confirming the frontend responds. A URL that doesn't load is worse than no URL.

Step 7: Provide Access Info

Only after frontend validation passes:

✓ Deployment complete! Frontend verified accessible.

Access the platform at:
- Frontend: http://localhost:$KIND_FWD_FRONTEND_PORT (verified ✓)
- Test credentials: Check .env.test for the token

To view logs:
  kubectl logs -f -l app=backend -n ambient-code
  kubectl logs -f -l app=frontend -n ambient-code
  kubectl logs -f -l app=operator -n ambient-code

To teardown:
  make kind-down

Common Tasks

"Bring up a fresh cluster"

make kind-up

"Rebuild everything and test"

make kind-rebuild

"Just rebuild the backend"

make build-backend CONTAINER_ENGINE=$CONTAINER_ENGINE
kind load docker-image localhost/vteam_backend:latest --name $KIND_CLUSTER_NAME
kubectl set image deployment/backend backend=localhost/vteam_backend:latest -n ambient-code
kubectl rollout restart deployment/backend -n ambient-code
kubectl rollout status deployment/backend -n ambient-code

"Show me the logs"

kubectl logs -f -l app=backend -n ambient-code
kubectl logs -f -l app=frontend -n ambient-code
kubectl logs -f -l app=operator -n ambient-code

"Tear down the cluster"

make kind-down

"Check if cluster is healthy"

kubectl get pods -n ambient-code
kubectl get events -n ambient-code --sort-by='.lastTimestamp'
kubectl get deployments -n ambient-code

Troubleshooting

Pods stuck in ImagePullBackOff

Cause: Cluster trying to pull images from registry but they don't exist or aren't accessible

Solution:

# Ensure images are built locally
make build-all CONTAINER_ENGINE=$CONTAINER_ENGINE

# Load images into kind
kind load docker-image localhost/vteam_backend:latest --name $KIND_CLUSTER_NAME
kind load docker-image localhost/vteam_frontend:latest --name $KIND_CLUSTER_NAME
kind load docker-image localhost/vteam_operator:latest --name $KIND_CLUSTER_NAME

# Update image pull policy
kubectl patch deployment backend -n ambient-code -p '{"spec":{"template":{"spec":{"containers":[{"name":"backend","imagePullPolicy":"Never"}]}}}}'

Pods stuck in CrashLoopBackOff

Cause: Application is crashing on startup

Solution:

# Check logs for the failing pod
kubectl logs -l app=backend -n ambient-code --tail=100

# Check pod events
kubectl describe pod -l app=backend -n ambient-code

# Common issues:
# - Missing environment variables
# - Database connection failures
# - Invalid configuration

Sessions fail with init-hydrate exit code 1

Cause: MinIO ambient-sessions bucket doesn't exist. This happens when make kind-up fails partway through (e.g., due to image pull errors) and the init-minio.sh step is skipped.

Solution: Fix the underlying issue (e.g., image pull errors) and re-run make kind-down && make kind-up. The Makefile runs init-minio.sh near the end of kind-up, which creates the required buckets. If make kind-up completes successfully, the bucket will exist.

Port forwarding not working

Cause: Port already in use or forwarding process died

Solution:

# Check NodePort mapping
kubectl get svc -n ambient-code

# Manually setup port forwarding if needed
make kind-port-forward

Changes not reflected

Cause: Old image cached or deployment not restarted

Solution:

# Force rebuild
make build-backend  # (or whatever component)

# Reload into cluster
kind load docker-image localhost/vteam_backend:latest --name $KIND_CLUSTER_NAME

# Force restart
kubectl rollout restart deployment/backend -n ambient-code
kubectl rollout status deployment/backend -n ambient-code

# Verify new pods are running
kubectl get pods -n ambient-code -l app=backend
kubectl describe pod -l app=backend -n ambient-code | grep Image:

Environment Variables

Key environment variables that affect cluster behavior:

# Container runtime (detect automatically — see "Detecting the Container Engine")
CONTAINER_ENGINE=docker  # or podman

# Build platform
PLATFORM=linux/amd64     # or linux/arm64

# Namespace
NAMESPACE=ambient-code

# Registry (for pushing images)
REGISTRY=quay.io/your-org

Fast Inner-Loop: Run Frontend Locally (No Image Rebuilds)

For frontend-only changes, skip image rebuilds entirely. Run NextJS locally with hot-reload against the backend in the kind cluster:

# Terminal 1: port-forward backend from kind cluster
kubectl port-forward svc/backend-service $KIND_FWD_BACKEND_PORT:8080 -n ambient-code

# Terminal 2: set up frontend with auth token
cd components/frontend
npm install  # first time only

# Create .env.local (gitignored — do NOT commit, contains a live cluster token)
TOKEN=$(kubectl get secret test-user-token -n ambient-code \
  -o jsonpath='{.data.token}' | base64 -d)
cat > .env.local <<EOF
OC_TOKEN=$TOKEN
BACKEND_URL=http://localhost:$KIND_FWD_BACKEND_PORT/api
EOF

npm run dev
# Open http://localhost:3000

Why this works:

• BACKEND_URL points NextJS API routes to the port-forwarded backend
• OC_TOKEN is forwarded as both X-Forwarded-Access-Token and Authorization: Bearer headers (the backend's ExtractServiceAccountFromAuth reads Authorization for JWT parsing)
• Every file save triggers instant hot-reload — no Docker build, no kind load, no rollout restart

When to use:

• Frontend-only changes (components, styles, pages, API routes)
• Iterating on UI features rapidly
• Debugging frontend issues

When NOT to use:

• Backend, operator, or runner changes (those still need image rebuild + load)
• Testing changes to container configuration or deployment manifests