Demo Local Rollout

Local Image Rollout

Use this skill for unpublished sha-<git-sha> test tags. Formal releases use semver and $release-cut-and-demo-roll.

Choose the cluster first

Do this before Harbor auth, Bazel, signing, or any cluster write.

Never apply the OpenBao / Kyverno / Argo path to farm01. Never helm-upgrade farm01 thinking it is demo.

If only elixir/web-ng/** changed and the target is demo, prefer $demo-web-ng-fastpath. That fast path is demo-only.

Shared workflow

1. Work from the checkout that has the commits under test (often a git worktree).
2. Symlink gitignored Bazel rc files if this is not the primary clone (see below).
3. Tag: sha-$(git rev-parse HEAD) (full 40-char SHA; that is what --stamp emits).
4. Read the currently deployed tag on the chosen cluster before changing anything.
5. Map git diff --name-only <deployed>..HEAD to images. Rebuild only what changed.
6. Harbor auth: ./buildbuddy_setup_docker_auth.sh
7. Build and push changed images. Copy unchanged images from the live tag to the new tag.
8. Then follow the cluster-specific roll section. Stop after that section; do not run the other cluster's roll.

Do not cut a release, edit VERSION, or create git tags. Do not roll demo-staging, production, or Docker Compose unless the user changes scope. Never push to staging.

Worktree Bazel remotes

.bazelrc try-imports %workspace%/.bazelrc.remote and .bazelrc.local. Both are gitignored (BuildBuddy API key). git worktree add does not copy them. Without them, --config=remote_base / --config=ci fails with PERMISSION_DENIED: Missing API key.

From the primary clone that already has the files:

PRIMARY=/Users/mfreeman/src/serviceradar
WT="$(pwd)"
ln -sfn "$PRIMARY/.bazelrc.remote" "$WT/.bazelrc.remote"
test -e "$PRIMARY/.bazelrc.local" && ln -sfn "$PRIMARY/.bazelrc.local" "$WT/.bazelrc.local"
test -f "$WT/.bazelrc.remote"

Changed image selection

git diff --name-only <currently-deployed-sha-or-tag>..HEAD

• go/cmd/agent/**, go/pkg/agent/**, go/pkg/mtr/** -> serviceradar-agent
• elixir/serviceradar_agent_gateway/**, shared agent control/proto -> serviceradar-agent-gateway
• elixir/serviceradar_core/**, rust/srql/** (NIF) -> serviceradar-core-elx (and usually serviceradar-web-ng)
• elixir/web-ng/** -> serviceradar-web-ng
• proto/** -> every image that consumes the changed generated code

When in doubt, rebuild slightly too much. Phoenix / mix.lock / MODULE.bazel / rules_elixir changes mean rebuild every Elixir image you will roll.

Images that use global.imageTag

Copy or rebuild every first-party image the target cluster will pull at the new tag. Inspect live deployments; do not copy demo-only images onto farm01.

Typical demo set: arancini, serviceradar-agent, serviceradar-agent-gateway, serviceradar-core-elx, serviceradar-datasvc, serviceradar-db-event-writer, serviceradar-faker, serviceradar-flow-collector, serviceradar-log-collector, serviceradar-rperf-client, serviceradar-tools, serviceradar-trapd, serviceradar-trivy-sidecar, serviceradar-web-ng, serviceradar-zen.

Typical farm01 set: serviceradar-agent, serviceradar-agent-gateway, serviceradar-core-elx, serviceradar-datasvc, serviceradar-flow-collector, serviceradar-log-collector, serviceradar-rperf-client, serviceradar-tools, serviceradar-trapd, serviceradar-trivy-sidecar, serviceradar-web-ng. farm01 usually has no faker, zen, bmp/arancini, or k8s-inventory.

serviceradar-log-collector-tcp shares the log-collector image. Do not invent a separate tag for it.

Build and push changed images

crane is on PATH (Homebrew). Do not assume /tmp/gobin/crane.

Staging no longer defines the legacy remote / remote_push profiles; the surviving ones are remote_base, cache_only, and ci.

On a Darwin workstation, running an oci_push target through Bazel builds linux/amd64 on RBE and then tries to execute linux jq/crane from runfiles (Exec format error). On macOS use make push_all, which routes through scripts/push_all_images.sh and pushes with host crane.

Build the OCI layouts remotely, then push with host crane:

bazel build \
  --config=remote_base \
  --noenable_platform_specific_config \
  --remote_download_outputs=all \
  --compilation_mode=opt \
  --stamp \
  //docker/images:agent_image_amd64 \
  //docker/images:agent_gateway_image_amd64 \
  //docker/images:core_elx_image_amd64 \
  //docker/images:web_ng_image_amd64

Layouts land under bazel-out/rbe_platform-opt/bin/docker/images/<name>_image_amd64. Bazel blob files are often symlinks and mode 0444; crane push of the layout fails with layout blob ... is a symlink. Materialize, push by digest, tag only sha-<commit> (do not retag latest):

BIN="$(bazel info output_path)/rbe_platform-opt/bin/docker/images"
# or the execroot path printed by the build
SRC="$BIN/agent_image_amd64"
DEST=/tmp/sr-oci-agent
rm -rf "$DEST"
mkdir -p "$DEST"
rsync -aL "$SRC/" "$DEST/"
chmod -R u+w "$DEST"
DIGEST=$(jq -r '.manifests[0].digest' "$DEST/index.json")
REPO=registry.carverauto.dev/serviceradar/serviceradar-agent
crane push "$DEST" "$REPO@$DIGEST"
crane tag "$REPO@$DIGEST" "sha-<commit>"
rm -rf "$DEST"

Repeat for each rebuilt image. Capture every digest.

On a Linux host the push targets can be run directly, since the runfiles jq/crane are the right architecture there. Use remote_base for the build; the remote_push profile referenced by older runbooks no longer exists.

Copy unchanged images forward

crane copy \
  registry.carverauto.dev/serviceradar/<image>:<old-tag> \
  registry.carverauto.dev/serviceradar/<image>:sha-<new>

<old-tag> is whatever the cluster is running (v1.4.31, sha-<old>, etc.). If crane digest of old and new match, the copy is a retag and existing signatures (if any) still apply.

farm01 roll (no signing)

export KUBECONFIG="$HOME/.kube/farm01.yaml"
helm get values serviceradar -n serviceradar
helm upgrade serviceradar "$CHECKOUT/helm/serviceradar" \
  -n serviceradar \
  --reuse-values \
  --set image.digests.<service>=sha256:<digest> \
  --rollback-on-failure \
  --timeout 15m

Use global.imageTag=sha-<new> ONLY when every first-party image should move. For a change touching one or two services, pin those with image.digests.<service> (see "Move only the services you changed") -- farm01 already carries digest pins for core and webNg, so you are updating an existing pin, not introducing a new mechanism.

--reuse-values keeps farm01-only settings (MetalLB VIPs, Gateway API attach, Trivy sidecar, empty registryPullSecret, local-path storage). Do not replace the live values file unless the user wants those template/value changes applied.

Use the local chart when helm/ matches the live chart version (helm get metadata serviceradar -n serviceradar). If helm/ diverged and the user only asked for new images, keep the live chart and only change global.imageTag (OCI oci://registry.carverauto.dev/serviceradar/charts/serviceradar --version <live> plus --reuse-values --set global.imageTag=...).

Update ~/src/gitops/clusters/farm01/serviceradar/values.yaml global.imageTag so GitOps matches live. Do not commit or push the gitops repo unless the user asks.

Verify:

for d in serviceradar-web-ng serviceradar-core serviceradar-agent serviceradar-agent-gateway; do
  kubectl rollout status deploy/"$d" -n serviceradar --timeout=180s
done
kubectl get deploy -n serviceradar \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .spec.template.spec.containers[*]}{.image}{" "}{end}{"\n"}{end}'
kubectl get pods -n serviceradar

Done when key deployments show sha-<new> and pods are Ready. Report the tag, rebuilt vs copied images, digests, helm revision, and any non-Ready pods. Do not mention signing.

carverauto demo roll (sign, then Argo)

demo admission is Kyverno-enforced. Sign every rebuilt digest with OpenBao cosign-release before changing the Argo app. Copied-forward images whose digest is unchanged keep their existing signatures.

OpenBao on the control plane is HTTPS. A plaintext http://127.0.0.1:18200 health check returns Client sent an HTTP request to an HTTPS server.

kubectl port-forward -n openbao-system svc/openbao-active 18200:8200

OPENBAO_ADDR=https://127.0.0.1:18200
OPENBAO_K8S_ROLE=forgejo-signing-runner
sa_jwt="$(kubectl create token -n forgejo-actions forgejo-signing-runner)"
vault_token="$(curl -skS \
  -H 'Content-Type: application/json' \
  -d "{\"role\":\"${OPENBAO_K8S_ROLE}\",\"jwt\":\"${sa_jwt}\"}" \
  "${OPENBAO_ADDR}/v1/auth/kubernetes/login" | jq -er '.auth.client_token')"
export VAULT_ADDR="$OPENBAO_ADDR"
export VAULT_TOKEN="$vault_token"
export VAULT_SKIP_VERIFY=true
export COSIGN_KEY_REF=hashivault://cosign-release
export COSIGN_YES=true
export COSIGN_DOCKER_MEDIA_TYPES=1
export COSIGN_REFERRERS_MODE=legacy
export COSIGN_TLOG_UPLOAD=true

cosign sign --key "$COSIGN_KEY_REF" \
  registry.carverauto.dev/serviceradar/<image>@sha256:<digest>

Re-mint the Vault token on 403 permission denied. Tear down the port-forward when signing is done.