Educates Upgrade Cluster Services

Educates Cluster-Service Chart Upgrade Protocol

The operator installs four upstream cluster services for EducatesClusterConfig (Managed mode) from Helm chart tarballs vendored at installer/operator/vendored-charts/: cert-manager, Contour, Kyverno, external-dns. The bytes are checked into the repo and embedded into the operator binary via //go:embed. This skill bumps one (or checks all) to a new upstream version.

installer/operator/vendored-charts/README.md is the canonical reference for this workflow — keep it and this skill in agreement.

If $ARGUMENTS names a service, upgrade only that one. With no argument, check all four for newer upstream versions and report before changing anything.

The same directory also holds the in-repo runtime subcharts (secrets-manager, lookup-service, session-manager, node-ca-injector, remote-access). Those are not in scope here — they are repackaged from source via make package-local-charts, not downloaded.

Where each piece of version state lives

A version is pinned in four canonical places that must stay in lockstep (a consistency test enforces it — see Verification):

1. installer/operator/Makefile → VENDORED_CHARTS list: name=version=url.
2. installer/operator/vendored-charts/SHA256SUMS: <sha256> <name>-<version>.tgz.
3. installer/operator/vendored-charts/embed.go: the //go:embed <name>-<version>.tgz directive, the <X>ChartVersion constant, and (Contour / external-dns / Kyverno) the <X>AppVersion constant. cert-manager has a single CertManagerVersion (its chart version equals its appVersion).
4. The tarball file itself on disk.

Beyond these four, some tests hardcode a version string rather than referencing the embed.go constant (an import cycle prevents the helm package's tests from importing vendoredcharts). The DirectoryConsistent test does not cover these, so they go stale silently and only surface as a CI test failure. The known one is installer/operator/internal/helm/load_test.go (TestLoadArchive_VendoredCertManager), which hardcodes the cert-manager version twice (the .tgz filename and the AppVersion assertion). Always finish a bump with the straggler grep below — do not rely on the four-place list alone.

Upstream sources and version semantics

Use WebFetch on the releases page to find the latest stable chart version (not a pre-release). The chart version and the bundled app version differ for Contour, external-dns, and Kyverno — read the new app version from the tarball's Chart.yaml (appVersion:) after downloading.

Kyverno is the awkward one: its chart version and binary version are on different numbering tracks and live in different places. The github.com/kyverno/kyverno/releases page lists the binary version (appVersion, e.g. v1.18.1); the chart version (<VER> in the URL above, e.g. 3.8.1) only exists in the Helm repo's index.yaml. They are not aligned and you cannot derive one from the other by inspection — you must resolve the chart version that ships a given Kyverno binary from the repo index:

# What chart version ships Kyverno binary v1.18.1? (stable, non-rc rows only)
curl -sSfL https://kyverno.github.io/kyverno/index.yaml \
  | yq -r '.entries.kyverno[]
      | select(.appVersion | test("rc")|not)
      | .version + "  =>  " + .appVersion' \
  | grep ' v1.18.1$'        # → 3.8.1  =>  v1.18.1

To browse the newest stable charts and the binary each one bundles (the first row is the latest stable chart), drop the grep:

curl -sSfL https://kyverno.github.io/kyverno/index.yaml \
  | yq -r '.entries.kyverno[]
      | select(.appVersion | test("rc")|not)
      | .version + "  =>  " + .appVersion' \
  | head

Equivalently with the Helm CLI (helm repo add kyverno https://kyverno.github.io/kyverno/ && helm repo update kyverno && helm search repo kyverno/kyverno --versions), whose CHART VERSION / APP VERSION columns are the same two fields. Either way, that chart version is what goes in <VER> for the download URL, the Makefile, SHA256SUMS, and the embed.go filename + KyvernoChartVersion; the matching appVersion is KyvernoAppVersion (still confirm it from the downloaded tarball's Chart.yaml per step 5).

Current versions: read them from the Makefile VENDORED_CHARTS line and the embed.go constants.

Update process (per chart)

Worked example: contour 0.6.0 → <VER>. Other charts are identical except cert-manager has no separate AppVersion constant.

1. Compute the new tarball's SHA256 from the upstream URL (the "I am deliberately trusting these bytes" step — make vendor-charts refuses to download anything whose hash isn't already recorded):

   curl -sSfL "https://github.com/projectcontour/helm-charts/releases/download/contour-<VER>/contour-<VER>.tgz" | shasum -a 256

2. installer/operator/Makefile — update the chart's VENDORED_CHARTS entry (version and URL).

3. SHA256SUMS — replace the chart's line with the new filename + hash.

4. make -C installer/operator vendor-charts — downloads, verifies against SHA256SUMS, and writes the new tarball into place. Then git rm the old tarball (nothing prunes it automatically).

5. embed.go — update the //go:embed filename, the <X>ChartVersion constant, and the <X>AppVersion constant. Read the new appVersion from the tarball:

   tar -xzOf installer/operator/vendored-charts/contour-<VER>.tgz contour/Chart.yaml | grep -E '^(version|appVersion):'

6. Grep the repo for stragglers — hardcoded references to the old version that live outside the four canonical places (notably tests). Run this with the OLD version string before you forget it, and update every hit:

   # e.g. old cert-manager version v1.20.2 → catches internal/helm/load_test.go
   git grep -n "<old-version>" -- installer/operator

   For cert-manager this always includes internal/helm/load_test.go (filename + AppVersion assertion). Skip the vendored-charts/ line for the tarball you just git rm'd.

7. Re-verify reconciler assumptions (the load-bearing step — see next section). The per-service reconciler gates readiness on specific workload names / webhook behaviour verified against a chart version.

8. installer/operator/vendored-charts/README.md — update the "Current contents" version table.

9. Kyverno only — also re-vendor the bundled policies. See "Kyverno special case" below.