Release

Breeze Release

This skill drives the communication and rollout pass for a release: the GitHub Release body (especially the self-hoster section), the marketing release notes, the technical docs, and the Discord announcement — plus the post-release deploy/promote steps the tag alone doesn't handle.

It assumes the code is already merged to main and green. Tagging is the trigger; this skill is about everything that surrounds it.

Infra specifics live in internal/ops/release-infra.md (gitignored; read it from the MAIN checkout at ~/breeze/internal/ops/release-infra.md when working in a worktree). This skill refers to those values as $EU_IP, $US_IP, $EU_FW, $US_FW, $JUMP. Read that file FIRST when you reach Step 6 — do not guess or reconstruct the values.

How a release actually fires

.github/workflows/release.yml triggers on push: tags: v*. The tag's contents are whatever commit it points at, and the version is the tag minus the v. There is no VERSION file — cutting a release is pushing the tag.

• CI builds + pushes GHCR images (api/web/portal/binaries) and creates the GitHub Release with generate_release_notes: true — i.e. an auto-generated commit list ("What's Changed" + Full Changelog). It does not write the curated Summary / Added / Improved / Fixed / Security / Self-Hosting sections. You add those afterward via gh release edit.
• Versions are bare semver everywhere except git tags. The git tag is v0.84.0; the agent version string, BREEZE_VERSION, and ldflags are 0.84.0.

Step 0 — Know the blast radius BEFORE tagging

A tag off main HEAD ships all accumulated merged-but-unreleased work, not just "the latest thing", and auto-runs every pending migration on prod via autoMigrate on API boot. This has surprised us (the 0.83.1→0.83.2 "hotfix" carried 327 files + 10 migrations). So always look first:

PREV=v0.83.3            # the last released tag
NEW=v0.84.0             # the tag you're about to cut
git -C ~/breeze fetch --tags
git -C ~/breeze diff $PREV..HEAD --stat | tail -1          # total churn
git -C ~/breeze diff $PREV..HEAD --name-only -- apps/api/migrations/   # migrations that will auto-run on prod
git -C ~/breeze log --oneline $PREV..HEAD                  # everything that will ship

• Full release of everything on main (the normal case): tag off main HEAD.
• True surgical hotfix: branch + tag off $PREV, cherry-pick only the fix. Do not tag off main HEAD if you don't intend to ship all of main.

If migrations are listed, you're committing prod to those migrations the moment the new API boots. Read them; confirm there are no large-table rewrites/backfills before you proceed (call those out in the upgrade notes if present).

Smoke-test privileged migrations as a NON-SUPERUSER role (learned the hard way, v0.95.0→v0.95.1). CI and the local docker-compose test DB both migrate as the Postgres superuser (POSTGRES_USER=breeze), which silently passes statements a least-privilege role would be denied. Prod is DO-managed doadmin — NOT a superuser, so a migration that only a superuser can run passes every test and then crash-loops the API on deploy (this is exactly what took EU down on v0.95.0: ALTER FUNCTION ... OWNER TO <role> requires the new owner to hold CREATE on the schema; a superuser bypasses that check, doadmin does not → permission denied for schema public). So: any migration in the range that does CREATE ROLE, ALTER ... OWNER TO, GRANT/REVOKE on a schema, CREATE EXTENSION, or a SECURITY DEFINER function — run it against a non-superuser role before tagging. Two ways: pipe the migration to a real managed DB (or a droplet's doadmin) inside BEGIN; ...; ROLLBACK; with ON_ERROR_STOP=1, or SET ROLE <nosuperuser-createrole-role> on a local DB that already has the full schema. If it only works as a superuser, fix it forward (grant the owning role the privilege for the one statement, then revoke) before it reaches prod.

Step 0.2 — Pre-cut gates: the sweep doc and the lab rigs

Every release ships the agent. release.yml builds and registers the full agent family on every tag, and the fleet promote is a standing rollout step — so "needs an agent release" is never a blocker or a special to-do, it is just what a release is. What an agent-side change does need is its real-host check done before the tag.

1. Read the newest docs/testing/release-sweeps/*.md and its "Before the release cut" list, plus docs/release-notes/next-release-draft.md → "Release to-do". Any sweep fix PR named there merges before the tag.
2. Rows a sweep left BLOCKED / PARTIAL for want of a live agent are run on the SSH lab rigs before tagging — not deferred to after the cut, and not left for a reporter to find. Use the non-prod rigs (the second Windows Server VM and the KIT lab-ubuntu-src VM are lab-enrolled against a local stack; the first Windows VM and the Ubuntu KVM rig are prod-enrolled — leave those alone unless the row is about prod upgrade behaviour). Recipe: pnpm wt-stack up at the commit to be tagged → cross-compile the agent family from that commit (GOOS=… CGO_ENABLED=0 go build -ldflags "-X main.version=<v>") → back up the rig's agent config → re-point server_url at the Mac's Tailscale/LAN address and re-enroll → run the rows → restore the config. Record results in the sweep doc.
3. A row that still cannot be run (no hardware for it) is carried into the release body's known-gaps note with the exact prerequisite, by name.

Step 0.4 — Author the in-app "What's New" entry (BEFORE tagging)

The post-upgrade splash (WhatsNewSplash, shown on dashboard load) renders an entry from WHATS_NEW_ENTRIES in apps/web/src/lib/whatsNew.ts — a list bundled into the web image at build time. No entry for the release = no splash after the upgrade, regardless of what the server version says. This got skipped for 0.106.0 and 0.107.x, so the splash never fired in prod even though the feature shipped in 0.105.0.

Ordering is the whole point: this is the one comms artifact that is pre-tag. The entry must be merged to main before the tag is pushed, or the build won't contain it. Everything else (release body, marketing notes, docs, Discord) happens after.

Add a new entry at the top of WHATS_NEW_ENTRIES (newest-first):

{
  version: '0.X.Y',   // bare semver — must equal the tag minus the 'v'
  date: 'YYYY-MM-DD', // release date
  title: '<one-line theme — same theme as the release body>',
  highlights: [
    // 2–5 short, user-facing bullets. Technician/MSP language, not commit
    // subjects. Same audience as the marketing notes, tighter than both.
  ],
  learnMoreUrl: 'https://breezermm.com/release-notes',
},

• Source the bullets from the same $PREV..HEAD sweep as Step 1 — the 2–5 things a technician would actually notice in the UI. Skip infra, deps, internal refactors. A mostly-internal release can honestly say "hardening and reliability" — or skip the entry entirely (skipping is a valid choice, just a deliberate one).
• Sanity-check: cd apps/web && npx vitest run src/lib/whatsNewState.test.ts src/components/whatsNew/WhatsNewSplash.test.tsx
• Already tagged without an entry? Do NOT backdate. Users' lastSeenVersion floor in localStorage is already baselined at the version they're running, so an entry versioned at or below the shipped release will never auto-show for them. Version the entry as the next release and fold the missed highlights into it.

Step 0.5 — After tagging: WAIT for the build, confirm ALL artifacts exist

Pushing the tag kicks off release.yml, which is a long, multi-job build (~20–40 min): api/web/portal GHCR images, agent binaries for every platform, the signed Windows MSI, the notarized macOS agent/viewer/helper, watchdog + user-helper, and the signed release-artifact-manifest.json (+ .ed25519/.minisig) and checksums.txt. The GitHub Release is created by the final job — you cannot gh release edit the body until then, and you must not roll out until every artifact is present.

A partial or half-failed build can still publish a Release with missing binaries — installers and agent auto-update would then 404 in prod. So gate explicitly:

# Block until the run finishes (re-invokes you on completion):
RUN=$(gh run list --repo lanternops/breeze --workflow Release --branch $NEW --limit 1 -q '.[0].databaseId' --json databaseId)
gh run watch "$RUN" --repo lanternops/breeze --exit-status

# Then confirm success AND a full asset set — compare the count to the previous release:
gh release view $NEW  --repo lanternops/breeze --json assets -q '.assets | length'
gh release view $PREV --repo lanternops/breeze --json assets -q '.assets | length'   # expected baseline (~40)
gh release view $NEW  --repo lanternops/breeze --json assets -q '.assets[].name' | sort

Before proceeding, verify the new release has the same asset families as the prior one — the macOS .pkg/.dmg/.app.tar.gz, the viewer/helper installers, the -unsigned BYO-signing inputs, and release-artifact-manifest.json + .ed25519 + .minisig + checksums.txt. A short asset list = signing or notarization failed midway.

• If the run failed, re-run just the failed jobs (gh run rerun "$RUN" --failed) and re-watch. Don't hand-publish a release off a red build, and don't roll out.
• gh run watch can exit non-zero while the run is still in progress (seen 2026-07-02) — before treating a watch failure as a build failure, check gh run view $RUN --json status: in_progress means keep waiting (a status-polling loop is more reliable than gh run watch).
• You can draft the release body and the Discord message (Steps 1–5) while the build runs — just don't publish the body or deploy until this gate is green.

Step 0.6 — Abuse-asset gate + hosted signing lane (fork-era releases, ≥ v0.105.0)

Since the installer-trust fork, the public release must contain nothing abuse-usable, and the signed hosted agent build comes from a separate private lane. Both are release-blocking; the tag lands as a draft (RELEASE_DRAFT_FIRST=true) and stays one until the hosted cutover completes.

1. Verify the public draft's assets — bytes and manifest, not job color:

• Every agent-family Windows exe and the MSI must be unsigned (edition=self-host in release-artifact-manifest.json; zero edition=hosted entries). Check the actual bytes: PE security-directory size 0 on the exe, no DigitalSignature stream in the MSI, and the -unsigned asset byte-identical (cmp) to its non-suffixed twin.
• Only Viewer/Helper installers are Authenticode-signed (currently an OV cert — SmartScreen reputation warnings after a cert change are expected, not a failure).
• Spot-check checksums.txt against downloaded assets.

2. Dispatch the private hosted signing lane. Repo name, dispatch command, edition choice (gap vs strict), and token gotchas live in internal/ops/release-infra.md ("Hosted signing lane") — read it, don't reconstruct. Key facts:

• The lane verifies the signed official manifest + tag↔sourceCommit binding before building, rebuilds the agent family as the hosted edition (control-plane allowlist baked in), signs with the production provider, and publishes only a private GHCR image — never public assets.
• It runs against the draft release, so its verify job needs the cross-repo PAT described in the infra doc (draft releases are invisible to read-only credentials — the PAT needs contents read and write on the public repo).
• Signing consumes production signing quota: get Todd's explicit go for the dispatch, per run.
• Order (fork runbook): public build green → asset gate above → hosted lane green → droplet flip → only then publish the draft → fleet promote. Each arrow is a separate explicit go from Todd — approval for one step never carries to the next.

Step 1 — Source the content (PRs + git only)

Read docs/release-notes/next-release-draft.md FIRST. It is the running scratch list where operator-visible details get written down at the moment they're introduced — new env vars, new log lines, new metrics, behaviour changes — rather than being reconstructed from a commit subject weeks later. Fold each entry into the body below (most of it belongs in Self-Hosting / Upgrade Notes), then clear the file in the same PR that publishes the release, so a stale entry can't be shipped twice. It is a supplement to the git history, not a replacement: still do the $PREV..$NEW sweep, because anything nobody thought to write down only exists there.

Release content otherwise comes from merged PRs and git history only — never blog posts, design docs, or product docs (those describe planned, not shipped, work).

git -C ~/breeze log --oneline $PREV..$NEW
gh pr view <N> --repo lanternops/breeze --json title,body
git -C ~/breeze log --format="%B" -1 <SHA>   # expand a squash-merge body

Skip from user-facing notes: chore(deps):, chore:, ci:/fix(ci):, internal refactor:, internal docs:/test:. Keep anything a self-hoster or operator would notice.

Step 2 — Write the GitHub Release body

The GitHub Release is the operator/self-hoster source of truth. Model the body on the most recent release for structure and tone:

gh release view $PREV --repo lanternops/breeze   # reference template

Structure (omit empty sections):

Breeze RMM **vX.Y.Z** — one-line theme of the release.

## Summary
- **Area** — what shipped, in operator language (#PR).

## Added
- **Feature** — what it does for the user (#PR)

## Improved
- ... (#PR)

## Fixed
- ... (#PR)

## Security
- Hardening summary; "Self-hosters are encouraged to upgrade." (#PRs)

## Self-Hosting / Upgrade Notes
<the most important section — see below>

The Self-Hosting / Upgrade Notes section (do not skip)

Self-hosters run their own droplets and read this section to decide whether an upgrade is safe and what they must touch. Always answer these four questions explicitly, even when the answer is "nothing required" — silence reads as "I don't know":

1. Upgrade command. Give both paths, because the right one depends on how the install pins its images (grep '^BREEZE_API_IMAGE_REF=' .env):

   • Digest-pinned (value ends in @sha256:… — every guided-setup.sh install for v0.112.0+): bash guided-setup.sh --upgrade 0.X.Y (fetch the current script from main first). It verifies the signed image inventory and rewrites BREEZE_VERSION and all four BREEZE_*_IMAGE_REF digests together, then pulls and restarts.
   • Tag-form (value ends in :${BREEZE_VERSION}): bump BREEZE_VERSION, then docker compose pull api web portal && docker compose up -d (and pnpm install if they build from source).

   Link the Upgrade Guide. Never write "set BREEZE_VERSION, then pull" as the only upgrade line. On a digest-pinned install that re-pulls the images already running, so nothing upgrades. Until v0.118.2, /health also reported the edited version because Compose copied BREEZE_VERSION into APP_VERSION (#7024), so operators believed they had upgraded. Only the web footer showed the real version. Every release note from v0.112.0 to v0.118.2 gave that line, and self-hosters were stranded on 0.115→0.116 and 0.117→0.118.2. Keep portal in the tag-form pull list: it is a separate container and was left behind for 11 days (stuck on 0.94.0 while api/web ran 0.98.1) when the line pulled only api web.

2. Database / migrations. State the count and that they're idempotent and auto-apply on boot via autoMigrate (unless AUTO_MIGRATE=false). Explicitly flag any large-table rewrite or backfill — that's the difference between a 2-second upgrade and a stalled boot. If nothing: "Database — nothing required."

3. New required environment variables. Anything the config validator now refuses to boot without (e.g. past examples: RELEASE_ARTIFACT_MANIFEST_PUBLIC_KEYS, IS_HOSTED). A new required env var must also be mapped in the api/web environment: block of their compose, not just .env — say so. If none: "No new required environment variables."

4. Behavior changes & feature flags. Anything whose default changed (call out grandfathering of existing orgs), and any feature gated behind a flag (e.g. PUBLIC_ENABLE_EDR_INTEGRATIONS) — name the flag and its default.

End with the compare link: **Full Changelog:** https://github.com/LanternOps/breeze/compare/$PREV...$NEW

Breaking changes

If there are any, set the tone up front (a bolded ⚠️ Breaking changes callout near the top and its own bullet in upgrade notes). Be concrete: what breaks, who is affected, what action they must take, and whether there's a grandfather/opt-out. If there are none, say "No breaking changes." somewhere in the upgrade notes — self-hosters look for that reassurance. Most Breeze releases are non-breaking; behavior-default changes (like the PAM UAC opt-in flip) are the usual "soft" breaking case and must be flagged.

Publishing the body

CI already created the release with the auto-generated "What's Changed". Preserve that — append your curated sections above it rather than clobbering:

gh release view $NEW --repo lanternops/breeze --json body -q .body > /tmp/auto-notes.md
# Write curated sections to /tmp/curated.md, then:
cat /tmp/curated.md /tmp/auto-notes.md > /tmp/release-body.md
gh release edit $NEW --repo lanternops/breeze --notes-file /tmp/release-body.md

Step 3 — Marketing release notes

Invoke /update-breeze-release-notes. That skill owns the breezermm.com content schema, the dev→user language translation, and the categories grouping for major releases. Don't duplicate its rules here — hand it the version range and let it work. (Its notes are for MSP owners/technicians; tone is lighter and more marketing than the GitHub body.) Remember the site only publishes via its deploy script (npm run deploy:notify) — a commit alone changes nothing live.

Step 4 — Technical docs

Invoke /update-breeze-docs in diff mode for the version range (e.g. "update docs since $PREV"). It maps changed code → affected docs pages, filters by audience, and bumps its review tracker. Required env vars, new feature workflows, and behavior changes from this release should land in the Starlight docs.

Step 5 — Discord announcement

Draft a short, friendly announcement for the community Discord. Keep it scannable — highlights, not the full changelog — and always include the self-hoster upgrade line and any breaking/behavior change. Show it to Todd before posting; don't post it yourself unless asked.

Template:

🚀 **Breeze vX.Y.Z is out**

<one-line theme>

**Highlights**
• <feature 1>
• <feature 2>
• <feature 3>
• <security note, if any>

**Self-hosting:** <one line — "`guided-setup.sh --upgrade X.Y.Z` (or bump `BREEZE_VERSION` + pull on tag-form installs), N idempotent migrations, no new env vars" OR the specific action needed>
<⚠️ one line if there's a breaking/behavior-default change>

📝 Full notes: https://github.com/LanternOps/breeze/releases/tag/vX.Y.Z

Tone: operator-to-operator, not corporate. Lead with what people can do now. If a release is mostly internal/security, say so plainly ("mostly hardening + reliability — recommended upgrade").

Step 6 — Roll it out

The tag built images and registered binaries; it does not deploy prod or update the agent fleet on its own. The intent is for cutting a release to flow straight into rollout — but always confirm with Todd before touching prod, then proceed. Three parts, per region: back up the DB → deploy the droplet → (later) promote the agent fleet.

Read internal/ops/release-infra.md now (main checkout: ~/breeze/internal/ops/release-infra.md) for the real droplet IPs ($EU_IP/$US_IP), firewall IDs ($EU_FW/$US_FW), and jump-host name ($JUMP) used below.

Prerequisite — get SSH access to the droplets

Two access paths, in order of preference:

1. Jump host ssh -J $JUMP root@<ip> — works when the bastion is up and you can complete its browser re-auth; the bastion's SSH is posture-gated with an interactive check, so headless sessions usually cannot use it — if it prompts with a login URL or hangs, don't burn time, fall back to (2).
2. Direct SSH + a temporary DO firewall exception. Each droplet sits behind a DO cloud firewall that only allows port 22 from an allowlisted IP. Add your current IP, deploy, then remove it afterward to restore posture:

MYIP=$(curl -s4 ifconfig.me)
for fw in $EU_FW $US_FW; do
  doctl compute firewall add-rules $fw --inbound-rules "protocol:tcp,ports:22,address:$MYIP/32"
done
# ... do the rollout via  ssh root@$EU_IP  /  ssh root@$US_IP ...
# then restore:
for fw in $EU_FW $US_FW; do
  doctl compute firewall remove-rules $fw --inbound-rules "protocol:tcp,ports:22,address:$MYIP/32"
done

add-rules/remove-rules are additive/surgical — they won't touch the existing allowlisted IP. Remove your exception once both regions are healthy and promoted (an RMM's prod SSH shouldn't stay open to a residential IP). If the fleet-promote step is deferred pending Todd's go-ahead, remove the exception anyway and re-add it when he approves — it's two commands.