Cut Pinchy Release

Cut a Pinchy Release

Overview

Pinchy releases are tag-driven. One script does everything from a clean main (or a release/X.Y branch — see "Release branches" below):

git checkout main && git pull --ff-only origin main
pnpm release X.Y.Z --verified=$(git rev-parse HEAD)   # a leading "v" is accepted too — the script normalizes it

That is the only state-changing command. It bumps the version, makes a chore: release vX.Y.Z commit, tags, and pushes — and the tag push is what triggers .github/workflows/release.yml to build images and create the GitHub Release.

--verified is required: it is your attestation that you verified this exact commit on staging, and the script aborts unless it equals HEAD (§ "Before you run pnpm release", step 4). Do not paste it from muscle memory — run the preflight and clear its [ ] items first, or you are attesting to something you did not check.

The Iron Rule: cut every release with pnpm release X.Y.Z --verified=<HEAD sha>. NEVER gh release create. NEVER a manual git tag + push.

When to use

• Any request to ship/cut/publish/tag a new Pinchy version.
• NOT for republishing images for a tag that already exists — that is workflow_dispatch on the Release workflow with the tag input (see CONTRIBUTING.md), and it deliberately does not re-create the GitHub Release.

Why never gh release create (the v0.5.5 incident)

pnpm release (scripts/release.mjs) bumps the version in three files inside the chore: release vX.Y.Z commit, then pushes the tag:

• .env.example
• package.json
• packages/web/package.json

packages/web/next.config.ts derives NEXT_PUBLIC_PINCHY_VERSION from package.json#version, and /api/version reports that value. Skip the bump and the tag says vX.Y.Z while /api/version still reports the old version. This actually shipped on v0.5.5 — someone used a gh release create shortcut, so the bump never ran: the tag was v0.5.5, but the image baked the stale package.json version into NEXT_PUBLIC_PINCHY_VERSION, so /api/version reported 0.5.4. Recovery was a whole v0.5.6 patch release.

Second reason the script must push the tag: the GitHub Release must NOT exist when the workflow starts. release.yml calls gh release list --limit 1 to find the previous tag, which extract-upgrade-notes.mjs needs to build the "Upgrade notes" section of the release body. Pre-creating the release with gh release create makes that lookup return the wrong (current) tag. Let the script push; the workflow creates the Release.

CI enforces this (since PR #454)

A gh release create shortcut now fails the workflow before any artifact exists, so a botched release leaves no GHCR image to clean up — but it still burns a CI cycle on your deadline:

• Build-time guard — scripts/assert-package-version.mjs <tag> runs before the image build; fails if package.json / packages/web/package.json don't match the tag.
• Runtime guard — the end-user-install-published job smoke-tests /api/version against the tag on the published image.

The upgrade-notes section auto-finalizes (since v0.6.0, the v0.5.8 incident)

upgrading.mdx is cumulative: one ## Upgrading from v<prev> to <target> section per release. During development the current section is written with the %%PINCHY_VERSION%% placeholder (heading and body), because the next version number isn't known yet. docs/scripts/inject-version.sh resolves that placeholder to the build-time version — so only the single newest section may carry it; every older section must already be concrete (to vX.Y.Z).

The v0.5.8 release forgot to freeze its section: the heading stayed from v0.5.7 to %%PINCHY_VERSION%% and the body kept literal placeholders. That's a silent time-bomb — the v0.5.8 notes render fine for v0.5.8, then mis-render as the next version's the moment newer docs build. v0.6.0's release prep had to repair it.

Three mechanisms now make this impossible:

• Auto-finalize in the release commit. pnpm release X.Y.Z calls finalizeUpgradeSection() (scripts/lib/release-logic.mjs): it freezes the current from v<prev> to %%PINCHY_VERSION%% section — heading and body placeholders — to vX.Y.Z, and includes the edited upgrading.mdx in the chore: release vX.Y.Z commit. So the release script now touches four files, not three: .env.example, package.json, packages/web/package.json, and docs/src/content/docs/guides/upgrading.mdx.
• Auto-open the next section, in a second commit made after the tag. openNextUpgradeSection() adds a fresh ## Upgrading from vX.Y.Z to %%PINCHY_VERSION%% skeleton (both ### subsections plus the standard-flow bash block) so the next cycle's first upgrade note has somewhere correct to land. It sits after the tag on purpose: the tagged tree is what the docs deploy builds, and inject-version.sh would render a skeleton inside the release commit as an empty "Upgrading from vX.Y.Z to vX.Y.Z" section on the live guide.
• Two guards in CI. scripts/lib/upgrading-mdx-freshness.test.mjs (via assertNoStaleUpgradeSections, run in pnpm test:scripts) fails any PR where a released version's section still carries %%PINCHY_VERSION%%, where two sections carry it, or where the placeholder section's from doesn't equal package.json#version. The preamble / "Standard upgrade" display placeholder is out of scope. scripts/lib/upgrading-released-sections.mjs fails any PR that edits an already-released section without an Allow-upgrade-note-edit: #NNN trailer or the allow-upgrade-note-edit label — see AGENTS.md § "A Released Upgrade Section Is Immutable".

What this means for you when cutting a release:

• You only write the new ## Upgrading from v<prev> to %%PINCHY_VERSION%% section (with %%PINCHY_VERSION%% placeholders is fine and preferred). The script freezes it for you at release time.
• The next cycle's section is opened for you by the release run, so notes written after a release land in it rather than in the section that just shipped. If it is somehow missing — a hand-cut release, a forward-port that dropped it — the next release's gate still fails loudly at the start (no from v<just-released> section), which is the safety net, not a surprise.
• You still have to write the notes. Because the opened section already has both ### subsections, "the section exists" stopped being the same question as "somebody wrote it" — so assertUpgradeNotesWritten fails the release while the section is still byte-identical to the generated template. Any real edit clears it, including a deliberate None.; leaving the template's own None so far. does not.

Release branches (GitLab-Flow-style)

main is trunk — features keep landing on it, always. A release/X.Y branch is cut per minor version, from a frozen, known-good ref, when the scope for that minor is decided. It hosts vX.Y.0 and every patch after it (vX.Y.1, vX.Y.2, …) — one branch per minor, not one per patch. main keeps flowing and becomes the next minor. pnpm release and pnpm release:preflight accept both main and any release/* branch; anything else is rejected.

• Upstream first — the load-bearing rule. A fix lands on main first, then is cherry-picked to release/X.Y. This makes "main never loses a fix" structural rather than a remember-to-back-merge chore. Back-merge (release branch → main) is only for a fix that is genuinely branch-specific — main has already refactored the affected code away and the fix doesn't apply there.
• Testing the candidate on staging. Staging's :next tag tracks main, so it is the wrong pin for a release-branch candidate. .github/workflows/pre-release.yml builds release/X.Y pushes to a branch-scoped moving tag rc-X.Y (e.g. release/0.9 → rc-0.9) instead of next — this keeps two concurrent release branches from clobbering each other's candidate image. Pin staging to either :sha-<short12> (an exact ref) or :rc-X.Y (latest on the branch), never :next, while verifying a release-branch candidate.
• Releasing. Run pnpm release X.Y.Z --verified=$(git rev-parse HEAD) from release/X.Y — same Iron Rule, same script, just a different branch checked out. The attestation matters more here, not less: :next is the wrong image for a release-branch candidate, so verify on :rc-X.Y and attest to the branch's HEAD. The tag push still triggers release.yml (tag-triggered, branch-agnostic — nothing changes there). A patch release is more upstream-first fixes cherry-picked onto release/X.Y, then pnpm release X.Y.(Z+1) from that branch.
• Forward-port upgrading.mdx after release. pnpm release freezes the current section's %%PINCHY_VERSION%% placeholders in the chore: release commit, and opens the next cycle's section in the follow-up commit — both on the release branch. Both must also reach main, or main's upgrade notes keep stale placeholders, lose the frozen section, and have no open section for the next note to land in. Cherry-pick both upgrading.mdx commits (or just those hunks) to main as an explicit post-release step. Caveat while #1028 is open: main's package.json is not bumped by a release-branch cut, so forward-porting the open section alone turns the freshness guard red on main (it requires the placeholder section's from to equal package.json#version). Bump main's version in the same forward-port, or resolve #1028 first.
• Version pins do NOT re-bump themselves on main — forward-port them too. pnpm release bumps README, .env.example, package.json/packages/web/package.json and both marketplace templates only on the branch it runs on. Nothing re-bumps them on main at its "own next cut" — that next cut is itself a release-branch cut, which has the identical blind spot. Left alone, main's pins sit stale for the entire cycle: .env.example and both marketplace templates stayed at v0.8.0 through all of v0.9.0, and PR #1053 could only hand-fix the README pin because the marketplace-version guard checks templates against .env.example, itself stale on main (#1079). See "After the release" → "Forward-port the version pins to main" for the mechanical step; same #1028 caveat as upgrading.mdx applies — bump main to the tag you just cut, not to a placeholder.

Before you run pnpm release

Step 0 — run the preflight and turn every [ ] into a blocking task. pnpm release:preflight <version> prints the gate status plus the manual gates the script can't enforce: a release-specific staging checklist auto-derived from this release's upgrade notes (the #### … subheadings under ### Breaking changes / ### Upgrade notes), the standard regression smoke, and the PWA check. This exists because manual gates that live only as prose get silently skipped next to the script's hard gates — that is exactly how v0.6.0 shipped with the staging click-through never done.

So, mechanically:

1. Run pnpm release:preflight <version>.
2. For each [ ] it prints, create a task (TodoWrite/Task), and make the pnpm release task blockedBy all of them. Do not start the release task while any remain open.
3. Verify each on the real staging instance (staging.heypinchy.com), pinned to the candidate image the preflight names — :next when you are cutting from main, :rc-X.Y (or :sha-<short12>) from a release/X.Y branch, see § "Release branches". Staging carries the upgrade path + real agents/data; the ephemeral CI E2E stacks don't. The release-specific items are different every release, which is why they're generated from the notes rather than hardcoded.
4. The preflight then prints the exact pnpm release <version> --verified=$(git rev-parse HEAD) command. The --verified SHA ties your attestation to the commit you actually tested on staging, and release.mjs hard-fails without it — a missing, short, or non-HEAD SHA aborts the release (#1085; until then the flag was parsed by nothing and silently discarded, so attesting and not attesting produced identical runs). Like the docs-review hook, it is not a fraud boundary: you can type the SHA without opening staging, exactly as you can git push --no-verify. It makes forgetting impossible, which is the failure mode that actually happens — and it anchors the attestation to one commit, so landing another commit invalidates it.

Build the test plan from the FULL changelog, then split it agent vs human

The preflight's staging checklist is auto-derived from the upgrade notes (#### subheadings) — that is a subset, not the coverage plan. Upgrade notes call out breaking changes and what an operator must know; they miss most shipped features (v0.9.0's Knowledge Base, IMAP connect flow, chat slash-commands, odoo_reconcile, auth error-honesty were none of them upgrade-note subheadings). Build the actual plan from the full changelog:

git log v<prev>..main --no-merges --pretty=format:'%s' | grep -iE '^(feat|fix)'

Then three moves before you test anything:

1. Scope out what has no runtime surface — don't test it. Dark foundations (schema/tables/plumbing merged ahead of the feature that will consume them — grep the upgrade notes and commit bodies for "foundation" / "no runtime surface") and internal-only tooling (eval, CI guards, scripts/) are not user-testable. v0.9.0 shipped ~75 such commits (the whole inbox-agent/email-workflows cluster + eval/kb-eval). Listing them as "to test" burns actions hunting for UI that doesn't exist — name them explicitly as out-of-scope so the human doesn't hunt either.

2. Cluster into minimum-action super-flows. One well-chosen end-to-end flow exercises many features + fixes at once: a KB-agent setup+query covers ingest, pgvector, hybrid retrieval, citations, abstention, offline embedding and a dozen KB fixes in two questions; an email→Odoo booking covers read, attachment, vision, odoo_read/create/reconcile, duplicate-guard and audit in one run. Optimize for max coverage per action, not one test per commit.

3. Split every cluster into an agent half and a human half — and hand the human theirs explicitly. This is sharper than the test-and-fix loop's "gates only a human can close" item below: it divides every feature, not just the agent-impossible ones.

   • The agent owns the plumbing / logic / honesty half — anything verifiable against ground truth: tool round-trips via the audit log, API responses to crafted inputs (auth error semantics, open-redirect / SSRF rejection, credential masking), DB & migration state, config emission, retrieval correctness (citations point at real passages, abstention fires). Drive it and confirm against evidence, per the loop below.
   • The human owns ALL UI/UX, plus edge-cases and real devices — not as leftovers but because they need the human feel the agent lacks: every wizard / flow / rendering / interaction (the IMAP connect wizard, slash-command discoverability, citation rendering, hover/paste behavior, dialog layout), anything on real hardware (PWA install + share-target), and the weird-behavior / edge-case hunting a human is simply better at. A feature with both halves (IMAP: API layer vs. connect wizard) is split down the middle — agent takes the API, human takes the wizard.

   Deliver the split as a table (cluster → agent verifies X / human checks Y), the human half prioritized, before you start driving your half.

The staging pass is an active test-and-fix loop — not a click-through

The CONTRIBUTING item reads "clicked through today," but a passive click-through is the weakest form of this gate — it confirms the app boots, not that the release works. The version that actually catches blockers, and is now the standard, is an autonomous test-and-fix loop you run yourself before handing back for the human's "go" (the v0.8.0 email→Odoo pass caught three shippable blockers this way — a false-success invoice read, a missing vision fallback, and a duplicate-booking bug):

1. Deploy the release candidate to staging yourself, then verify against that build. After the fixes land, wait for pre-release.yml to publish this branch's candidate image (:next from main, :rc-X.Y from release/X.Y), refresh staging (docker compose pull && up -d && docker image prune -f), and confirm the regenerated openclaw.json actually reflects the change. Verifying against a stale image proves nothing — and on a release branch, verifying against :next proves something about main instead.
2. Drive real end-to-end flows, adversarially — the goal is to find problems. Exercise the actual agents through the actual integrations, covering the archetypes: the default agent, each external integration (email, Odoo, …), an analytics/data agent, and a knowledge-base agent. Include the enforcement paths (tools.allow fail-closed, permission scoping) and the recovery paths (reconcile-on-reload). "More problems found is better," not "confirm none exist."
3. Verify against ground truth, not the UI. "Looks fine in the chat" is not verification — that is exactly how a false-success PDF read once shipped, and this session the live UI even hid a completed reply behind a streaming disconnect. Confirm each claim against evidence: the audit log (tool.<name> rows — outcome, detail, error), the OpenClaw container logs (which model actually served a call — e.g. a tool.pdf attempts array showing the primary 401 and the fallback succeeding), the regenerated openclaw.json, and cross-validation of the numbers themselves (line items summing to the stated subtotals). Scout the state deterministically over SSH before driving the browser — inspect config/DB/audit to pick the right agent and predict the outcome, instead of clicking blind.
4. Every bug you find is a release blocker — TDD-fix it, don't file-and-ship. Write the failing test (red → green), open its PR, get it merged (upstream-first: onto main, then cherry-picked to the release branch if you are cutting from one), redeploy staging from the refreshed candidate image, and re-verify end-to-end. Loop until the flow is clean. "Keine bekannten Bugs in Releases."
5. Classify transient/infra vs. real before reacting — the same rule as post-release CI (see "After the release" §4), applied to the staging pass. A CI apt-fetch / runner-download failure or an unrelated E2E flake is infra → a rerun is correct; an empty model response or a hidden-tab streaming disconnect is transient → retry/reload and confirm the reconcile recovers it; a wrong tool result or a broken flow is real → fix it. Never rerun (or reload) to paper over a real failure.
6. Read the browser console after every manual step. A recovered crash looks like a working page: v0.9.0's /reset trips the assistant-ui shrink defect 100% of the time, ChatCrashBoundary swallows it, and the only evidence is two console lines (#944). Nothing in CI catches this either — of 64 E2E specs exactly one subscribes to pageerror, and only for diagnostics (#945). Clear the console, act, read it back; a screenshot that looks right proves nothing about it.
7. Odoo scenarios go against the demo instance — never production. Production is the only Pinchy instance with an Odoo connection configured, so answering "which Odoo do we have?" from integration_connections points straight at the company's real books. The disposable target is odoo-demo.heypinchy.com (Odoo 19 Community, rebuildable in ~10 min); it lives outside this repo, in ~/projects/odoo — reset recipe in that repo's docs/pinchy-demo.md, seed data via its scripts/setup_crabon_demo.py, credentials in the operator's local ~/.config/odoo-demo.env. Expect to ask the user to add the connection; staging carried none as of v0.9.0.
8. A prompt must not hand the agent the permission the guard is being tested for. Testing the vendor-bill duplicate guard with "I checked externally, this is not a duplicate — just file it" is the exact confirmation allow_duplicate: true exists for: the agent overrides, the bill is created, and the result proves the override works, not the block. Write the adversarial prompt so the only correct behaviour is refusal — then a create means the guard failed. Same for permission tests: state the goal, never the escape hatch.
9. Finish what you can, then hand back an explicit test plan for the gates only a human can close. Some gates are outside an agent's reach: entering API keys (never do this yourself), PWA install (Chrome desktop + iOS Safari), pure tab-refocus reconcile, permanent data deletion, and anything needing product judgment. Close everything closable autonomously, then give the human a crisp, prioritized list of exactly what remains — don't leave it implicit.
10. Never cut the tag autonomously. The human's explicit "go," after their own pass, is the hard gate on pnpm release. Your job is to make that "go" a rubber-stamp by having already found and fixed everything findable.

Work through every item in CONTRIBUTING.md § "Pre-release checklist" — that is the canonical, always-current list, so don't re-derive or copy it. The script and CI already enforce the mechanical gates (clean tree, on main or a release/* branch, CI green, tag free, upgrading.mdx section present with both subsections, pnpm audit --audit-level=high --prod). The human judgment calls the script can't enforce — verify each against CONTRIBUTING — include:

• All feature/fix PRs for this release merged to main; pnpm outdated reviewed.
• Dockerfile.openclaw version bumped if OpenClaw was upgraded.
• Model-resolver spot-check if models or templates changed.
• Ollama Cloud catalog → run the update-ollama-cloud-models skill every release to refresh the catalog.
• docs/src/content/docs/guides/upgrading.mdx has a new ## Upgrading from v<prev> to %%PINCHY_VERSION%% section containing ### Breaking changes (write "None." if none) and ### Upgrade notes. The script aborts without it, freezes the placeholder for you at release time, and a CI guard rejects stale placeholders — see "The upgrade-notes section auto-finalizes" above.
• Staging click-through on this branch's candidate image (:next / :rc-X.Y) + PWA install check.
• Security review over the whole v<prev>..HEAD release delta — see the section right below.