Release QA

What the run touches, and what it must not

The runner pins HOME, USERPROFILE, PI_LENS_HOME, PILENS_DATA_DIR, PI_LENS_INSTALL_LOG and npm_config_cache inside its own scratch root, and passes that environment to every child — pi, the MCP server, node, and npm. All six matter. PI_LENS_INSTALL_LOG is the one that is easy to miss: scripts/warm-loader-cache.mjs (which prepare runs on every pack) prefers THAT variable for its install log and falls back to PI_LENS_HOME/install.log (or ~/.pi-lens/install.log when neither is set). The runner's first six runs pinned the pi-lens home, passed no environment to npm, and put 41 warm_loader_cache records into the maintainer's real ~/.pi-lens/install.log (#2619 review F1).

The pack runs in a git archive HEAD export inside the scratch root, never in the live checkout, because npm pack fires our own prepack (scripts/strip-dev-deps-for-pack.mjs rewrites package.json AND package-lock.json, restored only by postpack, with no signal trap — an interrupted pack leaves the checkout stripped) and prepare (rebuilds dist/, downloads grammars over the network, reinstalls the git hooks). A dirty checkout is REFUSED rather than packed as its last commit, because a report whose "QA target" is not what the operator is looking at is the failure this runner exists to end.

Cost and cleanup. A run installs a production dependency tree twice (the export, then the scratch project) and builds dist/, so budget ~1.1 GB under /tmp and about a minute of wall clock. The scratch root is removed on exit unless --keep — but an INTERRUPTED run leaves it behind, so ls -d /tmp/pi-lens-release-qa-* after a Ctrl-C session, and remove what you find.

Remaining side effects on the checkout: none that persist. git archive and git status are reads. The runner writes release-qa-report.md and release-qa-evidence/ into --out (the cwd by default; both are gitignored at the repo root). Everything else lands in the scratch root, which is removed unless --keep. Verify rather than trust: git status --porcelain before and after a run must be identical, and wc -l ~/.pi-lens/install.log must not change.

Never run any release probe without those pins — an unpinned probe writes into the maintainer's real ~/.pi and ~/.pi-lens (#2506).

3b. Pre-bump dry roll (before opening the bump PR)

The bump PR changes three things the unit suite reads as data: the package version, the CHANGELOG.md release headings, and the .changelog/ fragment population (which the roll DELETES). Tests calibrated against the pre-roll tree go red only on the bump PR, after the gate has already said ship. The 4.1.4 bump (2026-09-07) hit two: config-deprecation-registry demanded deprecatedSince <= the announcing release (the window said 4.2.0; the Deprecated entries shipped in 4.1.4), and tracked-control-bytes's Markdown floor of 80 had been calibrated while 110 fragments existed (64 files remain).

So, in a scratch export, roll for the target version and run the tests that read release state:

S=$(mktemp -d) && git archive HEAD | tar -x -C "$S"
node scripts/changelog-release.mjs <version> --root-dir "$S"
(cd "$S" && npm version <version> --no-git-tag-version >/dev/null && ln -s "$OLDPWD/node_modules" node_modules && npm run build >/dev/null && \
  npx vitest run $(grep -rl "CHANGELOG.md\|\.changelog\|package.json" tests/ | grep -v "^tests/support\|fixtures" | tr '\n' ' '))

Read the release body too: (cd "$S" && node scripts/changelog-extract.mjs <version> --summary) must list only audience: user entries and end with the internal count.

Every red here is a release-only calibration: fix it IN the bump PR (a trailing commit after the bump+roll commit is fine) and name it in the PR body. Also read the rolled section for a "next version" literal that the fragments assumed (4.2.0 in a Deprecated since … line when the release is a patch) and correct it in code, docs and the rolled section together.

The bump PR title is chore(release): <version> (refs #<tracker>) — the PR-title lint requires the conventional prefix AND an issue ref, and a title missing either reds ci-verdict on an otherwise green workflow (the 4.2.0 bump was retitled after exactly that).

4. Report

The runner writes release-qa-report.md and release-qa-evidence/<row-id>.*. The report you hand back carries, in this order:

1. the verdict line — ship / ship-with-caveats / don't ship / blocked / inconclusive — with its exit code;
2. the coverage arithmetic, quoted verbatim from the runner;
3. every non-PASS row with its cause or reason;
4. the pi version(s) driven and the QA target (tree, or the published version);
5. the DIFF rows derived in step 2 and where each landed.

ship-with-caveats requires the caveats to be named in the release notes. A caveat nobody wrote down is a defect that shipped.

What this is not

• Not a substitute for CI. Unit tests and Lint are still the per-PR gate.
• Not a per-language tool sweep — scripts/smoke-tools.mjs does that nightly.
• Not a model-driven test. Every row is model-free by construction; a row that needs a real LLM turn cannot be a release gate.