Corpus Remeasure

Re-measuring a package in the build-jail corpus

The loop is: edit → push → dispatch a debug run → watch → read the CELL LOGS. Each step has a trap that has already cost a run.

Corpus repo: nubjs/build-jail-corpus (public). Local clone: ~/nub-corpus-staging. nub: branch sandbox/integration at ~/.cache/nub/worktrees/integ.

1. Push the harness change — the queue is the trap

⛔ ALWAYS git commit -- <explicit paths>. A bare git commit sweeps in whatever is staged, and queue.ndjson is almost always staged because reading the corpus stages it. A commit carrying a stale queue releases rows other runners are actively measuring. Measured: one such commit carried 194 lines of stale queue.

cd ~/nub-corpus-staging
git commit -q -m "…" -- harness/search.mjs          # explicit paths, always
git show --stat --oneline HEAD | head -5             # one second; catches exactly this

The push races ~12 concurrent runners, so it needs a retry loop — and the cleanup before each attempt is what makes it work:

for a in 1 2 3 4 5; do
  git restore --staged . 2>/dev/null                 # unstage (checkout STAGES; restore is what unstages)
  git checkout -- records queue.ndjson 2>/dev/null   # drop working-copy edits — origin owns these
  git clean -fdq -- records 2>/dev/null              # reading the corpus leaves UNTRACKED records behind
  git fetch -q origin main
  git rebase -q origin/main >/dev/null 2>&1 || { git rebase --abort 2>/dev/null; sleep 3; continue; }
  git push origin HEAD:main >/dev/null 2>&1 && { echo "PUSHED"; break; }
  sleep 3
done
git fetch -q origin main
git merge-base --is-ancestor HEAD origin/main && echo "✓ LANDED" || echo "⛔ NOT LANDED"

⛔ Verify it landed. Never infer it from the push command's apparent success — read merge-base --is-ancestor, because a rejected push inside a loop is easy to miss.

Reading the corpus for analysis leaves two kinds of debris, both of which break the next rebase: git checkout origin/main -- records STAGES those files, and it leaves records your HEAD lacks as UNTRACKED. Before cleaning untracked records, confirm they exist at origin (git cat-file -e origin/main:<path>) so a clean can never destroy an unpushed measurement.

2. Dispatch the re-measure

The workflow has a packages debug input. When set, the runner SKIPS the queue entirely — no claim, no completion, no commit of results — and measures exactly those specs with --force, publishing only the ARTIFACT. That is what makes it safe to run against a live corpus.

gh workflow run corpus-queue-runner.yml -R nubjs/build-jail-corpus \
  -f os=macos \                       # linux | macos | windows
  -f nub_sha=<40-char sha> \
  -f chain=false \                    # ⛔ or it self-dispatches a whole chain
  -f slice=2 \
  -f packages="husky@3.1.0 detox@20.9.1"

• ⛔ nub_sha must be PUSHED. CI clones nubjs/nub anonymously. Check with git ls-remote https://github.com/nubjs/nub sandbox/integration before dispatching, or the run dies at clone.
• Pick the sha deliberately. Using the sha the fleet already built gets a binary-cache hit (~0 min instead of a ~26 min cold Windows build) AND makes the result comparable with corpus records. Use a NEW sha only when testing a nub change.
• Platform speed differs enormously. macOS/Linux ≈ 1 min/package; Windows ≈ 9 min/package average and up to ~40 for a package that walks the full 55-cell ladder, plus ~26 min for a cold build. Prototype on macOS or Linux whenever the question is not Windows-specific.
• ⛔ In run-batch.sh, --force must come BEFORE --file — it is parsed only as the first argument after <nub>. The workflow already does this. A run with the order wrong "succeeded" in 17 seconds having measured nothing.

3. Watch it — and gate the watcher, or it will lie to you

⛔ NEVER dispatch CI without a bounded background watcher. A stall is silent. And the watcher itself must gate, because gh run view --log returns an EMPTY string for an in-progress run — so grep -q <bad-thing> returns false and reports a PASS off nothing. That exact bug appeared inside a gate written to prevent it.

Three gates, in order, all mandatory:

1. Terminal state, on a BOUNDED loop that exits non-zero on timeout (for i in $(seq 1 240); do … sleep 45; done) so it can never hang forever.
2. Log non-empty — wc -l over 100 lines, else refuse to interpret.
3. The canary RAN — assert fixture canary OK: <n> files is PRESENT, never infer it from the absence of REFUSING TO RUN.

The canary is the harness's own guard: it installs puppeteer@25.4.0 and refuses the whole batch unless the fixture produces >5,000 files. If it refuses, nothing below it means anything — that is how a broken isolation change gets caught before it writes garbage.