Sandbox Bench

Sandbox bench: paired A/B perf runs for React and Next.js changes

Measures what a change is actually worth, end to end: two revisions ("arms") built into otherwise-identical Next.js apps, exercised by the bench/render-pipeline harness on Vercel Sandbox VMs, compared with paired statistics that treat the VM boot as the unit of replication. All heavy work happens on sandbox VMs; the laptop only orchestrates.

Scripts live in scripts/ next to this file and run from anywhere. Arms are git refs, resolved in cached clones of react and next.js; the Next side defaults to canary. Everything is cached content-addressed: first use of a new pair builds caches (~45-60 min extra, once); later runs boot straight into measurement.

One-time setup

1. node scripts/config.mjs show — if it reports NOT CONFIGURED, ask the user which Vercel team and project the sandbox VMs should run under (these are billed resources; never guess, never default), then node scripts/config.mjs set team=<slug> project=<name>. Config lives in ~/.config/sandbox-bench/config.json — never commit team/project names into the repo.
2. The Vercel CLI session must have access to that team. On a 403, stop launching (don't retry through it) and check whether access is already back: vercel whoami --scope <team-slug> plus one scoped read call (e.g. vercel sandbox ls) — grants drop and recover on their own, and a transient 403 needs no login at all. If verification still fails, run vercel login <team-slug> yourself as a background task (the token lives with the CLI session, not with the user). It opens a browser/device confirmation — relay the URL if one is printed — but keep re-running the verification pair every minute or two while it waits: access often returns before the login flow reports success, and once verification passes, kill the pending login and resume. After a 403 outage, expect in-flight runs to have died: run node scripts/bench-status.mjs and follow its recovery actions (measurement VMs will have hit their ~5h timeout if the outage was long — those cells need relaunching, not collecting).
3. react and next.js clones land in the cache on first use (or point reactRepo/nextRepo in the config at existing checkouts).

Before the first real run with a new configuration, sanity-check the plan with --dry-run (prints what would happen, touches nothing).

Workflow

1. Resolve what's being compared

• React PR: --pr <url|number> — base is computed automatically (merge-base of the PR head with react main).
• React refs: --arms base=<ref>,cand=<ref> — base FIRST. For a multi-commit branch, base is the merge-base with main, not cand^.
• Next.js PR: --next-pr <url|number>. The React side defaults to whatever each Next ref vendors (that's what would ship); pass --react-ref only to pin both arms to one specific React build.
• Next refs: --next-arms base=<ref>,cand=<ref>.

Exactly one side varies; the other is identical in both arms. That isolation is what makes the numbers attributable — never vary both.

2. Gate correctness before spending bench compute

A bench number from an arm that fails its own tests is meaningless. For any arm that is not already CI-green upstream (hand-assembled branches, cherry-picks with resolved conflicts, local commits):

node scripts/sandbox-gate.mjs --arms cand=<ref>

The bench itself enforces the primary gate: every react arm's commit must have green CI on the react repo, checked automatically before any build or VM is spent. PRs and main-history commits normally satisfy this with no extra work. For local or unpushed refs (no CI exists), gate on a VM with sandbox-gate.mjs and then pass --allow-ungated to the bench. The VM gate runs the full test suite in prod mode (the channel that gets benched). PASS requires seeing the actual test counts in the output. If a gate fails, report the failures and stop — do not bench a broken arm. Each arm is gated in its own lockfile's environment. Bench the exact sha the gate prints (a branch ref can move between gate and bench).

3. Launch the bench (background, non-blocking)

bash -c 'node scripts/sandbox-e2e.mjs --pr <url> --label <slug> \
  2>&1 | grep --line-buffered -v "^live "; exit ${PIPESTATUS[0]}'

For React PRs, launch BOTH suites (separate background tasks; they share arm builds and caches):

bash -c 'node scripts/sandbox-ssr.mjs --pr <url> --label <slug>-ssr \
  2>&1 | grep --line-buffered -v "^live "; exit ${PIPESTATUS[0]}'

The e2e suite measures the Node path through a real Next.js app; the ssr suite measures the react repo's flight-ssr-bench fixture — 8 variants (Fizz and Flight+Fizz, Node and Edge web streams, sync and async), each sequentially with Flight script injection and behind an HTTP server at c=1/c=10. Edge cells are the ssr suite's headline (the e2e suite cannot see that path); its Node and Fizz-only cells attribute an effect to the Flight layer, the Fizz layer, or the stream plumbing. The fixture (the workload) is pinned to one ref for both arms — react main by default — so only the React builds differ; if the PR itself edits the fixture, the launcher says so and the run does not measure those edits. Next PRs run the e2e suite only.

• Run it as a background task and proceed on its completion notification. Never hold a foreground wait; never poll in a loop — the rule is about control flow, not status relay: reading the output tail to answer "how's it going" is always fine.
• The harness handles the invariants internally: both arms in the same VM, interleaved ABBA, paired per (vm, run); detached remote execution (transport drops don't kill runs); build fingerprints recorded in every result row.
• live ... lines are streaming estimates for progress display only. Never stop a run early because a live p-value looks good, and never report a live number — sequential peeking manufactures false positives. Only the final analysis counts.
• Defaults (16 VMs × 2 paired runs) implement the methodology; don't reduce VM count to save time — boots are the unit of inference, and fewer boots means wider intervals, not faster answers.
• Sandbox compute is internal capacity, not a budget: launch, relaunch, and confirm runs without asking about cost or shrinking them to save it.
• The Next side's default, canary, is the latest published canary release (the launcher prints its version and sha), so repeat benches reuse the built snapshot until a new canary ships.
• Useful flags: --bench-env KEY=VALUE (runtime-only env for the bench process — it does NOT affect the snapshot's app build), --isolate-routes (tail investigations), --no-profile (skip the CPU capture that runs by default after the timed runs), --prepare (build caches only — use when two cells will share an arm, to avoid duplicate builds racing).
• CPU profiles are captured by default: one profile pass per arm runs strictly AFTER the timed runs (it cannot touch the numbers), costs ~45-60 min extra VM wall-clock, and lands in <runDir>/prof-vm<N>/ as standard V8 .cpuprofile files. Cross-VM profile diffs are highly stable (observed 16/16 sign agreement on real movers), so one profiled cell suffices to rank hot paths. Analysis caveats: aggregate by (functionName, line, column) — bare minified names collide across the bundle — and never diff arms by minified name (the minifier renames between builds); match positions or code snippets instead.
• The bench exercises Next's node-streams path (__NEXT_USE_NODE_STREAMS is inlined as true for the node runtime at build time). React changes that only touch the EDGE stream configs are not exercised end-to-end and will (correctly) bench as no detected difference.

4. Read the result like a skeptical data scientist

The goal is the truth about the change, not making its author feel good. The final analysis prints, per route/phase/metric, the boot-level mean, ±95% CI, and p across boots. Apply the policy in references/methodology.md:

• Claim only boot-level p < 0.01, with the CI, on an A/A-validated team/config (see methodology).
• The PR is a hypothesis, not an explanation. Claims come from the analysis output alone. When the numbers agree with the PR's story, check whether the captured data actually discriminates that mechanism from alternatives — a latency win attributed to smaller payloads should come with a document-bytes delta; if the bytes didn't move, the story doesn't hold and the report says so.
• Use every captured metric, and voice anything that does not add up: one metric family moving against the others, effects with no byte-level or RSS trace, throughput moving without latency, sign flips across boots. An inconsistency you cannot explain belongs in the report, not in the drawer.
• The within-run p shown in brackets is a diagnostic, never a claim.
• Check the fingerprint header first: two distinct fingerprints = valid A/B; "inconsistent fingerprints" = invalid, report no numbers. The fingerprint hashes both bundlers' compiled server files — arms touching only client files can still legitimately show identical fingerprints with different version strings.
• Per-boot values are printed; if boots disagree in sign, say so.
• Any claim that will drive a decision gets one independent confirmation run before it's stated as fact.

Re-analyze any past run without re-running it: node scripts/bench-analyze.mjs <runDir>.