Benchmarking

Benchmarking

Target-agnostic methodology to measure and prove micro-optimizations in any hot path of this project. Theory does not count: every claim must survive a byte-identical correctness gate and a statistically clean benchmark before touching src/.

Ground rules

1. Behavior-identical or explicitly flagged. A variant must produce byte-identical output (side effects included) versus the baseline. Intentional behavior changes are flagged experiments, and the final decision belongs to the user.
2. Realistic workloads only. Fixtures replicate what the real call sites feed the target, in production-like ratios.
3. Sequential measurement, never parallel. One hyperfine comparison at a time, on an otherwise idle machine.
4. Noise floor discipline. Differences under ~2%, or within one stddev overlap, are neutral. Neutral results do not justify code changes.
5. Composition is not additive. Re-measure winners combined (see the JIT cliff in Findings).

Pipeline

Phase 0: Scope

• Map the target functions, every consumer (grep -rn "fnName" src/), and the call frequency of each. Frequency decides the verdict: a 5x win on a function called once per run has zero real impact, and a lookup table built at module load never amortizes for it. The final report must weigh gains against real call counts.
• Extract the exact runtime constants the target depends on (formatted prefixes, environment values, shared state) so the harness replicates them byte by byte.

Phase 1: Hypotheses (parallel)

Three independent lenses, each producing structured hypotheses:

• V8/JIT: inline caches and megamorphic property loads, instanceof vs Array.isArray vs getPrototypeOf, typeof chain ordering, inlining budgets (function size), builtin fast paths, deopt triggers.
• Allocation/GC: per-call allocations, intermediate strings, methods called only as a boolean test, lookup tables for closed domains, pre-allocation when sizes are known.
• Strings/algorithms: rope (ConsString) vs array+join, repeated scans over the same data, scan order by selectivity, hoisting invariant checks out of loops.

Each hypothesis states: claim, concrete replacement code, the workload where the effect appears, and a behaviorChange boolean. With subagents, run the lenses in parallel (independent and read-only).

Phase 2: Harness

One directory per investigation under the gitignored temp/:

mkdir -p ./temp/bench/<target>/impl ./temp/bench/<target>/results
cp .claude/skills/benchmarking/templates/* ./temp/bench/<target>/

./temp/bench/<target>/
  run.ts                static: generic driver, runs unmodified
  gen_variants.py       static skeleton: fill only the `variants` dict
  fixtures.ts           per target: fill every TODO(target) marker
  impl/baseline.ts      per target: faithful port of the current code, zero project imports
  impl/<variant>.ts     per target: generated, one file per hypothesis, minimal diff over baseline
  results/*.json        per comparison: hyperfine exports

Runner: bun --bun run.ts ... (~11ms startup). Fallback only when Bun is not available: node --import=tsx run.ts ... (~125ms). Never mix runners between the two sides of a comparison. The runner picks the engine: Bun measures JavaScriptCore, Node + tsx measures V8.

run.ts is generic because fixtures.ts exports a fixed contract: TargetModule (surface of the module under test), verifyCases (string per case), benchModes (iterations plus a checksum-returning run), and resetSideEffects/sideEffectsSnapshot for shared counters.

Requirements:

• Faithful baseline port: copy the target nearly verbatim (types stay, the runner executes TS), inlining constants and stubbing shared state so the module is self-contained.
• Hot/cold fixture split: common cases weighted ~5x over exotic ones, matching what production sees.
• Checksum output: every bench mode accumulates and prints a checksum. Prevents dead-code elimination and doubles as a cross-variant sanity check.
• Calibration: tune iterations until each process does 500-800ms of real work, so startup does not dominate. Check with time bun --bun run.ts bench-x impl/baseline.ts.
• Verify is the gate: compares baseline vs variant over all fixtures (outputs and side-effect counters), exits non-zero on mismatch. No variant is benchmarked before passing it, except flagged behavior-change experiments whose mismatch is inspected and confirmed as only the intended difference.
• Domain sweep for enumerable inputs: fixtures sample the input space, which is not enough for numeric kernels with enumerable domains (a byte, a bounded integer, a nanoseconds field). Brute-force compare baseline vs variant over the domain (costs seconds). A variant here passed an 8-sample gate while diverging by 1 ULP on 29% of the domain.

Phase 3: Variants

gen_variants.py generates each variant from the baseline via exact-match replacements with occurrence-count assertions. Fill only the variants dict, copying old blocks verbatim from impl/baseline.ts (indentation included). A not-found or miscounted block aborts generation: that is the protection against silently benchmarking a baseline-identical variant. Never hand-edit near-identical variant files.

Phase 4: Measurement

One comparison at a time. hyperfine already runs its warmups and runs sequentially, so isolation only breaks at the orchestration level: never start a comparison while another is running.

hyperfine --warmup 3 --runs 10 \
  -n baseline 'bun --bun run.ts bench-x impl/baseline.ts' \
  -n variant  'bun --bun run.ts bench-x impl/variant.ts' \
  --export-json results/variant.json

• --warmup 3 --runs 10 is the project standard. Same runner on both sides, always.
• Export JSON, record mean, stddev, min, max per side, and hyperfine outlier warnings verbatim (they change interpretation).
• With subagents: one per comparison, in a sequential loop (for ... await), never parallel().

Phase 5: Interpretation

• Under ~2%, or within one stddev overlap: neutral, discard.
• Outlier warnings with inflated mean: compare medians, or re-run.
• A regression is as valuable as a win: it proves the current code right and blocks future "improvements". Document it.
• Behavior-change variants that measure neutral are dead (all cost, no gain).

Phase 6: Composition and focused workloads

• Combine winners into a combined variant and re-measure against baseline: gains do not add up.
• Wins diluted in the mixed workload (a branch only some inputs reach) get a focused bench mode exercising only that path, measured baseline vs variant vs combined. A 0.998x "neutral" here measured 2.9x focused.

Phase 7: Anomaly forensics

When a combination behaves unexpectedly, isolate before discarding:

1. Micro-mutation bisection: variants differing by exactly one ingredient until the toxic pair appears.
2. Deopt check: node --trace-deopt ... | grep -c deoptimizing against baseline.
3. JIT on/off: run both shapes with --max-opt=0. A tie interpreted plus divergence with JIT on means an optimization-quality cliff, not algorithmic cost. Document the toxic shape.