Writing Benchmarks

Writing a benchmark

A benchmark is a unit of work (what one iteration measures) expressed as a declarative *.bench.ts file that exports exactly config + task. The bench run CLI owns orchestration — the file never calls the runner itself.

Canonical API reference: WRITING_BENCHMARKS.md (repo root) — read it for the full field tables. Runnable templates: examples/01-hello.bench.ts through 11-logging.bench.ts. Reference implementations:

• benchmarks/sandbox/tti.bench.ts — minimal sandbox benchmark (create → exec → destroy, one metric).
• benchmarks/sandbox/dax.bench.ts — rich benchmark (work measured inside the sandbox, pre-measured phase steps, TaskError, legacy results writer).

1. Define the unit of work

Before writing code, pin down:

• Timed boundary — what starts the clock and what ends it. For TTI that's sandbox.create() → first command succeeding; destroy is excluded. Write the boundary in the file's header comment.
• Uniformity — the task body must run identically for every participant. Push provider differences into participant config (sandboxOptions, timeout, destroyTimeoutMs), never into if (provider === ...) branches inside the task.
• Headline metric(s) — the numbers the task reports via measure() and what counts as a success (exit-code checks, scoring.success.requireData).
• Load profile — which of iterations, concurrency, staggerDelayMs define the benchmark vs. stay CLI-overridable. A shape may set slug/name/staggerDelayMs, never iterations/concurrency. See § Horizontal load.

2. Horizontal load — iterations × concurrency

Horizontal load (how many units of work run at once, and how fast they ramp) is controlled entirely by three config/CLI knobs — the task function itself never manages parallelism:

• iterations — total tasks per participant. Each iteration is one full unit of work (one sandbox create → work → destroy). This is the horizontal volume knob: iterations: 100 launches 100 sandboxes per participant.
• concurrency — max tasks in flight at once. 1 = sequential (one-at-a-time cold starts), N = up to N units of work running simultaneously. iterations: N, concurrency: N is a burst: all N sandboxes launch at once.
• staggerDelayMs — task i launches at i * staggerDelayMs. Combined with concurrency: N this is a staggered ramp (e.g. 100 sandboxes, one every 200ms) — measures how TTI degrades as load ramps.

The named run shapes are just presets over these knobs: sequential = {concurrency: 1}, burst = {concurrency: N}, staggered = {concurrency: N, staggerDelayMs: D}.

Two related knobs that change ordering/fan-out, not load:

• groupBy — 'participant' (default: each participant runs to completion before the next) vs 'round' (participants take turns per iteration, so every participant's Nth task sees the same external conditions).
• phases — split iterations into named segments (cold/warm). Total iterations = sum of phase iterations; --iterations scales each phase.

In CI, horizontal scale comes from the matrix: one job per provider, each passing --provider <name> plus the same --run-key so they share one platform run, and --iterations/--concurrency from workflow inputs.

3. Decompose the work into steps

Wrap each phase in step('name', fn) — each step becomes a timed, labeled row on the platform run page.

Sandbox-benchmark conventions (follow tti.bench.ts):

• Steps: create → the work → destroy. Keep names short; display.steps maps them to labels.
• Always destroy in finally: step('destroy', () => withTimeout(sandbox.destroy(), timeout), { reportConcurrency: false }).catch(err => log('destroy failed', { level: 'warn', ... })).
• Guard every external call with withTimeout (benchmarks/src/util/timeout.ts) and a provider overridable timeout.
• log() with { level, meta } for diagnosis — the worker log uploads as a run artifact. Log the sandbox id after destroy (sandboxId() util).
• Step return values shaped like { stdout, stderr, error, exitCode, ... } are captured as step output, not returned — pass captureOutput: false or wrap.
• Work measured inside the sandbox (a script printing BENCH_* lines): return { data, steps, latencyMs } from the task with pre-measured TaskStepRecord[], and set groupBy: 'round' so the runner honors them (participant mode ignores pre-measured steps). See dax.bench.ts.
• Errors: throw TaskError (code + data) for domain failures; let plain Errors bubble for unexpected ones.