Dynamo Kv Replay Parity

Dynamo KV replay parity

Compare two revisions of Dynamo offline KV-aware replay using deterministic virtual-time reports and statistically paired real-time measurements. Use the existing replay and benchmark harnesses; extend them only when a required signal is unavailable.

This campaign intentionally excludes round-robin routing. It also does not require a same-timestamp event-only progress scenario. Validate those concerns with focused tests outside this skill when a change specifically affects them.

Required inputs

Resolve and record before running anything:

• immutable baseline and candidate commit SHAs;
• Rust toolchain, build profile, flags, and host characteristics;
• each artifact's exact Cargo features from the relevant manifests;
• Mooncake trace path, SHA-256 checksum, and deterministic slice rule;
• engine, topology, concurrency, worker counts, block sizes, and native G1 capacities;
• the canonical-report exclusion allowlist; and
• the performance run-order seed, CPU placement, timing scope, and invalidation rules.

Use the project root .venv/bin/python for Python analysis and uv pip for any approved installation. Do not compare against moving branches or reuse a binary after changing its checkout. Do not describe the configuration as "all features"; record explicit feature names. Feature names can differ between dynamo-mocker and dynamo-bench. The current campaign is native-G1-only; do not add removed KVBM replay features or runtime arguments to make a historical command line build.

Stage 1: Pin revisions and artifacts

1. Confirm the baseline is an ancestor or otherwise document the comparison relationship.
2. Create isolated checkouts for both revisions using the same host and toolchain.
3. Apply any temporary determinism correction identically to both revisions. Keep it out of the measured semantic delta and record its patch checksum.
4. For the current dynamo-bench harness on Linux, build one release artifact per revision with exactly replay-bench and no default features. --canonical-reports-jsonl requires replay-bench, which also selects the seeded router.
5. Reuse that artifact across all native-G1 correctness and performance rows. Do not build separate engine, topology, or production-routing artifacts. If the named manifest feature or runtime contract no longer exists, stop and report that this protocol needs updating rather than guessing a replacement matrix.
6. Copy the artifacts to a temporary campaign directory. Record exact features, binary SHA-256, binary size, and .text size.
7. Return each checkout to its original branch after extracting the binaries.

Build the current benchmark artifact with:

cargo build --release -p dynamo-bench --no-default-features \
  --features replay-bench --bench offline_replay_bench

Never build while collecting performance samples.

Campaign concurrency

Treat one engine/topology/frozen-configuration comparison row as the unit of node placement. When scheduler capacity permits, allocate multiple nodes and assign independent rows to them. The nodes do not need to be homogeneous because no individual comparison crosses nodes. Keep the baseline, candidate, all determinism repetitions, and lifecycle evidence for one row on the same node. Use the same prebuilt revision artifacts and inputs throughout the campaign, and record the node characteristics and CPU placement for every row.

Native replay is normally single-core: pin each process to one physical core after confirming that assumption during preflight. Keep the CPU-set size, NUMA placement, and affinity identical between baseline and candidate for a row.

For correctness, run the baseline and candidate repetitions for a row concurrently when its node has sufficient resources. Pin concurrent processes to disjoint CPU sets, give each process separate output paths, and ensure they do not share mutable state. Keep every repetition in a separate process even when several repetitions run at the same time.

For performance, keep the entire row's warmups and 60-pair measurement series on its assigned node. Execute only one performance invocation at a time on that node with a fixed CPU placement, and keep each randomized baseline/candidate pair adjacent. Different performance rows may run concurrently on different nodes, but do not pool one row's pairs across heterogeneous nodes. Do not run another replay, build, profiler, or unrelated workload concurrently on a node collecting performance samples.

Multi-node and within-node correctness parallelism are campaign throughput optimizations; they must not change workload concurrency or weaken performance isolation.

Stage 2: Establish deterministic reports

Require the harness to control every known entropy source:

• assign stable request UUIDs during workload creation;
• use an owned seeded RNG for temperature sampling and equal-score tie selection;
• sort routing candidates by (worker_id, dp_rank) before deterministic selection;
• use stable request ordinals and same-time event sequence numbers;
• recursively sort JSON object keys;
• sort only explicitly unordered collections such as per-request records;
• preserve semantically ordered event and lifecycle arrays;
• exclude exactly /summary/wall_time_ms, /summary/processed_tokens_per_s, and /summary/processed_output_tokens_per_s.

This exclusion list is closed: include every other field, and stop for review before adding another exclusion. Make semantic fields such as request UUIDs deterministic rather than dropping them. Require parity output metadata to report replay_bench: true; otherwise the seeded routing contract was not active.

Run baseline twice and candidate twice in separate processes. Each revision must produce one unique canonical digest. This is an entropy-leak check, not a statistical trial. If a revision is internally unstable, stop and diagnose it; do not increase repetitions and average the outputs.

Stage 3: Qualify a long interaction-heavy corpus

Prefer a fixed contiguous 5,000-request Mooncake window over many parity repetitions. Preserve arrival order and prefix locality. Record the starting offset, request count, and checksum. Do not randomly sample rows, duplicate a shorter trace, or silently claim a 5,000-request campaign when fewer usable requests exist.

The committed lib/bench/testdata/mooncake_trace_1000.jsonl fixture is suitable for a quick harness preflight, not the authoritative long-corpus campaign.

Qualify and freeze one configuration per comparison row, or per explicitly named configuration family when rows genuinely share every relevant parameter. For each frozen configuration, prove that it exercised every applicable path:

• KV-overlap-sensitive routing;
• immediate placement, plus queued placement only when queueing is explicitly enabled;
• the row's bounded preemption or retraction band at the block-capacity edge;
• disaggregated prefill/decode handoff;
• terminal cleanup.

Use coverage counters, lifecycle traces, or report evidence rather than inferring these paths from successful completion. Target the bounded pressure band recorded for the qualified seed. When no seed exists, start with one to three pressure events to prove the lifecycle. For throttle-oriented disaggregated vLLM and SGLang seeds, target 10 to 20 fully readmitted pressure events so repeated scheduling is exercised. Zero means the edge was not exercised; repeated preempt/re-admit cycling, a rapidly growing pressure count, or failure to advance virtual time invalidates the fixture. Tune capacity or concurrency minimally and identically for baseline and candidate within the row or family, and back off rather than accepting a pressure flood. Never tune the revisions separately.

The offline replay CLI leaves the router queue threshold unset by default. In that mode, all route decisions are immediate and zero queued placements are expected; engine-side scheduler waiting is not router queue coverage. If queue lifecycle coverage is required, enable it through an explicit queue-capable harness or forced fixture and record the exact threshold. Account for every route decision, but never relabel scheduler waiting as queued placement.

Start from qualified internal-polynomial seeds

For the canonical 5,000-row Mooncake window with the internal polynomial mocker, read internal-polynomial golden points before searching for capacity edges. Treat those configurations and observed counters as suggested starting points, not universal constants or substitutes for qualification on the pinned baseline. Requalify one frozen configuration identically on both revisions.

Use the reference's expected preemption, retraction, reuse, worker, and handoff signals as drift detectors. Treat queue counts as expected signals only when queueing was explicitly enabled. Matching lifecycle counts do not waive an unstable canonical digest; stop correctness and performance work for that row until both internal determinism and cross-revision parity are established.

Stage 4: Run byte parity

Run the 5,000-request corpus for this authoritative matrix. Treat both disaggregated rows as the primary scheduler and handoff requalification. Keep the aggregated rows as secondary parity coverage for the corresponding engine semantics.

KVBM G2-G4 replay was removed from the current mocker. It is not an unsupported row in this matrix and must not be emulated with ignored flags. A campaign that needs historical KVBM parity must pin a historical revision together with the matching historical skill protocol. For each current row:

1. Produce canonical baseline and candidate outputs with the frozen configuration.
2. Compare their bytes or SHA-256 digests exactly.
3. Verify the coverage evidence independently of the digest.
4. Delete matching full reports and retain their digests.
5. Preserve full reports and a focused diff only when outputs disagree.

TensorRT-LLM can be additional smoke coverage, but it is not a substitute for either authoritative engine semantic.

Stage 5: Classify semantic differences

Byte mismatch is a review gate, not an instruction to preserve incorrect behavior. Allow an intentional mismatch only when the candidate is demonstrably more faithful to the specified engine, scheduler, or routing semantics.

For every proposed exception, record:

• the exact fields, requests, or lifecycle events that differ;
• the baseline behavior and why it is incorrect or less faithful;
• the candidate behavior and the semantic source of truth supporting it;
• why the difference is caused by the intended change rather than leaked entropy;
• a focused regression test that fails on the old behavior and passes on the correction;
• any downstream report or API compatibility impact; and
• the reviewer-visible disposition.

Use PASS_WITH_SEMANTIC_EXCEPTIONS only when every byte difference is covered by such a record. Unexplained, incidental, or merely convenient differences fail. Do not patch the candidate back to known-wrong behavior just to obtain identical bytes.