Dynamo Frontend Benchmark

Pitfalls & gotchas (read this)

Benchmark methodology

• Closed-loop, not open-loop. Fixed concurrency means you measure concurrency / latency, NOT the server's max throughput. Idle frontend cores usually mean the system is latency-bound (each request spends most of its life waiting between streamed tokens), not that the frontend is slow. To push the frontend toward saturation: raise concurrency AND lower per-request latency (smaller block size → more frontend KV work; shorter outputs).
• Congestion collapse at high concurrency. Pushing concurrency too high can lower throughput (latency explodes faster than concurrency rises). Sweep concurrency to find the knee; don't assume "more load = more throughput".
• Client/server core contention. aiperf is CPU-heavy (client-side tokenizes every prompt, manages every stream across ~25 procs). Co-located with the mockers on OTHER_CORES, it can saturate those cores and starve the mockers — making a "collapse" that's really the load generator running out of CPU. Always check the CPU split (pidstat mocker vs mpstat on OTHER_CORES); if cores are pegged but the mocker is low, the client is the bottleneck.
• Cold-start first run is systematically slow even with warmup — discard it.
• Don't build while benchmarking. Compiles steal cores and ruin the run.

aiperf

• The finalizer hangs/deadlocks on large runs ("processing records…"). The per-request profile_export.jsonl is written incrementally — kill the finalizer and use extract_throughput.py. Don't wait for profile_export_aiperf.json.
• Orphan processes. aiperf's controller spawns many workers; killing the parent can orphan them. Worse: if you ran a capture with sudo, aiperf ran as root and a non-root pkill can't reap it — use sudo pkill -9 -f aiperf. Stray aiperf workers hold ZMQ/mmap resources and make the next run stall.
• --benchmark-duration N (time-based) avoids the giant fixed --request-count
  • finalizer problem for profiling loads.

Profiling

• Off-CPU needs root. Tracepoints (sched:sched_switch) and BPF (offcputime) require root even at perf_event_paranoid=-1 (tracefs event files are root-readable only). On-CPU perf -F.. -g works non-root at paranoid≤1.
• Native perf --off-cpu is often NOT compiled in (needs BUILD_BPF_SKEL=1); it silently no-ops with a warning. Use offcputime-bpfcc / bpftrace instead.
• No frame pointers in release builds → BPF user-stack walking truncates. Prefer perf --call-graph dwarf; bcc still gives good kernel stacks + partial user frames. analyze_folded.py handles the bcc - separator.
• Async-runtime off-CPU is dominated by worker park (futex) which is benign idle, not contention. Look for app-level lock frames (parking_lot/Mutex) to find real serialization. A blocked async task ≠ a blocked thread.

Topology / environment

• Block size must match frontend and mocker. And very large block sizes break the current mocker: at BLOCK_SIZE=2048 requests are received but the mocker never emits a token (40s hang → client cancel, output_tokens=0, "Failed to publish response"). 512 and 1024 work; 64 is realistic. Smoke-test a single request after any block-size change.
• Block size is a lever, not just a detail. Smaller blocks → more blocks per prompt → more frontend KV-routing work (radix tree, hashing) AND more mocker block bookkeeping. At bs=64 a 60k-token prompt is ~940 blocks and the mocker's KV bookkeeping can dominate (~48% of its CPU); at bs=512 (~117 blocks) it drops to ~3%. Pick the block size deliberately for what you're stressing.
• CPU isolation doesn't survive reboot (isolate.sh sets runtime cgroup cpusets on system.slice). Re-run sudo bash scripts/isolate.sh after every reboot. unisolate.sh reverts. Check: cat /sys/fs/cgroup/system.slice/cpuset.cpus.effective.
• NATS doesn't auto-start after reboot (user binary); etcd usually does (systemd). After a reboot, restart NATS before start.sh.
• jemalloc is frontend-only here (via FRONTEND_LD_PRELOAD); the mocker runs on glibc, so its alloc churn can show glibc-arena lock contention (__lll_lock_wait under __libc_free/Vec::finish_grow) in off-CPU. Preload jemalloc on the mocker too if that matters.
• DYN_RUNTIME_NUM_WORKER_THREADS and DYN_RUNTIME_MAX_BLOCKING_THREADS are applied to every runtime the bindings build, including the one the pyo3 async bridge builds for itself. Thread counts are still worth checking in /proc/<pid>/task: if the bridge builds its runtime before a DistributedRuntime is created, the process ends up with two runtimes and twice the threads the configuration describes (a warning says so).

Known result (calibration): with mock workers, the Dynamo frontend is rarely the bottleneck — it's latency/IO-bound, sitting ~60–85% of its pinned cores with ~0 internal lock contention. Frontend micro-opts therefore show flat e2e throughput on this setup; their value is CPU-efficiency/headroom. To make the frontend the bottleneck, use small block size + high concurrency, or real backends, or move the client off-box.

Script reference (scripts/)

• env.sh — config; set DYN_REPO; everything else overridable.
• start.sh — launch frontend (pinned, optional FRONTEND_LD_PRELOAD/FASTOKENS_*)
  • NUM_WORKERS mockers; port preflight, etcd worker-count verify.
• stop.sh — teardown both + drain etcd to 0.
• run_aiperf.sh — one measured run (CONCURRENCY/REQUEST_COUNT/WARMUP_REQUESTS).
• isolate.sh / unisolate.sh — CPU isolation (sudo; Lite by default, --full for max).
• smoke.sh — single-request sanity check (use after any topology/block-size change).
• profile_oncpu.sh — on-CPU perf + flamegraph (non-root): --frontend/--mocker/--pid.
• capture_offcpu.sh — off-CPU bcc + perf (sudo): --frontend/--mocker/--pid.
• analyze_folded.py — top self-time (on-CPU) or innermost-frame + category (off-CPU).
• extract_throughput.py — robust throughput/latency from raw aiperf JSONL.