Serving Systems

serving-systems

This skill bundles the curated reference material for LLM and multimodal serving-system development as a topic library under references/. Open the specific reference whose topic matches the task; do not preload everything.

How to use this skill

1. Read this file once to learn what's covered.
2. Open references/platforms/ first. Exactly one backend's directory is present — the one this run targets. Its floor.md is the optimization floor for your hardware.
3. For the active task, identify the one or two topics that match it (use the index below).
4. Open references/<tier>/<topic>.md directly with your file-read tool. Each is self-contained.

Start here: your platform's floor

The default-on optimizations are not the same across hardware, and applying one platform's floor to another produces wrong work — eliminating padding is correct on NVIDIA and inverted on Trainium; graph capture is required on NVIDIA and does not exist on Apple Silicon.

Open references/platforms/<backend>/floor.md for the backend present in this workspace. Only that platform's directory is materialized, so there is no ambiguity about which applies.

Read past floor.md before writing a conclusion. Each platform directory also holds the measurement discipline: turning a counter capture into a bound verdict, proving which kernel library actually ran, and the A/B protocol for before/after claims. Before quoting a "compute-bound"/"bandwidth-bound" verdict, a percent-of-peak or percent-of-bandwidth number, or a kernel-library-tuning recommendation, use that discipline instead of estimating from assumed model geometry (weight bytes, layer split, KV head/dim): an assumption is a hypothesis, not a measurement.

Portable contracts vs platform implementations

Topics split into two kinds, and the distinction is load-bearing:

• Contracts (algorithms/, models/, tooling/, frameworks/) state the problem, the invariants any implementation must satisfy, and the failure modes. These are the same on every backend.
• Implementations (platforms/<backend>/) give the technique for specific hardware.

Where a contract has a platform implementation, the contract links to it. Read the contract first — it tells you what must be true; the platform file tells you how to get there here.

Reference index

Each entry is one file under references/. The bracketed phrase shows what triggers it.

Platforms

One directory per compute backend, each with floor.md, hardware.md, and profiler.md plus its own kernel and framework notes. Only the selected backend's directory is present.

• references/platforms/ — start at floor.md.

Serving algorithms (portable contracts)

• references/algorithms/async-scheduling.md — Hide host scheduler work behind accelerator compute. Contract; mechanism is per-platform.

• references/algorithms/batched-sampling.md — Per-request sampling parameters in one kernel pipeline, without per-request host sync.

• references/algorithms/chunked-prefill.md — Split long prompts into chunks interleaved with decode, preventing a long prefill from stalling decode latency.

• references/algorithms/continuous-batching.md — Requests join a running generation loop between steps. Contract; the KV strategy inverts between backends.

• references/algorithms/cross-attention-kv-cache.md — Cross-attention KV cache for encoder-decoder decode (Whisper, mllama): compute encoder-context K/V once at prefill, read every step. Non-causal, no RoPE, separate pool.

• references/algorithms/disaggregated-serving.md — Separate prefill and decode worker pools with KV transfer between them.

• references/algorithms/heterogeneous-kv-cache.md — Memory management and prefix caching for hybrid models (full-attn + sliding-window, attention + SSM/Mamba, attention + linear).

• references/algorithms/moe-routing-dispatch.md — MoE routing and dispatch — top-k gating, token-to-expert dispatch/combine, grouped-GEMM expert FFN, expert parallelism, expert load balancing.

• references/algorithms/paged-attention.md — Block-based non-contiguous KV storage with a page table per request. Applies where the backend has a discrete memory pool.

• references/algorithms/parallelism.md — TP, PP, EP, DP, SP and combinations. Multi-device backends only.

• references/algorithms/quantization-schemes.md — Precision, granularity, calibration, checkpoint layout. Hardware support is generation-gated.

• references/algorithms/radix-prefix-caching.md — Share KV cache across requests with common prefixes via a radix tree with LRU eviction.

• references/algorithms/speculative-decoding.md — Draft proposals verified in one target pass. Contract; variable accept length is handled per-platform.

• references/algorithms/structured-output.md — Grammar-guided decoding (XGrammar, Outlines, llguidance), JSON mode, regex constraints, tool calling, logits biasing.