Phase 5 Decode Optimization

Purpose

Phase 4 optimized prefill while preserving Phase 3 correctness. Phase 5 does the same for decode — but the dominant optimizations differ because decode runs at M=1 per token, calling all N layers once per generated token. This phase is a thin orchestrator: it dispatches the same shared optimization skillset as Phase 4 — opt-merge-multi-launch-kernels (ELF-merging), opt-buffer-object-reuse (host↔NPU runtime-overhead reduction), and opt-layout-alignment (host-side layout) — each of which owns its own recipe and failure modes. Decode amplifies the value of static weight BOs (weights are loaded once but read on every token). These wins are things you compose from the kernels, not behaviors inherited from a reference; the reference's builders are worked examples. Every optimization is an experiment: apply, re-measure, re-run Phase 3 gate; revert if correctness regresses.

For scale, the reference deployment llama3.2-1B took decode from ~500 ms/token → 92 ms/token (5.4×) by composing these optimizations — an illustrative datapoint for what they buy on one model, not a target every deployment must hit.

Phase 5 PASS criteria (HARD GATES)

1. Correctness preserved: after every applied optimization, make verify (the token-set gate vs HF bf16) still PASSES — this is the Phase 3 correctness gate, re-run between optimization skills. make diagnosis per-layer cosine is NOT a gate (the verify subsystem retired threshold-based diagnosis; compare_pair reports cosine with no pass/fail), and note it only probes prefill — decode regressions (KV cache, per-token BO reuse) surface in make verify's 32-token generation, not in diagnosis. If make verify regresses to FAIL, revert the change and document why.
2. Decode time/token strictly < Phase 4 baseline, measured with make profile at the same canonical prompt.
3. Per-skill outcome documented in <model>/docs/development_progress/phase5_decode.md: for each optimization skill it invoked, record applied / skipped / reverted, the latency delta, and a one-line reason.

The "≥ N optimization skills applied" check is NOT a gate — some models legitimately need only merge + static weight BOs. The gate is the outcome (decode time improved + correctness preserved), not the process count.

Knowledge base references

PRIMARY:

• programming_examples/llms/llama32_1b/docs/profile.md — the reference deployment's profiling breakdown; the reference for what "good" looks like
• programming_examples/llms/<model>/docs/development_progress/phase4_prefill.md — Phase 4 baseline (prefill numbers + the integration path used)
• programming_examples/llms/llama32_1b/multi_launch_builder/o_gemv_ffn_multi.py — decode-specific merge pattern + 2-K extern kernel rename, in code
• programming_examples/kernel_registry/details/GEMV_bf16.md — per-kernel constraints / placeability notes (the authority for that kernel's hard limits): K_max=8160, combined channel reads ≤ 255, L2 cap — relevant when GEMV K > 8160 or M is large

REFERENCE EXEMPLARS (read/mirror to compose your own decode ELFs; import directly only on a bit-for-bit kernel-sequence match):

• programming_examples/llms/llama32_1b/multi_launch_builder/rms_gemv_rope_multi.py — fused 6-launch decode ELF for RMSNorm + Q/K/V GEMV + RoPE Q/K
• programming_examples/llms/llama32_1b/multi_launch_builder/o_gemv_ffn_multi.py — fused 8-launch decode ELF (with 2-K extern rename for K=8192 Down)
• programming_examples/llms/llama32_1b/multi_launch_builder/lm_head_gemv_multi.py — vocab-partitioned LM Head GEMV (part of the model's decode assembly, built in Phase 3/finalize; profiled here, not a separate optimization)
• programming_examples/llms/llama_kernel_builder/ — the shared toolkit every decode-ELF build uses (KernelCache, stitching, external_kernels).

Workflow

Step 1: Measure Phase 4 baseline

Capture the decode time/token before invoking any optimization skill — this is the number every skill must beat:

cd programming_examples/llms/<model>
flock -x -w 1800 /tmp/mlir-air-npu.lock make profile

Record: ms/token + per-layer + LM head time breakdown (where the budget goes today) as the baseline. These numbers gate every optimization below.

Step 2: Apply optimization skills

decode draws on the same shared optimization skillset; the dominant patterns differ because decode runs at M=1 per token, calling all N layers per token.

Each skill owns its recipe + success self-check + failure modes. The LM Head GEMV (vocab-partitioned) is part of the model's decode assembly built in Phase 3/finalize, profiled here — not a separate optimization skill.