Phase 0 Build Cpu Reference

Purpose

Phase 0 establishes the inputs every downstream phase needs. It does NOT build a full-model CPU oracle — the reference is HuggingFace transformers in bf16, accessed through the verify/ subsystem's HfRunner. Phase 0 produces three things:

1. <model>_weights.py — Config dataclass + HF weight loader. Maps HF safetensors names to the per-layer LayerWeights the NPU pipeline consumes.
2. <model>_cpu_helpers.py — the small set of NumPy helpers that production prefill/decode import (default: rms_norm, attention_reference, softmax). These are NOT a per-kernel oracle catalog — each leaf kernel ships its own NumPy reference inside its llama_kernel_builder/<kernel>/run.py harness (Phase 1 uses those).
3. HF bf16 baseline confirmation — verify the target model loads with torch_dtype=torch.bfloat16 and runs the canonical prompt through HfRunner, producing a sane top-1 and non-degenerate logits. This is the Phase 0 gate.

Why HF bf16 directly, and what still needs a NumPy helper

The reference is HF transformers in bf16 — same dtype as the NPU, so NPU-vs-reference is a fair fight (bf16 vs bf16), and there is no hand-written 480-line full-model forward to keep correct. HF's per-layer hidden states (used by Phase 2/3) are captured by HfRunner in diagnosis mode (lite_mode=False), which returns layer_intermediates[].ffn_out plus final_hidden_normed.

NumPy helpers are still required for two narrow cases that HF (a black box that only exposes end-to-end forward + per-layer hidden states) cannot serve:

1. Per-kernel references (Phase 1): HF cannot expose a single kernel's output (RMSNorm alone, RoPE alone, etc.). Each leaf kernel's standalone harness (llama_kernel_builder/<kernel>/run.py) carries its own NumPy F32 reference for that kernel. <model>_cpu_helpers.py only holds the helpers production code itself imports at runtime.
2. CPU fallbacks (production): e.g. prefill cpu_attn=True uses attention_reference when the NPU FlashAttention kernel is unavailable for the configured head_dim; the LM-head final norm uses rms_norm.

Phase 0 PASS criteria (HARD GATES)

Three checks, each catching a different bug class:

1. Weights loadable (catches weight-name / shape mismatches): <model>_weights.py loads every expected tensor; every layer index 0..n_layers-1 has q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, attn_norm, ffn_norm (plus q/k/v bias if qkv_bias=true).
2. HF baseline runnable (catches HF auth / config / dtype issues): HfRunner(model_name, config, max_seq, lite_mode=True).prefill(tokens) on the canonical prompt runs without error, returns a sane top-1 token, and produces logits with no NaN and a non-degenerate distribution.
3. Config consistency (catches config-extraction errors): every field extracted in Step 1 (n_layers, emb_dim, n_heads, n_kv_heads, head_dim, hidden_dim, vocab_size, rope_theta, tie_word_embeddings) matches HF's config.json exactly.

The per-layer and final-logits cosine checks that used to live in Phase 0 now live in Phase 2 (single-block) and Phase 3 (full-model), comparing NPU against HF bf16 intermediates via the verify/ diagnosis path. Phase 0 only confirms the HF baseline and helpers/weights are in place.

Knowledge base references

Read these BEFORE acting:

• programming_examples/llms/llama32_1b/llama32_1b_weights.py — reference Config dataclass + HF weight loading pattern to copy.
• programming_examples/llms/llama32_1b/llama32_1b_cpu_helpers.py — the canonical small NumPy helper file; mirror its scope (only production-imported helpers), not a per-kernel catalog.
• programming_examples/llms/verify/runners/hf_runner.py — the bf16 HF reference runner; this is how Phase 0 confirms the baseline and how Phase 2/3 obtain per-layer intermediates.
• programming_examples/llms/verify/README.md — the verify subsystem methodology (HF bf16 reference, top-k token-set gate, cosine as diagnosis).

Workflow

Step 1: Read the HF config

Fetch config.json for the target model from HuggingFace. Extract:

• num_hidden_layers → n_layers
• hidden_size → emb_dim
• num_attention_heads → n_heads
• num_key_value_heads → n_kv_heads (default to n_heads if absent → MHA)
• intermediate_size → hidden_dim
• vocab_size → vocab_size
• rope_theta → rope_base (default 10000.0 if absent)
• head_dim (compute as emb_dim // n_heads if absent)
• tie_word_embeddings (affects whether to load lm_head.weight)

Step 2: Architecture compatibility check

Confirm the model is in-scope before scaffolding anything downstream:

• Architecture must be in ["LlamaForCausalLM", "MistralForCausalLM" (only if no sliding window), "Qwen2ForCausalLM", "Qwen3ForCausalLM"] — i.e., a decoder-only with RMSNorm + SwiGLU + RoPE + GQA/MHA
• Reject if: MoE layers, sliding-window attention, MLA, encoder-decoder
• Reject explicitly with a clear message; don't scaffold a model the rest of the pipeline can't handle

Step 3: Generate <model>_weights.py

Copy programming_examples/llms/llama32_1b/llama32_1b_weights.py to <model>/<model>_weights.py. Modify:

• LlamaConfig dataclass defaults → match Step 1 values
• HF weight name remapping in load_weights() — most LLAMA-derived models share the same names (model.layers.<i>.self_attn.q_proj.weight etc.); confirm via inspecting the safetensors index. If different, write an explicit mapping.
• generate_rope_lut() — verify rope_base is parameterized and uses the new value
• LM head: load lm_head.weight only if tie_word_embeddings is False
• If qkv_bias=true (Qwen2 / Qwen3 family): add bq, bk, bv fields to LayerWeights, parallel to wq/wk/wv. The bias is loaded just like the projection weights but with a different HF key (q_proj.bias etc.). The bias is applied on the HOST around the bias-free NPU kernels (exploiting RoPE linearity: RoPE(q + bq) = RoPE(q) + RoPE(bq)); the application detail belongs to Phase 2 (phase-2-single-block-validation Step 2) — surface it in TODO.md as a Phase 2 prerequisite. Re-derive the bias-on-host wrapper from the HF reference impl.