Phase 7 Independent Evaluator

Purpose

The deploy-new-llm chain is autonomous. Phase 6 wires up a make verify gate that's supposed to compare NPU vs the HF bf16 reference via the verify/ subsystem's top-k token-set check. But the deployment agent wrote (or copied) both the gate AND the production code it exercises — nothing structurally prevents the gate from being mocked, pointed at a stale baseline, or wired to a verify-only code path that bypasses the real kernels. Phase 7 closes this by:

1. Spawning a FRESH subagent with no context from the deployment
2. Auditing the make verify code path (anti-reward-hacking) — confirming the gate actually drives the production kernels vs HF bf16 and gates on the token-set check, not a mocked / stale / verify-only path. This is what makes the independent re-run meaningful: without it, "the agent re-ran the gate it wrote itself" proves nothing.
3. Re-running the end-to-end make verify independently — the shared verify/ top-k token-set check (vLLM-aligned) of NPU vs HF bf16 on the real production prefill/decode path. A clean independent PASS, on a gate the audit confirmed is real, is the verdict.
4. Producing a short evaluation report humans can read in 2 minutes.

The deployment is NOT considered trustworthy until this report is written and its overall verdict is PASS or PASS-with-warnings.

Phase 7 PASS criteria (HARD GATES)

1. make verify audited (anti-reward-hacking): subagent has read the Makefile target + the model's verify_adapter.py + the shared programming_examples/llms/verify/{verify_runner,comparators,runners/hf_runner}.py, and confirmed the gate (a) drives the model's PRODUCTION run_npu_prefill / run_npu_decode_step via the NpuRunner (not a mock), (b) compares against HF transformers in bf16 (torch_dtype=torch.bfloat16), (c) gates on compute_topk_set_check (top-5 set inclusion at first divergence), not a hardcoded return True / > 0 threshold.
2. make verify PASSES under the fresh subagent run: token-set gate exits 0 (no npu_vs_hf FAIL), on the audited-real gate.
3. Short evaluation report written with the audit verdict + the re-run result + the measured numbers behind them. Verdict PASS or PASS-with-warnings.

If the subagent declares PASS without showing measured numbers OR without auditing the verify code path, the report is REJECTED — re-spawn with stricter instructions.

Knowledge base references

• programming_examples/llms/verify/README.md — the verify methodology the audit checks against (HF bf16 reference, top-k token-set gate, cosine-as-diagnosis)
• programming_examples/llms/verify/comparators.py — the gate implementation (compute_topk_set_check) the audit confirms is real
• <model>/docs/development_progress/ + the kernel_registry rows with Used by = <model> — Phase 1 catalog (subagent compares its measurements against this)
• programming_examples/kernel_registry/supported_kernels.md — kernel-by-kernel ground truth

Workflow

Step 1: Spawn fresh subagent with independence + audit constraints

Use the general-purpose Agent type. Critical instructions in the spawn prompt:

• Independence: do NOT read <model>/docs/development_progress/{LESSONS,progress,phaseN_*}.md BEFORE forming your own measurements. You may CITE them AFTER measuring (compare your numbers vs claimed).
• Re-derivation: every PASS/FAIL verdict must be backed by a number YOU measured or a code path YOU read during this audit, not a number copied from a deployment doc.
• Reward-hacking smell test: if the deployment-claimed numbers look surprisingly good, audit the gate implementation FIRST — does make verify really drive the production prefill/decode vs HF bf16 and gate on the token-set check, or did the agent shortcut to "if NPU output exists → PASS" / point the verify_adapter NpuRunner at a mock / compare against a stale cached baseline?

Step 2: Audit make verify — anti-reward-hacking

BEFORE running anything, READ:

1. The Makefile: what command does make verify invoke?

   grep -A 5 "^verify:" <model_dir>/Makefile

   It should run the shared runner ../verify/verify_runner.py --runner=<model>.verify_adapter --prompts topk_token ... — not a model-local copy of the runner.
2. <model_dir>/verify_adapter.py: this is the model's only verify code. Confirm its NpuRunner (the .prefill() / .decode_step() methods) imports and calls THIS model's production run_npu_prefill / run_npu_decode_step from <model>_inference.py — NOT a stub, NOT the llama32_1b copy. (The runner/comparators/report/HF runner live once in the shared programming_examples/llms/verify/; the adapter is the per-model hook.)
3. programming_examples/llms/verify/runners/hf_runner.py (shared): confirm the reference loads with AutoModelForCausalLM.from_pretrained(..., torch_dtype=torch.bfloat16).
4. programming_examples/llms/verify/comparators.py + programming_examples/llms/verify/report.py (shared): confirm the gate is compute_topk_set_check (top-5 set inclusion at first divergence) and report.has_failure() drives a real exit code — not a hardcoded return True / > 0 threshold.
5. If any of (2)-(4) fails: report [FAIL] verify gate is reward-hacked and stop here. Tag the deployment as needs-remediation.