V2 Perf Iteration

V2 Perf Iteration — the measurement-driven optimization loop

This is the perf-optimization loop of the v1 multi-agent campaign that ran for months (the old repo-root WORKFLOW.md is now a superseded stub pointing here), upgraded for Runtime-V2's measurement reality. Siblings: v2-model-support (bring-up; its Phase (d) is this skill), v2-kernel-writing (the per-KERNEL inner loop this skill dispatches into when a lever is kernel-body work). The main thread (or one lead orchestrator) runs the loop and does all edits/commits/box-ops; subagents measure-parse / analyze / plan / review / record. No subagent dispatches another subagent, and box operations never go inside a subagent (v2-model-support/references/box-orchestration.md).

Goal + verdict metric — pin ONE per campaign, then do not drift. The loop is metric-agnostic; what is non-negotiable is that a single PRODUCTION verdict config is declared up front and every lever's verdict-grade Δ is measured there. Worked example — the concluded 2026-06/07 DSv3 campaign's instance: e2e decode tpot, bs=1, TP8 EP2, MTP off, toward 8 ms/token (SGLang 7.99 on the same box proves it reachable); v2 clean baseline 2026-07-07: 12.069 ms/tok vs v1 ~9.787; TP<8/local runs are triage only. A new campaign (e.g. Qwen3-8B single-GPU bs=1024 throughput) writes its own goal line in this exact shape — metric, config, target, why-reachable — and restates it in EVERY dispatch prompt. Per-task numbers follow the TIER hierarchy (v2-kernel-writing/references/validation-debug.md §8): TIER-1 = in-MPK %globaltimer / per-position slowest-CTA at the production grid — the ONLY verdict tier; faithful harness corroborates (TIER-2); cudaEvent-wall / standalone-warm are diagnostic only.

⚠️ The .claude/agents/mpk-* defs still carry v1-era DSv3 framing (150µs/MoE-layer, TP4, per_position_grid.py, scratch/ helper scripts that are git-ignored/machine-local). Every dispatch prompt MUST restate the CURRENT campaign goal, the v2 toolchain commands from the quickstart below, and the artifact paths — otherwise the agent drifts to the stale v1 pipeline (each def now carries a "V2 / new-campaign note" saying exactly this).

Environment prerequisites (what must exist on the machine)

In-repo (travels with every clone): the parser python -m mirage.mpk.prof (python/mirage/mpk/prof.py, tracked), the demo --profiling plumbing, this skill's references/ + tools/. Machine-local / degradations:

• scripts/v2_perfetto_export.py / perfetto_analyze.py / perfetto_depgraph.py are UNTRACKED on this branch — a fresh clone will not have them under scripts/. Archived copies travel in .claude/skills/v2-perf-iteration/tools/ — run them from there (or copy back to scripts/, which stays git-ignored for local files). mirage.mpk.prof summary/check/pagewait is the tracked no-dependency fallback for text-table analysis.
• The TP8 box — only for multi-GPU verdict configs (v2-model-support/references/ box-orchestration.md; §1-2 there are site-specific). Single-GPU campaigns run the whole loop locally.
• experiment_history/ — git-ignored ⇒ empty on a fresh clone; create INDEX.md + the journal at step 9 of the first iteration. The kernel-lever anti-loop that must survive clones is v2-kernel-writing/references/m1-decode-evidence.md (in-repo).
• User-level agents (~/.claude/agents/: mpk-perf-analyzer, ablation-logic-reviewer, codex-task-dispatcher) — same-account only; the rest of the roster is in-repo at .claude/agents/. Codex MCP — machine-configured (.mcp.json is git-ignored); absent ⇒ reviews degrade to subagent-only (state it). Personal memory — optional context.
• ~/ref_vllm_sglang.md (analyzer/planner per-kernel reference table) — machine-local; absent ⇒ rank gaps against the SGLang/vLLM numbers recorded in the campaign goal line and in-repo docs, and say the external table was unavailable.

The loop

(1) MEASURE ─▶ (2) ANALYZE ─▶ (3) PLAN ─▶ (4) NEXT-MOVE ─▶ (5) REVIEW-BEFORE-ACT
                                  ▲                                   │
                                  │ (new bottleneck / re-plan)        ▼
(9) RECORD ◀─ (8) LOOP-OR-LAND ◀─ (7) VALIDATE + RE-MEASURE ◀─ (6) IMPLEMENT

Full roster card (what each agent consumes/returns + key discipline): references/loop-agents.md.

1. MEASURE. The mpk-profiler pattern updated for v2: GPU-safety pre-flight → the canonical config → profiled run → parse → cleanup+zombie-guard → report. At TP8 the box session belongs to the MAIN THREAD (setup/poll split, retries, verify-STOPPED — follow v2-model-support/references/box-orchestration.md; do NOT re-derive box mechanics here): rsync → --use-v2 --profiling run (quickstart below) → retrieve the per-rank *_v2prof.npy → v2_perfetto_export.py + python -m mirage.mpk.prof summary/check. Dispatch mpk-profiler itself only for local-GPU triage runs or offline parsing of an already-retrieved buffer — never for box ops. Report = tpot (n-of-N) + the per-task-type consumer-body table (µs/instance × count × layers = ms and % of tpot) + per-position slowCTA

• correctness precondition (coherent output, routed-MoE non-null).

2. ANALYZE (optional on small iterations, mandatory on a fresh baseline). Ranked TODO split kernel-level (body ≫ SOTA ref at M=1 shape) vs system-level (dep-wait, page-wait, role-coordination overhead, AR/skew — the v2 runtime-overhead axis that made v2 12.07 vs v1 9.79). It reads experiment_history/INDEX.md first.

3. PLAN. µs-derived ranked batch plan; every lever: target position + arithmetic + on-critical-path reasoning + correctness risk + dispatch tag. Anti-loop is MANDATORY: check experiment_history/INDEX.md AND v2-kernel-writing/references/m1-decode-evidence.md (the DEAD/WIN/UNTESTED map) — a dead lever is only re-proposable by naming what's different.

4. NEXT-MOVE. One concrete single-iteration move with a falsifiable predicted Δ ("tpot 12.07 → ~11.5 because attn consumer body 108 → ~99µs and attn is on the CP").

5. REVIEW-BEFORE-ACT (MANDATORY, user-locked). Every non-trivial conclusion — root-cause, ablation verdict, dead/alive, ceiling, perf claim — goes through ablation-logic-reviewer (first-principles re-derivation) AND a Codex MCP cross-check (mcp__codex__codex, DEFAULT params) BEFORE you act on it or report it settled. When stuck: detailed multi-turn Codex discussion BEFORE escalating to the user (escalate only when both agree there's no room).

6. IMPLEMENT (main thread routes by tag):

• [ENGINEER] kernel-body / new-op / port work → the v2-kernel-writing skill (SPEC→IMPLEMENT→WIRE→VALIDATE→PERF→REVIEW; its Stage 2 dispatches v2-kernel-engineer, and ferret-kernel-agent/kda-kernel-agent are its beat-a-target engines).
• [FERRET]/[KDA] standalone beat-the-SOTA kernel rewrite → ferret-kernel-agent (frozen-gate autonomous loop) or kda-kernel-agent (verdict-grade honest transfer); routing one-liners in references/loop-agents.md.
• [CODEX] scoped investigation/experiment → codex-task-dispatcher.
• [MAIN] builder / plan / scheduling / system change → main thread edits directly. Every lever lands env-gated default-OFF; default build byte-identical.

7. VALIDATE + RE-MEASURE. Math-changing → mpk-correctness-gate (test-mode + non-null MoE + the TP8-nondeterminism-aware gates: deterministic canary, poison-fill, coherence-in-envelope — see validation-debug.md §7). Math-neutral → token-identity on a deterministic config. Then re-run step (1); the verdict is the TIER-1 TP8 number, and the predicted Δ is confirmed or refuted — say which.

8. LOOP-OR-LAND decision rules:

• WIN (predicted Δ held at TP8) → land: mpk-commit-reviewer gate (staged-path, default-OFF byte-identity, message mechanism+Δ+sign-off) then commit. BLOCK → fix, re-gate.
• NULL/REGRESS → revert the lever (or leave default-OFF-dead), INDEX row WITH the why — recording the death is the deliverable, not a failure.
• Bottleneck shifted / lever class exhausted / stalled → back to (3) re-plan. No stall-stop: a stall means the next idea isn't found yet (planner researches refs).
• STOP only at goal, user halt, or hardware down.

9. RECORD. mpk-memory-keeper appends the journal entry + INDEX one-liner (esp. NULL/REGRESS) + folds structural lessons into personal memory. This closes the anti-loop: steps (2)-(4) read what (9) wrote.