--- name: validate-performance description: Measures the throughput difference between two branches with force-balanced routing. Use when the user asks to "benchmark the performance", "measure throughput", "check for a slowdown", "check throughput did not regress", or "compare step time". The user specifies the model and the mesh; sequence length defaults to 2048 and dtype to bf16 — do not infer any of it from git diff. --- # Validate Performance Runs the same short training on the base branch and on the feature branch with force-balanced routing, then reports the step-time difference. It measures; it does not gate. For loss correctness see **validate-correctness**. The user picks the **model** and the **mesh**, e.g. "benchmark qwen3-30b-a3b at pp=2 ep=8". Sequence length defaults to 2048 and dtype to bf16; take either from the user when given. If the model or the mesh is vague, ask. Runs live under `` as a directory, named by a `` that spells out every dimension: `pp2-dp1-cp1-ep8-seq2048-bf16`. Include the defaults, so a log identifies itself. DP is not a config field — it is inferred from the world size and appears in the tag for identification only. The wandb run is grouped as `performance/`, so several meshes for one model land in one group and stay directly comparable, while the correctness pass groups separately and can never be mistaken for it. ## Prerequisites - Activate `.venv` in the repo root: `source .venv/bin/activate`. - A tokenized corpus for the model — run **setup-benchmark-inputs** if `workspace/datasets/dclm-baseline/toktxt/` is missing. No checkpoint is needed. - `world_size % PP == 0` with `CP` and `EP` each dividing `world_size / PP`, `sequence_length % (2 * cp_size) == 0`, and `global_batch / dp >= 2 * pp` (where `dp = world_size / (pp * cp)`). - Keep sequence length, dtype and global batch identical across the two arms, and across meshes you intend to compare to each other. - **A clean tree.** The arms are switched with `git checkout`, so commit the feature work first. The two runs must be **strictly sequential**. One tree serves both arms, so a checkout landing while a run is pending or in flight silently benchmarks the wrong branch. `` is not ``, and `` differs per model: | `` | `` | `` | |---|---|---| | `qwen3-30b-a3b` | `qwen3` | `global-batch` | | `qwen3.5-35b-a3b` | `qwen3.5` | `global-batch` | | `deepseek-v2-lite` | `deepseek-v2` | `sequence` | | `gpt-oss-20b`, `gpt-oss-120b` | `gpt-oss` | `global-batch` | ## Step 1: Create the two run files ```bash R=$(git rev-parse --show-toplevel) W=$R/workspace/validate-performance/ mkdir -p $W $R/logging/validate-performance/ cp $R/.agents/skills/validate-performance/templates/validate.py $W/-base.py cp $R/.agents/skills/validate-performance/scripts/launch.sh $W/launch.sh git rev-parse --abbrev-ref HEAD > $W/FEATURE ``` The launcher is copied into `workspace/` because the checkouts below switch branches, and `.agents/` is tracked. The template carries the defaults as real values, so only a non-default request needs editing: `training.sequence_length = 2048` and `training.fp8 = False`. The `` suffixes name them — `-fp8` is `training.fp8 = True`, and `fp8` is the only dtype knob there is. Fill every `` in that file, then copy it so the two arms are identical by construction: ```bash cp $W/-base.py $W/-feat.py diff $W/-base.py $W/-feat.py ``` The diff must print nothing. Each run takes its wandb name from its own filename, so nothing inside the two files differs. `workspace/` and `*.log` are both gitignored, so the run files and the logs survive every checkout below without ever making the tree dirty. ## Step 2: Run both arms Each fence re-derives its own paths, because shell variables do not survive between commands. ```bash R=$(git rev-parse --show-toplevel); W=$R/workspace/validate-performance/ G=$R/logging/validate-performance// git checkout main bash $W/launch.sh $W/-base.py 2>&1 | tee $G-base.log ``` ```bash R=$(git rev-parse --show-toplevel); W=$R/workspace/validate-performance/ G=$R/logging/validate-performance// FEATURE=$(cat $W/FEATURE) git checkout $FEATURE bash $W/launch.sh $W/-feat.py 2>&1 | tee $G-feat.log ``` Under SLURM, wrap each `bash $W/launch.sh …` in `srun -N -W 0 -o ` — see **launch-with-slurm**. ## Step 3: Report ```bash R=$(git rev-parse --show-toplevel); G=$R/logging/validate-performance// python3 $R/.agents/skills/validate-performance/scripts/compare.py $G-base.log $G-feat.log ``` It prints median step time, tokens per second and peak memory for each arm, plus the percentage difference. Quote those numbers; do not convert them into a pass or fail. ## Why the fixture looks like this `templates/validate.py` sets `training.benchmark = True`, which force-balances routing by overwriting the router's top-k with a round-robin over token index. Expert imbalance is the largest source of step-time variance, and removing it is what makes a short run readable. It also makes the loss meaningless, which is why correctness lives in a separate skill. `training.max_steps = 8`, because step time settles within a few steps once routing is balanced. `compare.py` drops step 1: it carries the JIT warmup and the `torch.compile` trace, which dwarf the steady-state step time. It compares medians of whole runs and never pools individual steps. Within a run step times are near-duplicates, while two runs of identical code sit at different offsets, so pooling counts correlated measurements as independent and reports significance for that offset.