Validate Correctness

Validate Correctness

Runs the same short training three times — the base branch twice, the feature branch once — and judges the base-vs-feature difference against the base-vs-base difference.

The user picks the model, the mesh and the sequence length, e.g. "validate deepseek-v2-lite at pp=2 ep=2 seq=4096". Sequence length is filled per run, defaulting to 2048 when the user expresses no preference. Dtype is bf16 unless asked otherwise, and is the one dimension carried in the template as a real value. If the model or the mesh is vague, ask.

Runs live under <model> as a directory, named by a <tag> that spells out every dimension: pp2-dp1-cp1-ep8-seq2048-bf16. Include the defaults, so a log identifies itself. DP is not a config field — it is inferred from the world size and appears in the tag for identification only. The three runs are base0, base1 and feat0, and the wandb group is correctness/<model>.

Method

Two runs of identical code do not agree past step 1. FlashAttention's backward is not reproducible even under torch.use_deterministic_algorithms(True), which reaches neither it nor the Triton kernels; measured on this codebase the forward is bit-reproducible while gradient norms differ ~8% between two identical runs.

So a base-vs-feature delta means nothing on its own. base0 and base1 are identical code, so their difference is the run-to-run floor, and the feature difference is read as a ratio against it. A fixed tolerance cannot substitute: measured drift spans 3.6x across meshes, and --tolerance 5e-3 both rejected a change that four meshes later showed to be indistinguishable and rejected identical code on two of four meshes.

Prerequisites

• Activate .venv in the repo root: source .venv/bin/activate.
• A tokenized corpus for the model — run setup-benchmark-inputs if workspace/datasets/dclm-baseline/toktxt/<tokenizer> is missing. No checkpoint is needed; runs start from fresh weights.
• world_size % PP == 0 with CP and EP each dividing world_size / PP.
• A clean tree. The arms are switched with git checkout, so commit the feature work first — pushed or not. Uncommitted changes either follow you onto the base branch or block the checkout.

Runs must be strictly sequential, since one tree serves both arms. Never queue all three and let the scheduler interleave them: a checkout landing while a run is pending or in flight silently executes the wrong branch. Each run must finish before the next starts.

<tokenizer> is not <model>, and <moe-load-balance-type> differs per model:

Step 1: Create the three run files

Copy the template to base0, fill in every <placeholder>, then copy that filled file to base1 and feat0. Filling once and copying afterwards makes the three identical by construction rather than by discipline. An unfilled copy fails unevenly, so scan the file before launching: the numeric fills — the mesh sizes, <sequence-length> and <global-batch-size> — are unquoted, so a leftover there is a SyntaxError on every rank, while the quoted ones — <model>, <tokenizer>, <moe-load-balance-type>, <wandb-project> — parse and surface far later, a forgotten <wandb-project> only at wandb.init.

R=$(git rev-parse --show-toplevel)
W=$R/workspace/validate-correctness/<model>
mkdir -p $W $R/logging/validate-correctness/<model>
cp $R/.agents/skills/validate-correctness/templates/validate.py $W/<tag>-base0.py
cp $R/.agents/skills/validate-correctness/scripts/launch.sh $W/launch.sh
git rev-parse --abbrev-ref HEAD > $W/FEATURE

The launcher is copied into workspace/ because the checkouts below switch branches, and .agents/ is tracked.

Fill the placeholders in that file, then:

cp $W/<tag>-base0.py $W/<tag>-base1.py
cp $W/<tag>-base0.py $W/<tag>-feat0.py

Each run takes its wandb name from its own filename, so nothing inside the three files differs. Confirm that, because it is the whole basis of the comparison:

diff $W/<tag>-base0.py $W/<tag>-base1.py
diff $W/<tag>-base0.py $W/<tag>-feat0.py

Both must print nothing. workspace/ and *.log are both gitignored, so the run files and the logs survive every checkout below without ever making the tree dirty.

Step 2: Run the base arm twice

Each fence re-derives its own paths, because shell variables do not survive between commands.

R=$(git rev-parse --show-toplevel); W=$R/workspace/validate-correctness/<model>
G=$R/logging/validate-correctness/<model>/<tag>
git checkout main
bash $W/launch.sh $W/<tag>-base0.py 2>&1 | tee $G-base0.log
bash $W/launch.sh $W/<tag>-base1.py 2>&1 | tee $G-base1.log

Under SLURM, wrap each in srun — see launch-with-slurm for the flags that matter:

srun -N <nodes> -W 0 -o $G-base0.log bash $W/launch.sh $W/<tag>-base0.py
srun -N <nodes> -W 0 -o $G-base1.log bash $W/launch.sh $W/<tag>-base1.py

Step 3: Run the feature arm

R=$(git rev-parse --show-toplevel); W=$R/workspace/validate-correctness/<model>
G=$R/logging/validate-correctness/<model>/<tag>
FEATURE=$(cat $W/FEATURE)
git checkout $FEATURE
bash $W/launch.sh $W/<tag>-feat0.py 2>&1 | tee $G-feat0.log