Flash Attn Training

6. Trap list

1. ggml.h edits invalidate ~141 CUDA objects — ~1 h rebuild. Batch header changes. New .cu files need a cmake re-configure (the ggml-cuda CMake globs *.cu).
2. DLL locks. A running ace-server holds ggml-base.dll/ggml-cuda.dll; any ggml change needs the app down (/api/shutdown or dev-rebuild.bat). ace-train.exe is NOT held, so trainer-only edits build with the app up. Never kill ace-server (Node respawns it).
3. Packed-output alignment gap. Autodiff builds the packed gradient as ggml_scale(packed, 0) + ggml_acc(dO); garbage in the O→LSE alignment gap becomes NaN. Both CUDA and CPU forwards zero the gap explicitly. Zero-width at every tested geometry, so tests never see it — keep the memset.
4. In-place SCALE hazard. ggml_scale is in ggml_op_can_inplace; it is safe only because the packed tensor always has a view child. The backward asserts dst->data != fwd->data.
5. GQA at B>1 cannot be parity-tested against the manual chain (ggml MUL_MAT backward asserts on broadcast src0) — that is why dit_expand_heads exists. Flash mode skips the expansion (native GQA), which also disarms the CUDA REPEAT_BACK cap on Nkv·max(S,enc_S)·B. Measured: batch 1 still wins on throughput and loss.
6. The server always emitted --crop-max, which the engine treats as a user pin → the flash lift never fired from the UI. cropMax 0 now means "omit the flag". Quality presets must not re-pin it in flash mode. Any new trainer flag with an engine-side "user set it" sentinel has this exact failure mode — check the arg emitter.
7. The parity tool must seed the loss gradient with 1.0 (ggml_set_loss only allocates) and assert a non-zero reference gradient, or both arms compare 0 vs 0 and pass vacuously.
8. dit_sa_mask never produces a fully-masked key column (pad columns stay open for padded query rows) — use dit_ca_mask for the exactly-zero-gradient assertion.
9. Fully-masked query rows: the fused op defines O=0, LSE=0; soft_max_ext produces NaN. Exclude them from reference diffs, check them directly.
10. TF32 A-operand lane map ≠ accumulator map. mma.cuh's tile<16,8,float> is the C/D map; using it as the tf32 A operand gives deterministic garbage. Derive with a probe kernel.
11. Patch files are LF; a scratch tree extracted under core.autocrlf=true is CRLF and every hunk fails. Replay with git -c core.autocrlf=false -c core.eol=lf archive. Export patches hunk-filtered: several patches share ggml.c and ggml-cuda.cu.
12. rocm-build.yml did not apply patches until 1b7e50d5 — every workflow that builds the engine needs the apply loop now that the trainer references patch-provided symbols.
13. MAX_FREE_BLOCKS (ggml-alloc) was 256; LoKR dim 256 (19k-node graph) overflowed it. Now 1024 via alloc-free-blocks.patch. Inference-shared → smoke generation after touching.
14. Workflows die with the VSCode/Claude process. Long unattended runs need the window open; machine sleep is "never" on this box (checked).
15. --mirror bf16 means bf16 COMPUTE, not just bf16 storage — it rounds activations and gradients at every trainable-layer GEMM, and the adapters it trains are audibly coarse ("bitty", Rob 2026-09-02). Use --mirror bf16-f32: same BF16 residency, an in-graph ggml_cast to F32 at each mul_mat site, and over 12 same-seed epochs on mika it is bit-identical to --mirror f32 while bf16 drifts to 7.8e-3. It costs ~180 MB of transient arena and ~25% step time against f32 at equal crop, and buys 2.5× the flash auto-fit crop (1542 vs 610). Only --bwd mm carries it — the out_prod fallback arm keeps the forward cast alive and silently spends the ~8 GB back.
16. Disk. Probe runs write adapters; a campaign filled D: to 2.4 GB free and artifacts were deleted for space. Clean scratch dirs between grid cells.
17. A bench tool's own reference arm can be non-contiguous where the trainer's never is. fattn-train-test --bench-lm's blocked arm fed a ggml_cont(view) straight into the reference attention chain, whose backward hands back a transposed (non-contiguous) gradient — GGML_OP_CONT's backward asserts on that and the tool produced no table at all. The trainer never hits it because a ggml_reshape always sits between the cont and the chain, and RESHAPE's backward re-conts. Fix: wrap each bench-arm tensor in a shape-preserving ggml_reshape too, so the bench pays the same backward copy the trainer pays. Any bench harness that hand-builds a reference graph needs to mirror the trainer's node shapes, not just its op sequence.
18. --max-len filters, it does not truncate. Songs longer than it are skipped outright, so alloc_seq = min(max_len, longest SURVIVING sample) — pinning a value above the whole corpus's longest track yields an empty dataset (no-samples), and a VRAM-model cell "at S=1024" is really whatever the longest surviving song happens to be. Pick the dataset for the S you want, then report the actual S; don't trust the flag to hit a number.
19. The exact-mode naive auto-fit can pick a maxLen whose own estMb already exceeds free VRAM, then die on cudaMalloc with a hard access violation (0xC0000005) instead of a clean lm_fatal — reproduces identically on a pre-flash binary, so it is not new. Root cause is the same non-attention polynomial (c2f/c2h) the flash branch's naive_nonattn_scale now corrects around; the exact-mode fix is owed (see §8) and needs its own gate since it moves every shipped run's estMb.

7. Numbers worth remembering (5090)

8. Open items (as of 2026-09-02)

• R4 (MM3 DiT) port remains (this skill is its brief) — R2 (AS1.5 LM) and R3 (MM3 LM) are DONE, both off by default pending ear tests.
• Cross-attention backward kernel: dK/dV role split blocked by smem; a dQ split exists in the plan doc (reverted, −2.7%).
• DIT_FLASH_LOKR_RETENTION refit after the apply reorder; batch>1 VRAM term.
• Exact-mode arena polynomial under-predicts 13–18% (masked by LoKR over-count; fix gated to flash).
• LM exact-mode c2f/c2h non-attention polynomial is ~2.2× light on the naive path (−11.9% to −12.9% measured, same class as the DiT's exact-mode item above); the flash branch's naive_nonattn_scale corrects around it but the exact-mode fix itself is owed and needs its own gate, since it would move every shipped run's estMb/auto-fit maxLen.
• LM G5/G6 ran on albumF, not album I — the box has no albumI* tensor dir, and the plan's ear pair (G7) is specified on album I/E3 lineage. album I codes need Preprocess + Extract via the Training Studio batch pipeline before G7 can run as written.
• LM G7 ear test (twin album I adapters, staged in _experiments/_LISTENING) — not run, needs Rob; the flash checkbox stays off until it lands.
• Pre-existing bugs surfaced while porting R2, neither fixed (both reproduce on a pre-flash binary): mm3-lm-train crashes at export with a ggml-backend.cpp tensor-write-out-of-bounds assert; the LM exact-mode naive auto-fit can pick a maxLen that OOMs via access violation instead of a clean fatal (trap 19).
• Ear validation of anything trained since the first flash adapter, and of the LoKR reorder.
• Low-VRAM training profiles for users (B1) — deferred by Rob until the 32 GB path is nailed.