Mm3 Backend

The trap list (each cost real debugging — do not relearn)

1. ComfyUI's wrapper NEGATES the DiT output; the diffusers reference (and our port) does not. mm3.dit.output_negated in the GGUF records Comfy's behavior. Do not "fix" the sign.
2. tokenizer.ggml.pre = qwen2 is misleading — the reference uses the slow Qwen2Tokenizer (single-digit regex = classic GPT-2 pre-tokenization, which bpe.h implements). Matching the KV's llama.cpp meaning ({1,3} digit grouping) breaks token parity.
3. Scheduler sigmas must replicate float32 linspace(1, 1/30, 30) rounding — deriving i/steps is wrong in the 7th digit and it matters.
4. AR iteration 0 is fed back but never emitted (emitted frame j = iteration j+1). A one-frame indexing slip degrades conditioning parity 49×.
5. The semantic code embeds via the LM's token_embd, not depth.audio_embd.
6. Caption hygiene: splitlines() for caption, split("\n") for lyrics — mixing them leaks a trailing \n into the template. Empty lyrics → we substitute [instrumental] (the reference rejects empty; this substitution is a HOT-Step decision).
7. Condition resample is plain nearest, not nearest-exact (differs on 199/689 positions).
8. Never use std::normal_distribution for reproducible noise (stdlib-dependent bytes) — mm3_fill_noise uses splitmix64 + Box-Muller.
9. GGUFs live in the models/mm3/ subdir deliberately: the ACE registry scan globs only the models root (unknown-arch warnings + 17 GB header reparse per boot if placed there).
10. Single-seed spectral/genre judgments are meaningless — the reference's own 11-seed spread spans 272× in flatness and wanders off-genre with minimal captions. Structured 3-section captions (mm3-captioning skill) are the adherence lever. Compare distributions, not takes.
11. VRAM: f16 stack ≈ 22.5 GB + KV (288 kB/position) + ~3 GB compute headroom. Engine-side arbitration evicts idle ACE modules before MM3 warm; Node-side releaseVram() handles the reverse on backend switch and before ACE gens. ~600 MB stays in the CUDA pool after unload (returns on process exit — not a leak).
12. The LM GGUF is not interchangeable with stock Qwen3-8B GGUFs (extended 200 k vocab, untied head) and llama.cpp alone cannot run music generation. It IS interchangeable with a depth-pruned distilled composer — see "Alternative composer LMs" below.
13. read_wav_buf returns INTERLEAVED [T,2]; the DAV encoder wants PLANAR [L:T][R:T]. Use audio_io_read_wav_buf (audio-io.h), which de-interleaves — never the raw reader. mm3-preprocess sliced the raw reader's output as {p, p+T} and made "left" the FIRST HALF of the song with L/R alternating. Since L≈R, that duplicates every sample: an exact 2× time stretch, one octave down. Every cached target was the song in slow motion and five LoRA runs learned to generate slow motion (2026-08-15, fixed 82b2852).
14. VERIFY PREPROCESSING BY DECODING A TARGET AND LISTENING — metadata cannot catch this class of bug. #13 survived a full day because T is PER-CHANNEL frames, so latent_frames / duration stayed at exactly 86.1328 Hz and every arithmetic check on the manifest passed. The DAV parity gate passed too (it is fed encode_ref.py's planar dump). The manifest was written by the same buggy code being checked, so it corroborated itself. One listen to a decoded target found it. POST /mm3/voc-decode?frames=N with raw f32 [128,N] returns a WAV — there is no excuse not to. Objective version of the same gate: encode a 440 Hz sine and measure what comes back (was 220.0 Hz, i.e. ratio 0.5000; correct is 440.0 Hz / 1.0000). A pure tone cannot be argued with, and it brackets which stage is at fault.
15. Rob's ear beat every metric, twice. He called "slow motion, too deep" on the first adapter and again on the third; both times it was explained away as regression-to-the-mean (which produces a genuinely similar description) and five runs of hyperparameter tuning followed on corrupt data. When the user reports a physical symptom — speed, pitch, duration — treat it as literal and test it literally before reaching for a statistical explanation.
16. Gate every trained adapter on ||delta||/||W|| BEFORE any ear test. Healthy LoRA merges move weights 1–5% Frobenius; at lr 5e-4 × 8k steps ours hit median 17% (max 34%) and at scale 1.0 that is a damaged model, not a strong style — jumbled inside a single 689-latent window, invariant to rank/crop/CFG (AdamW makes total movement ≈ lr×steps regardless of rank, which is why every knob "did nothing"). Measure against the ComfyUI f16 checkpoint (mm3-weights/comfy/diffusion_models/), whose keys match the export directly; target median ≤5%. SimpleTuner's reference recipe is lr 5e-5.
17. Training crops must not straddle conditioning-rollout seams. mm3-condition builds the cache from independent 60 s segments; a crop across a seam pairs continuous audio with conditioning that jumps to an unrelated rollout mid-window — teaching "conditioning lies, smooth over it" (mean-collapse pressure). The seams were parsed and never consulted for a week (~13% of crops at 689, 27% at 1378); fixed 5117281 with reject-and-retry.
18. Filter groups at EXPORT, never constrain them at TRAINING. Measured (runs 09/10, matched ~2.6% delta, same groups): full-set training + MLPV surgery = coherent with clear lyrics; --target mlpv trained-from-scratch = intrusions and jumble, with LESS style at matched delta. Gradient denied its natural pathway (q,k routing) emulates it destructively through the remaining groups, so "safe-group" deltas from a constrained run carry structure-entangled content the base attention cannot support. The winning recipe: train ALL sites at modest delta, then zero q,k rows + proj heads in the export (groupfilter.py pattern — q,k rows are B[0:4096] of the fused qkv; ablation-proven: q,k = structure poison, proj_in/out = seed-dependent fuzz, MLP+V+out = timbre).
19. An MM3-only install must not kill the server at boot. The startup gates in hot-step-server.cpp (registry_scan empty → exit 1; partial ACE synth without LM → exit 1) predate MM3 and knew nothing about it: a user with only models/mm3/*.gguf got a dead engine → empty model dropdowns for BOTH backends + the MM3 "weights missing" CTA, while the Model Manager (Node disk scan, checks subdirs) said everything was installed (GitHub issue #118). Both gates now fall through when mm3_weights_present() (mm3-model.h — filename-only probe of <models> + <models>/mm3) is true; ACE handlers already degrade per-request with an empty registry. Any future boot-time hard-exit must ask "can MM3 still serve?" first.

Validation bar for MM3 changes

Forced-replay parity against the fixtures (never sampled-path comparisons — RNG can't match torch). Established floors: per-module ≥ 0.999 corr vs the bf16 dumps (the dumps' own floor, ~1.6e-2 relRMSE) or ≥ 0.9999 vs an fp32 CPU rerun of the reference module. Full-clip replay: 0.9988. If a change should be bit-neutral, prove it with the deterministic seeds.

Deeper reference (read on demand)

Everything below is out of this file to keep it cheap to load. Open the one you need:

• reference/features.md — Feature subsystems: Sampler plugins: shared with ACE; Caption composer: plain English -> Structured Caption, no LLM; Lyric timestamps: use Whisper, not attention; Analysis tool: where the vocals sit
• reference/performance.md — Performance, caches and streaming: Alternative composer LMs; Low step counts go THIN, not dull; MM3 Plank: the AR code cache; Streaming player: listen while it renders; MM3 AR cache: the speedup the Plank is not; Saved plans: the AR cache on disk; Performance budget
• reference/training.md — Training and runtime adapters: Runtime LM adapters; Native LM LoRA training: ace-train mm3-lm-train; Native codes export: ace-train mm3-codes; Training: DiT yes, LM never