Mm3 Lm Adapter Training

MM3 LM Adapter Training — the working recipe

Goal this recipe is tuned for: an album clone. You want a generation that could be another track off that record. Memorisation is acceptable and in fact desirable; style bleed is acceptable. This is NOT tuned for a surgical trigger LoRA that leaves the base model otherwise untouched.

Established over a 6-album, 30,000-step sweep on 2026-08-23/24 (album B, the album-C artist, album K, album M, album L, the album-A artist), each album laddered by ear across 20 checkpoints. Working log: docs/plans/ 2026-08-23-mm3-albumC-style-adapter-findings.md (gitignored, local only).

THE CURRENT BEST RECIPE

Keep this block up to date — it is the thing the in-app trainer should eventually be dialled to, and docs/plans/ is gitignored so nothing there survives a fresh clone.

WHOLE-SONG since 2026-09-11. The crop-750 / history-2048 recipe below this block's history taught adapters that lost the song's arc: one sung passage, minutes of looped instrumental, no ending (vocal share 0.16 on album P against the album's 0.68). Nothing in a 30 s window ever scores more than 30 s at once. On the 2026-09-10 overnight those adapters ended 12 of 72 renders across six fresh albums while the pristine base ended 36 of 36 on the same prompts, with no dependence on album length or genre. Training every track as ONE sequence (--max-frames 9000, no prefix, flash) ended 7/12 on albums B and S and 6/12 on album P at the 360 s ceiling, and lifted album P's vocal share to 0.42. Cost on an RTX 5090: 6-12 s/step, 29 GB peak at rank 128 (the engine clamps its buffers to the album's longest track since 2026-09-11, so short albums cost less). Whole-song ending times follow the LYRIC SHEET's length more than the album's. Not yet re-heard for likeness at the time of writing; the ledger docs/plans/mm3-endings-checklist.md has the tables.

--lm mm3-lm-q8_0.gguf
--rank 128 --alpha 128 --adapter-type lora --hot-pizza          # HOT-PiZZA: PiSSA with principal-subspace dropout (2026-09-06)
--pissa-frozen-f16 --pissa-cache-dir <adapters>/mm3-lm-adapters/_pissa-init-cache
--optimizer adamw --lr 8e-5 --lr-end-frac 0.005 --warmup 25    # AdamW tied Prodigy by ear, 2.3 GB lighter (2x LR broke vocals)
--attn flash                                                   # no --prefix-frames: under the whole-song window the track is its own history
--max-frames 9000 --drop-over-frames 9000                      # WHOLE SONG (2026-09-11, Rob): every track as one sequence; tracks over 360 s left out
--crop-mode structured --crop-start-frac 0.2 --crop-end-frac 0.15   # only matters for a track longer than the window (none, with the drop)
--crop-start-tiles 3 --crop-anchor song
--rank-dropout 0.1                                             # the mask IS the method under --hot-pizza; never 0
--steps 600 --save-every 100                                   # Balanced = 600, Fast = 300, Thorough = 900 (2026-09-11); stop on STEPS
--depth-loss-weight 1.0 --depth-loss-frames 128
# captions: per-track <stem>.mm3.txt from MOSS/Gemini, and ONLY those. No --caption-file:
# the shared caption killed endings (0/6 vs 4/6) and was removed on 2026-09-09.
# NO --reg-* prior by default (it was a workaround for the shared caption; costs likeness)
--trigger "<artist>" --trigger-prepend
--holdout 0.15 --eval-every 250 --eval-crop 500

Previews: every 50 steps (= every checkpoint), 40 s, control + baseline off, rendered on q8_0 at MLP 1.0 — the same dials generation uses.

Rob, 2026-09-06, on the stacked HOT-PiZZA recipe above (blind-hotpissa-recipe letter F): "sounds fantastic, this should be the default in-app." Cost on albumA: 4.7 s/step, 39 min for 500 steps, peak 25.9 GB — against the 2026-09-05 LoKr/Prodigy/exact/4096 line's 5.9 s, 49 min, 30.4 GB. Every lever was first tied individually in a blind set, then stacked and heard. The earlier LoKr line stays in the git history of this file.

2026-09-07, the safe stack. An overnight one-lever-at-a-time speed trial (overnight-speed/COSTS.md + blind-speed/RESULTS.md in the album A hub) found prefix 1024 (69 of 90) and crop 500 (67.5) tie the reference (68), while the prefill chunk 256 → 1024 cuts 17% off the step with an identical step-1 loss. Combined and heard blind (blind-confirm/RESULTS.md): safe stack 67 vs the crop-750 recipe 70.5, inside the ~6 noise floor, at 3.1 s/step and 26 min per 500 steps (was 4.5 s, 37 min). That is the recipe block above. What did NOT survive: doubling the LR (every 2x arm 3-4 under, and a 2x-LR/300-step stack produced a vocal-free plan on 1 of 6 songs across two seeds) and turning the acoustic loss off (lowest score). Held-out loss every 50 has the same shape in every arm — it follows the seed's crop order, not the adapter — so it is not a stopping signal.

Presets (Rob, 2026-09-07, MM3_LM_PRESETS in mm3Train.ts, a Recipe row in the Training Studio card; the route lays preset under the request's own fields, so {preset:'thorough'} alone trains Thorough):

(Superseded 2026-09-11: Fast 300 / Balanced 500 / Thorough 1000 at crop 750 with a 2048- or 4096-frame history. That regime's record — Fast's Simlish vocals of 2026-09-07, Balanced's 4/6 on album B — is in this file's git history.) All three share flash, AdamW, HOT-PiZZA r128, f16 factors and the acoustic loss; tracks longer than the window are left out (longTracks: 'exclude', engine --drop-over-frames). MM3_LM_DEFAULTS carries the Balanced values, so an empty API body trains Balanced. Prior preservation is OFF unless the request names a corpus (since 2026-09-09).

Rob, 2026-08-25, on the LoKr configuration this replaced: "the closest we've ever gotten to artist replication." Crop 750 = 30 s = ~3 s/step; he set it by ear after finding 15 s steps at crop 4272 unworkable and the shorter crop better, not merely faster.

At render: everything 1.0 for adapters trained WITH the acoustic loss (2026-08-25 onward) — previews and generation now default there. The old "MLP 0.63-0.75" dial was damage control for the timbre fault and applies only to PRE-FIX adapters (their sidecar recommendedScales override the defaults).

19.7 GB VRAM measured at crop 750 (the acoustic loss adds its frozen depth decoder, ~1.2 GB). The crop settings changed on 2026-08-24 and everything auditioned before that date was trained at --max-frames 128 --crop-mode random — five seconds per step. Treat pre-2026-08-24 ear results about the BASE and the OPTIMIZER as void (see below); the rank and MLP-dial findings stand, because they were measured against each other under the same broken crop.

Do this first or nothing works

Export RVQ codes. The trainer reads codes, not audio. A dataset that has only ever been captioned has none, and the failure is a bare "no usable samples".

ace-train mm3-codes --jsonl --dataset <dataset.json> \
  --rvq models/mm3/mm3-rvq-53kpooled-f32.gguf \
  --enc models/mm3/mm3-enc-f16.gguf \
  --out server/data/training/datasets/<slug>/mm3-codes

~1 minute for 12 whole tracks. Writes <out>/codes/<id>.codes.

Steps: 750–2000, and the loss will NOT tell you which

The single most important finding. Held-out cross-entropy bottoms out very early and then rises for the rest of the run — but the checkpoint that actually sounds right is 1–8x later than that minimum:

Never pick a checkpoint by held-out loss. It measures generalisation to unseen songs by the artist; the goal is a clone of the seen ones. Keep the eval on as a divergence alarm only. Save every 250 and audition the ladder.

Beyond ~2500 steps nothing improved in any album. 5000 steps is ~2x wasted time.

The table above is from the rank-64 sweep. Rank 256 was auditioned at ck1000 and ck2000 only; ck2000 won, and the finer rungs have not been walked at that rank. Do not assume the rank-64 optimum transfers.

The two axes: likeness vs coherence

They move in opposite directions with training length, and conflating them is why "more steps" felt ambiguous for so long:

• Likeness (does it sound like the band) rises with steps.
• Coherence (does the song hold together, are the vocals articulate) falls with steps at MLP 1.0. Symptom: jumbled or "stroke-like" vocals.

The MLP scale slider is a generation-time knob and recovers coherence at a late checkpoint without giving back likeness. It should therefore never cost a training run — the only training decision is the step count. Default the slider to 1.0 (4 of 6 albums preferred it); drop to 0.5 if vocals jumble.