Adapter System

Adapter (LoRA/LoKR) System

The deepest institutional-knowledge subsystem in HOT-Step CPP. Every rule below was paid for with a shipped bug or a failed research direction. Deeper detail (merge/runtime internals, cache-key construction, cross-arch research history) lives in reference.md in this folder.

Glossary (read first — no prior context assumed)

• DiT — Diffusion Transformer, the C++ model that denoises audio latents (32 layers on the XL base). Lives in engine/src/dit.h / dit-graph.h.
• LoRA — low-rank adapter: two small matrices A, B whose product B@A is a delta added to a base weight. LoKr — LyCORIS Kronecker-product variant (.lokr_w1/.lokr_w2 tensors). DoRA — adds a per-row multiplicative rescale; two on-disk namings, both supported in MERGE MODE ONLY: LyCORIS dora_scale (LoKr path) and PEFT lora_magnitude_vector (LoRA path, since 2026-07-19). With DoRA the delta carries only alpha/rank; user strength × group scale blends the decompose factor (s = u·(m/‖W+Δ‖) + (1−u)), so strength 0 does NOT fully disable a DoRA adapter.
• Delta (Δ) — the full-size weight difference an adapter contributes: W_adapted = W_base + scale·Δ.
• Merge mode — deltas are baked into base weights on load, before GPU upload (adapter-merge.h). Runtime mode — deltas live as separate GPU tensors, applied each sampling step as y = W@x + Δ@x (adapter-runtime.h + dit-graph.h).
• aceReq — the JSON generation request (AceRequest) the Node server sends to the C++ engine's /synth endpoint.
• Sideband — extra HOT-Step-only request fields the engine parses separately from AceRequest (struct ServerFields, engine/tools/hot-step-server.cpp:569) and stores in the global g_hotstep_params (engine/src/hot-step-params.h:205).
• ModelKey — the cache key under which a loaded DiT is stored in model-store.{h,cpp}. Two requests with the same key reuse the same loaded (adapter-merged) model.
• Basin re-base — before merging, nudge the loaded base T toward the adapter's original training base S: base ← base + β·(S − base) (see Institutional knowledge).

When to use this skill

• Adding/changing anything in engine/src/adapter-*.h, the adapter parts of dit.h, dit-graph.h, hot-step-sampler.h, or hot-step-server.cpp.
• Working on multi-adapter stacking, Sum/Blend, per-section [Header]{k=v} lyric directives, adapter VRAM quantization, or trigger words.
• Debugging: adapter inaudible, wrong strength, wrong flavor after toggling settings, CUDA errors during sampling with section adapters, slow adapter loads.
• Anything involving converting adapters between DiT bases or architectures.

Golden rules (each one reverses a shipped bug or dead research month)

1. GGML clobbers input-tensor buffers after every compute. Any constant GPU input — especially the per-frame LoRA section masks — must be re-uploaded every sampling step, in all compute paths, and again after any mid-sampling graph rebuild (CFG-cutoff). Upload-once produces the "nil adapter" bug: masks read ~0, output is pure base model. Existing re-upload sites: engine/src/hot-step-sampler.h:605, :659, :716, :940, :1097. Any NEW graph input must join every one of those sites. (Commits a6db135, 2052bc3.)
2. Cache-key discipline (commit 168dcb5 — the law). Every input that changes the merged weights or loaded deltas MUST be part of ModelKey, and a failed adapter load must NEVER be cached as a success. Key construction: engine/src/pipeline-synth.cpp:168-199; hash/eq: engine/src/model-store.cpp:48-94; failure path: engine/src/dit.h:648-658. If you add any parameter that affects merged/loaded weights, add it to the key AND the hash AND the equality — all three.
3. Basin re-base applies ONCE PER STACK, first adapter only (merge: engine/src/dit.h:565-586; runtime: adapter_runtime_rebase in adapter-runtime.h, invoked after the first adapter stages). Applying it per adapter resets the running base on every merge (at β=1 only the last adapter survives) — in runtime mode it would duplicate the base correction N×. Works in BOTH modes since 7aac3c2 (runtime folds β·(S−T) into the staged delta sum — matmul distributivity makes it output-identical to merge). NOT supported on the per-section masking path (masked per-frame deltas can't carry an always-on base correction; engine warns + skips, and the cache key mirrors that skip in pipeline-synth.cpp).
4. Do NOT re-attempt regional self-attention isolation. Implemented in commit 0f3bf6d, reverted in ee041e1: penalising cross-section self-attn logits broke musical continuity (multi-second silence gaps, degenerate later sections). Self-attention carries BOTH adapter identity AND musical coherence — you cannot suppress one without the other. The adapter_section_isolation param is still plumbed end-to-end but dormant (engine ignores it). See reference.md §Reverted for the structurally safer untried alternative.
5. Weight similarity does NOT imply adapter transferability. LoKR cross-base conversion to non-turbo XL bases fails even though the bases are ~99% weight-identical — the root cause is basin-sensitivity, not weight drift (Institutional knowledge below).
6. Never use real Q4_K for runtime delta storage. Its per-superblock optimizer stalls the load for minutes across ~360 tensors. The "q4_k" string is now aliased to Q4_0 (engine/src/adapter-runtime.h:136-141). Q4_0/Q8_0 require ne0 % 32 == 0; anything else falls back to BF16.
7. Never assume runtime deltas are BF16. They may be Q4_0/Q8_0. mul_mat dequantizes transparently, but any code that reads delta bytes directly assuming BF16 crashes (a delta-L2 diagnostic did exactly this — removed in 46603bf).
8. LM echo sideband gotcha: never whitelist synth-request fields. Sideband/ServerFields-only params (adapter_runtime_quant, adapter_section_align_at, rebase_*, …) do NOT survive the engine /lm round trip. server/src/routes/generate.ts rebuilds synth requests as {...aceReq, ...LM-generated-fields-only} (lines 238 and 320). A field whitelist here once made the Adapter VRAM and Alignment Timing knobs silently dead on the default path (fixed in 8ea519b/168dcb5).
9. Edit the right files. The compiled server is engine/tools/hot-step-server.cpp — engine/tools/ace-server.cpp is upstream reference code, not compiled. The live sampler is engine/src/hot-step-sampler.h — dit-sampler.h is dead upstream code. pipeline-synth-ops.cpp:9 includes hot-step-sampler.h; losing that include on an upstream sync is silent (compiles, all solvers/schedulers/guidance go dead). After any sync run engine/verify-hooks.ps1.
10. Build & git rules. After ANY engine edit: .\dev-rebuild.bat from repo root, immediately — NEVER engine/build.cmd directly (you cannot reliably tell whether the app is running; Node auto-respawns ace-server → infinite respawn + file-lock loop), never cmake --clean-first (20+ min CUDA recompile). TypeScript changes: npx tsc --noEmit, not npm run build. Git: stage explicit paths only — never git add -A or add -f (re-adds gitignored models/checkpoints/local docs); commit locally often; push only with explicit user approval.

How an adapter reaches the GPU (request flow)

1. UI → Node: server/src/services/generation/translateParams.ts:83-143 maps params.loraStack → req.adapters [{name, scale}] (supersedes the single req.adapter), plus adapter_mode, adapter_runtime_quant, per-group scales, basin re-base fields (through :128), and trigger words for all stacked adapters (:131-143). Per-section directives in lyrics are parsed here (see below).
2. Node → engine /synth: the engine parses the JSON twice — once as AceRequest (engine/src/request.{h,cpp}), once as the ServerFields sideband (hot-step-server.cpp:569+).
3. Worker resolves and stages: adapter names → paths via the registry, with an absolute-path fallback for unregistered adapters (hot-step-server.cpp:1079). Fills g_hotstep_params.adapters and all sideband fields BEFORE ace_synth_load — the merge reads them during load (hot-step-server.cpp:1129 comment).
4. Cache lookup: pipeline-synth.cpp:168-199 builds the DiT ModelKey; model-store returns the cached DiT or loads via dit_ggml_load.
5. Load (dit.h): merge mode → sequential adapter_merge per stacked adapter (deltas accumulate: W ← W + s1·Δ1 + s2·Δ2 + …); runtime mode → skip QKV/gate_up fusion (dit.h:420 — runtime deltas need individual projections), then after GPU alloc load runtime deltas (dit.h:608+).
6. Sampling (hot-step-sampler.h): deltas applied per step; section masks (if active) built and re-uploaded every step.

Per-section (regional) adapter masking — shipped, P1+P2

Different adapters active in different song sections, driven by lyric directives like [Chorus]{albumB2=1; albumN=0} (keys = adapter filename stem, or positional #2/2, 1-based).

• Gate: activates only with directives in lyrics AND ≥2 stacked adapters (translateParams.ts:104). Forces adapter_mode=runtime (server :114; engine double-checks, hot-step-server.cpp:1157-1159). When the gate is unmet, directives are still stripped from lyrics (stripAdapterDirectives) so they never reach the LM/encoder as garbage tokens. The UI stack controls sit behind Advanced mode (advancedAdapters in ui/src/stores/globalParamsStore.ts).
• Parser (server/src/services/generation/adapterSections.ts): Sum/Blend applied per section (blend budget default 0.75); directive-less sections use stack default scales; {…} with no key=val pair is treated as lyric text and left alone; all-typo keys warn and fall back to defaults; weights clamp ≥ 0.
• Engine load (dit.h:618-640): each adapter goes into its OWN DiTLoRA in m->loras[], UNIT-scaled (1.0) — the per-frame mask carries the effective scale; loading with the stack scale would double-scale (dit.h:628-631). This costs N× VRAM vs the summed path, hence Q4_0 delta quantization (~¼ size).
• Graph (dit-graph.h): one [1,S,1] F32 mask tensor per adapter, shared across layers. Frame-indexed projections get Σᵢ (Δᵢ@x) ⊙ maskᵢ; token/global projections (cross-attn k/v, cond_embed) get the adapter's scalar mean section weight instead — a frame mask is the wrong axis there (dit-graph.h:430-433). Known limitation: text conditioning is therefore a constant blend across the song.
• Sampler (hot-step-sampler.h): P1 builds an initial frame→section map proportional to section character counts (:306-323), with a ~0.5 s triangular crossfade between sections (ce57675). P2: at adapter_section_align_at fraction of steps (default 0.55, UI "Alignment Timing"), estimate x0, run alignment extraction on a private scheduler, map frames → dominant lyric token → section via the header-anchored token map (pipeline-synth-ops.cpp:1419-1495), median-smooth, rebuild masks (:338-385). Falls back to the P1 map on failure.

Timestep-dependent adapter gating (interval experts / MoE) — shipped 2026-07-20

Per-adapter gain curves g(t) over flow-matching t (1=noise → 0=clean): each stacked adapter's per-frame mask is multiplied by its interpolated g(t) at EVERY model evaluation (upload_lora_masks(t_val) in hot-step-sampler.h — the single helper that replaced all five raw mask-upload sites). Different adapters can own different slices of the trajectory ("structure" expert early / "timbre" expert late — TD-LoRA / TimeStep Master scalar mixing). Key facts:

• Rides the per-section machinery: a curve-carrying stack with no lyric directives gets a synthetic single whole-song section from translateParams.ts (+ forced runtime mode); allowed at stack size 1 (gates in dit.h and hot-step-server.cpp accept >=2 adapters OR gains active). Composes with real per-section directives (mask × gain).
• Curves are NOT in the ModelKey — deliberate: they change per-step mask uploads only, never loaded weights, so mixing retunes with zero model reload (the |sect marker still applies via the synthetic section). Do not "fix" this by adding them to the key.
• Direct-graph caveat: masks are pinned resident on direct graphs and normally skip per-step re-upload; with gains active upload_lora_masks runs on EVERY evaluation regardless (!dit_graph.direct || lora_gains_active). Host masks stay UNSCALED — scaling happens into a scratch buffer at upload so P2 rebuilds and repeated evals never compound gains.
• Wire format: adapters[].gain_curve = uniform samples of g(x) over [0,1] (33 from the UI) + gain_domain: "steps" (x = remaining-steps fraction, engine maps each eval's t to its nearest schedule index) or "t" (x = raw flow-matching t). UI windows are ALWAYS "steps"; trained-expert / router curves must be "t" (must match the training axis). This split exists because shifted schedules are wildly nonuniform in t — shift-3 @ 20 steps spends 17 steps above t=0.5, so a t-domain "0–50%" window starved the late adapter to 3 tail steps ("all I hear is the first adapter"). Window edges are smoothstep ramps CENTERED on the bound so adjacent windows crossfade summing to 1. Every [DiT] Step log line appends gains=[..] when gating is active — check there first when a mix sounds one-sided.
• Training side: Side-Step --timestep-window-min/max (rejection-resampled logit-normal; discrete mode filters the 8-step schedule) trains matching interval experts. T-LoRA: the high-noise expert overfits fastest — lower rank.
• Inherits per-section constraints: runtime-only (no DoRA rescale), N× VRAM (Q4_0 knob), no basin re-base. runtime_lowrank is CLOBBERED to plain runtime (server + engine) — windows currently require the full-delta path; masked low-rank apply is untried future work.
• VRAM trap (shipped bug, fixed same day): windows force runtime mode server-side, but getGlobalParams() gated adapterRuntimeQuant on the selected mode — a Merge-mode user with windows got full BF16 deltas (~8 GB/adapter on XL, 32 GB with 2 + reload churn) and their Merge-VRAM "Low" setting meant nothing. Quant now flows (and the Adapter VRAM knob shows) whenever windows are active. Any future knob that matters on a forced path must gate on the EFFECTIVE mode, not the selected one.