--- name: model-management description: Explains how model files, checkpoints, GGUF quantization, and the Model Manager work in HOT-Step CPP. Use when adding/converting/quantizing models, debugging "no GGUF models found" or missing-model/wrong-model failures, working on the model download service or Model Manager UI, or answering which component (LM/DiT/VAE/text-encoder) needs which file. --- # Model Management — Models, Checkpoints & Quantization All paths are repo-relative to the repo root (`d:\Ace-Step-Latest\hot-step-cpp`). All commands are Windows PowerShell (use `;` to chain, never `&&`). ## Terminology (read first) - **LM** — the language model (Qwen3-based) that turns caption+lyrics text into audio codes. Files: `acestep-5Hz-lm-{0.6B,1.7B,4B}-*.gguf`. - **DiT** — Diffusion Transformer, the denoising model that generates audio latents. Files: `acestep-v15-*.gguf`. One DiT GGUF is self-contained (also carries the condition encoder, FSQ tokenizer/detokenizer, `silence_latent`, `null_condition_emb`). - **Text encoder** — Qwen3 embedding model that encodes the caption for the DiT. Files: `Qwen3-Embedding-0.6B-*.gguf`. - **VAE** — decodes latents to 48 kHz audio (and encodes audio to latents for cover/repaint/extend). Files: `vae*.gguf` / `scragvae*.gguf`. - **PP-VAE** — optional post-processing VAE ("polish" re-encode pass). Files: `pp-vae-*.gguf`. - **GGUF** — the GGML binary weight format the C++ engine loads (mmap'd). Each GGUF declares its role in the header key `general.architecture`. - **Quant** — reduced-precision weight encoding (Q4_K_M, Q8_0, MXFP4, ...) to shrink VRAM/disk. Produced from BF16 GGUFs by `engine/tools/quantize.cpp`. - **Adapter** — LoRA/LoKr fine-tune delta (`.safetensors`), lives in `adapters/`, not `models/`. - **Model Manager** — the "Get More Models" modal in the UI + Node download service that fetches curated files from HuggingFace. ## When to use this skill - Installing, converting, or quantizing model files; deciding where a file must live. - Debugging: engine won't start, `/synth unavailable`, model missing from dropdowns, crash-loops, corrupt downloads. - Modifying the Model Manager (route `server/src/routes/modelManager.ts`, service `server/src/services/modelDownloadService.ts`, UI `ui/src/components/model-manager/`), or the catalogue `server/src/data/model-registry.json`. ## Golden rules 1. **Two independent registries exist — don't confuse them.** The Node-side curated catalogue (`server/src/data/model-registry.json`) controls what is *downloadable*; the C++ engine's startup scan (`engine/src/model-registry.h`) controls what is *usable*. A file can show "installed" in the Model Manager yet be invisible to generation dropdowns (and vice versa). WHY: they use different detection logic (filename presence vs GGUF-header architecture) and different directory depths. 2. **GGUF files must sit in the models root, not a subfolder.** The engine scans only root-level `.gguf` files (`engine/src/model-registry.h:234-267`), but the Node installed-check also scans one subdir level (`modelDownloadService.ts:177-208`). A GGUF in a subfolder = "installed" in the UI, dead to the engine. WHY: silent classic confusion — no error anywhere. 3. **Never hand-edit or hand-build a GGUF's tensor set for a recognized architecture.** A recognized-arch GGUF with a missing tensor kills the whole ace-server process: `gf_load_tensor()` prints `[GGUF] FATAL: tensor 'x' not found` and calls `exit(1)` (`engine/src/gguf-weights.h:164-168`). Node respawns it, so a bad model file mid-request looks like a random engine crash-loop. 4. **Quantize from BF16 sources only, and never quantize the VAE.** `quantize.exe` reads BF16 GGUF input; VAE-arch tensors and small/critical tensors (`silence_latent`, `scale_shift_table`, `null_condition_emb`, 1-D tensors, text-enc `embed_tokens`) are deliberately never quantized (`engine/tools/quantize.cpp:89-109`). WHY: quantizing these destroys audio quality or breaks generation outright. 5. **Rebuild rules apply here too:** any change to `engine/src/model-registry.h`, `model-store.h`, `gguf-weights.h`, etc. means rebuild via `dev-rebuild.bat` at repo root — never `engine/build.cmd` directly (you cannot reliably tell whether the app is running; Node auto-respawns ace-server, and killing it uncleanly causes an infinite respawn + file-lock loop). Never `cmake --clean-first` (20+ min CUDA recompile). 6. **Runtime DLLs do not go in `models/`.** Catalogue entries with `role: "runtime"` (cuBLAS, cudart, ONNX Runtime, cuDNN) install **next to `ace-server.exe`** (`modelDownloadService.ts:110-118`). WHY: missing DLLs there are the #1 cause of the "crashed 3 times within 30s — giving up" loop. 7. **A model that only exists on this machine does not exist.** Adding a model the app resolves at runtime is a three-part act: upload the weights to Hugging Face, add the entry to `server/src/data/model-registry.json`, and put it in a pack if a feature requires it. Do all three or none — a feature gated on a file that was never uploaded looks fine here and is dead for every user. Verify with `node server/scripts/check-release-prereqs.mjs`. WHY: MM3 training shipped in v1.3 demanding `mm3-rvq-*.gguf` and `mm3-enc-*.gguf` that had never left the dev box (#137, fixed 96d442fb). 8. **Model/adapters directory paths are restart-required config.** `ACESTEPCPP_MODELS` / `ACESTEPCPP_ADAPTERS` env vars override defaults `/models` and `/adapters` (`server/src/config.ts:53-54,100-101`); the engine gets them as spawn-time `--models`/`--adapters` flags. ## Directory layout (expected) ``` models/ # config.aceServer.models (default /models) acestep-v15-*.gguf # DiT (arch "acestep-dit"): base/sft/turbo/merge x BF16/Q8_0/Q6_K/Q5_K_M/Q4_K_M/MXFP4/NVFP4... acestep-5Hz-lm-{0.6B,1.7B,4B}-*.gguf # LM (arch "acestep-lm") Qwen3-Embedding-0.6B-*.gguf # Text encoder (arch "acestep-text-enc") vae-*.gguf, scragvae-*.gguf # VAE (arch "acestep-vae"); ScragVAE/Regrind = drop-in decoder variants pp-vae-*.gguf # PP-VAE (arch "pp-vae") vae-*.safetensors # safetensors VAE, classified by filename prefix only vae-*.onnx # ONNX VAE — DECODER-ONLY (see failure table) / # HF safetensors checkpoint dir: config.json + model.safetensors # (or model.safetensors.index.json for sharded) — classified by config.json content onnx/ # ONNX Runtime / TensorRT model dirs (config.aceServer.onnxDir) supersep/*.onnx # stem-separation nets (Cover/Stem Studio) whisper/ggml-*.bin # whisper.cpp models (config.ts:253-255) adapters/ # config.aceServer.adapters .safetensors # ComfyUI single-file LoRA (alpha baked in) /adapter_model.safetensors # PEFT directory format ``` ## Which component needs which model Verified in `engine/src/model-store.h:68-81` (ModelKind comments): | Component | Model file | Notes | |---|---|---| | LM (`MODEL_LM`) | `acestep-5Hz-lm-*.gguf` | ONE shared instance for generate + ace-understand — enforced by identical ModelKey (`model-store.h:17-22`) | | Text encoder (`MODEL_TEXT_ENC`) | `Qwen3-Embedding-*.gguf` | | | Cond-enc + DiT + FSQ tok/detok | the **same** `acestep-v15-*.gguf` | Self-contained; also holds `silence_latent` + `null_condition_emb` (`model-store.h:112-118`) | | VAE encode + decode | `vae*.gguf` (has `encoder.*` and `decoder.*`) | | | PP-VAE polish | `pp-vae-*.gguf` | Request flag `pp_vae_reencode` (`engine/src/request.h:146`); availability = `GET /api/models/pp-vae` scans for `pp-vae*.gguf` (`server/src/routes/models.ts:50-65`) | | ORT/TRT acceleration | `models/onnx/` subdirs | `MODEL_*_ORT` kinds | | SuperSep stems | `models/supersep/*.onnx` | + ONNX Runtime/cuDNN DLLs beside ace-server.exe | | Whisper transcription | `models/whisper/ggml-*.bin` | + `tools/whisper/whisper-cli.exe` (`config.ts:254-255`) | **Synth pipeline needs DiT + Text-Enc + VAE simultaneously.** Missing any one → `/synth unavailable` warning (server stays up LM-only if an LM exists), or exit 1 if no LM either (`engine/tools/hot-step-server.cpp:2600-2616` — the compiled server; the same block exists in the UNCOMPILED reference copy ace-server.cpp:1686-1711). Request-level selection: `synth_model`, `lm_model`, `vae` are **filenames** resolved against the engine's scanned registry; empty string = first matching entry (`engine/src/request.h:125-142`). ## Procedure: quantize a BF16 GGUF ```powershell # From repo root. Binary lives at engine\build\Release\quantize.exe .\engine\build\Release\quantize.exe # Example: .\engine\build\Release\quantize.exe models\acestep-v15-turbo-BF16.gguf models\acestep-v15-turbo-Q4_K_M.gguf Q4_K_M ``` - Valid TYPEs (case-insensitive, `quantize.cpp:8`): `Q2_K Q3_K_S Q3_K_M Q3_K_L Q4_K_S Q4_K_M Q5_K_S Q5_K_M Q6_K Q8_0 NVFP4 MXFP4`. IQ3/IQ4 quants seen on disk are **not** producible by this tool. - Mixed-precision policy mirrors llama-quantize: "important" tensors (`v_proj`, `down_proj`; L variants add `o_proj`) bumped one tier; `embed_tokens` always Q6_K (Q8_0 for Q8_0/NVFP4/MXFP4) (`quantize.cpp:40-54,74-86`). - Streaming write, low memory. Prints `Quantized N/M tensors` + compression ratio. - Output goes straight into `models\` root → picked up on next engine restart. ## Procedure: convert HF safetensors → BF16 GGUF (`engine/convert.py`) - **No CLI args.** Hardcoded: reads checkpoint dirs from `engine/checkpoints/`, writes GGUFs to `engine/models/` (`convert.py:14-16`). **Neither directory exists in this working tree** — create `engine\checkpoints\`, put the HF checkpoint dir inside, run it, then move the output GGUF to repo-root `models\`. - Classification is by checkpoint **directory name**: `acestep-5Hz-lm*` → LM, `acestep-v15*` → DiT, `Qwen3-Embedding*` → text-enc, and exactly `vae` → VAE (`convert.py:55-64`). Skips outputs that already exist. - Alternative: the engine loads safetensors checkpoint dirs **directly** (drop `/` with `config.json` + `model.safetensors` into `models/`) — conversion is optional. Sharded (`model.safetensors.index.json`) and diffusers (`diffusion_pytorch_model.safetensors`) layouts supported (`engine/src/weight-source.h`, `engine/src/model-registry.h:328-375`). ## Procedure: convert ComfyUI int8 DiT safetensors → Q8_0 GGUF (`engine/convert-comfy-int8.py`) For ComfyUI `comfy_quant` int8 DiT checkpoints (int8 `.weight` + F32 `.weight_scale` scalar or per-row + `.comfy_quant` JSON tensor per layer), including **ConvRot** files (`"convrot": true` + `convrot_groupsize`). Per-tensor and per-row int8 grids are exactly representable in Q8_0 (block scale = tensor/row scale), so weights are **repacked bit-faithfully** — no dequant/requant round trip. Needs a **donor GGUF** of the same architecture (any convert.py-produced `acestep-v15-*.gguf`) to supply `silence_latent` and the `acestep.*` config KVs, which ComfyUI files lack. Aborts on any tensor-shape mismatch vs the donor. ```powershell python engine\convert-comfy-int8.py models\-BF16.gguf models\-Q8_0.gguf --name ``` ConvRot handling: rotated `decoder.*` weights stay rotated and are recorded in GGUF KV `acestep.convrot_map` (`name:group;...`); the engine applies the matching group-wise Hadamard rotation to that linear's activations at inference (`dit.h` load + `dit-graph.h`/`dit-alignment-graph.h`, commit 182faef). Rotated encoder/tokenizer/detokenizer weights are dequantized + **unrotated** to BF16 offline (run once per generation — not worth graph wiring). `--no-runtime-rotation` builds an all-BF16 unrotated reference GGUF of the same quantized model, used for same-seed A/B validation of the engine rotation path. **Adapter merge mode is refused on ConvRot models** (deltas are unrotated); runtime adapter mode works (deltas consume raw activations). **ConvRot cardinal rule — any code reading ConvRot base weights for unrotated-space math must unrotate them first** (`convrot_transform_rows` in `engine/src/convrot.h`, fast radix-4, self-inverse). Violation signature: generation "succeeds" but output is garbled full-band noise. First instance: the runtime basin re-base nudged deltas with β·(S−T) using rotated T — fixed in `adapter_runtime_rebase` (commit 22820ae) by unrotating T for `acestep.convrot_map` tensors. Audit any future weight-reader (TRT export, distills, external merge scripts) against this. Producing ConvRot files from a local checkpoint: `pip install convert_to_quant` (needs torch+CUDA, triton-windows) then `ctq -i -o --int8 --scaling_mode row --convrot --dynamic_convrot --comfy_quant --save-quant-metadata` (~35 min for a 5B XL on an RTX 5090, learned rounding included). First applied 2026-07-15: hrktxz xl_sft_turbo (plain int8) → `acestep-v15-xl-sft-turbo-comfy-int8-Q8_0.gguf`; merge-base-sft-turbo-xl-thirds self-quantized with real ConvRot → `...-convrot-Q8_0.gguf` (+ `...-convrot-ref-BF16.gguf` reference). Numerical parity: rotated-path error 0.9% vs original F32 weights; skipping rotation → ~140% (i.e. rotation is load-bearing). ## Procedure: add a model manually 1. Copy the `.gguf` into `models\` **root** (not a subfolder — golden rule 2). 2. Restart the engine (`dev-rebuild.bat` restarts everything, or restart the app). The scan runs only at ace-server startup. 3. Check the newest `logs\\ace_engine.log` for `[Registry] -> DiT` (or LM/VAE/...). A `WARNING: skipping X (unknown architecture)` means the GGUF header lacks a recognized `general.architecture` (`acestep-lm|acestep-dit|acestep-text-enc|acestep-vae|pp-vae`, `model-registry.h:99-130`). 4. The file now appears in `GET /api/models` (Node proxies engine `/props`; buckets `lm`, `embedding`, `dit`, `vae` — `hot-step-server.cpp:2326-2329` — the compiled server, not the uncompiled ace-server.cpp). ## Procedure: publish a new model so users can get it Local conversion/quantization is only half the job. Until these steps are done the model does not exist for anyone but you. 1. **Upload the weights.** `huggingface_hub` is installed; the token lives in `~/.cache/huggingface/token` (account `scragnog`). **Ask the user before pushing to a public repo** — it is outward-facing and hard to walk back. ```python from huggingface_hub import HfApi HfApi().upload_file(path_or_fileobj='models/mm3/.gguf', path_in_repo='.gguf', repo_id='scragnog/', repo_type='model', commit_message='Add ') ``` 2. **Add the registry entry** to `server/src/data/model-registry.json` — `id`, `filename`, `role`, `subdir`, `displayName`, `quant`, exact `sizeBytes` (the downloader validates size ±5%), `repo`, `description`, `tags`, and a `companions` LICENSE entry if the weights carry one. The JSON round-trips exactly under `json.dumps(indent=2, ensure_ascii=False)`, so it can be edited programmatically without reformatting the whole file. 3. **Add it to a pack** if a feature needs it, and check the reverse: a feature that resolves files by *prefix scan* rather than by registry id (e.g. `resolveMm3TrainModels` takes the newest `mm3-rvq-*.gguf` on disk) will not be satisfied unless a published filename matches the prefix. 4. **Credit the author** in the HF model card if the weights are not ours, and keep the upstream licence. Community encoders and adapters are other people's work. 5. **Verify**: `node server/scripts/check-release-prereqs.mjs` — checks every entry resolves on HF at the claimed size, packs reference real ids, and runtime data files are packaged. Exit 1 = do not ship. ## Procedure: drive the Model Manager via API Routes in `server/src/routes/modelManager.ts`, mounted at `/api/model-manager`: ```powershell Invoke-RestMethod http://localhost:3000/api/model-manager/registry # catalogue + installed flags (dev app) Invoke-RestMethod -Method Post -Uri http://localhost:3000/api/model-manager/download -ContentType 'application/json' -Body '{"fileId":""}' # GET /downloads = SSE progress stream; POST /download//cancel | /resume; DELETE /files/ ``` Download mechanics: HuggingFace URL `https://huggingface.co/{repo}/resolve/main/{repoPath || filename}`, resume via HTTP Range + `.part` file, 3 attempts (0/2s/5s), validation before rename (size ±5%, `MZ` header for `.dll`, `GGUF` magic for `.gguf`) — `modelDownloadService.ts:352-473`. Details and data shapes: [reference.md](reference.md). ## Key files | Path | Role | |---|---| | `engine/src/model-registry.h` | Engine startup scan/classification of `--models` and `--adapters` dirs | | `engine/src/model-store.h` | Refcounted VRAM ownership; EVICT_STRICT (default) vs EVICT_NEVER (`--keep-loaded`); ModelKey caching incl. adapter extras | | `engine/src/gguf-weights.h` | mmap GGUF loader; truncation guard; FATAL exit on missing tensor | | `engine/src/safetensors.h`, `engine/src/weight-source.h` | safetensors parser + format-agnostic layer (GGUF/safetensors) | | `engine/tools/quantize.cpp` → `engine/build/Release/quantize.exe` | BF16 GGUF → K-quant/FP4 GGUF | | `engine/convert.py` | HF safetensors checkpoint dir → BF16 GGUF (hardcoded dirs) | | `engine/tools/ace-server.cpp` | Startup validation, `/props` endpoint | | `server/src/config.ts` | `aceServer.models/adapters/onnxDir`, `keepLoaded`, warm-on-startup, whisper paths | | `server/src/services/modelDownloadService.ts` | Download jobs, resume, validation, installed-check, variant filtering | | `server/src/routes/modelManager.ts` | `/api/model-manager/*` REST + SSE | | `server/src/routes/models.ts` | `/api/models` (proxies engine `/props`), `/api/models/pp-vae` | | `server/src/data/model-registry.json` | Curated catalogue: 152 files, 9 packs | | `server/src/index.ts` | ace-server spawn/respawn limiter (152-156, 284-308); first-launch CUDA DLL bootstrap (318-380) | | `ui/src/components/model-manager/` | Modal UI: `ModelManagerModal.tsx`, `ModelCatalogueTab.tsx` (7 tabs), `ModelRow.tsx`, `StarterPackCard.tsx`, `DownloadProgressBar.tsx`, `useModelRegistry.ts`, `useDownloadStream.ts` | ## Failure signatures | Symptom | Cause | Fix | |---|---|---| | `[Server] ERROR: no models found` + engine exit 1; Node retries 3x then gives up | Empty/wrong models dir (`ACESTEPCPP_MODELS`), or nothing classifiable — and no MM3 weights either. Since the issue-#118 fix, MM3 weights (`mm3-*.gguf` at root or in `mm3/`) keep the server alive MM3-only ("No ACE-Step models … continuing MM3-only") | Point at the right dir / install models; restart | | `[Registry] WARNING: skipping X (unknown architecture)` | GGUF header lacks a recognized `general.architecture` | Convert via `convert.py`, or it's not an ACE-Step GGUF | | `[Server] WARNING: /synth unavailable, missing: VAE` (etc.) | Partial install — synth needs DiT+Text-Enc+VAE together | Download the missing role (Model Manager quick-start pack) | | `[GGUF] FATAL: '' is truncated or corrupt ... file is only N bytes` | Interrupted download / prematurely renamed `.part` | Delete and re-download | | `[GGUF] FATAL: tensor 'x' not found` then process death | Recognized arch, wrong/incomplete tensor set — kills ace-server mid-request | Remove the bad GGUF | | Download "completes" then `Invalid GGUF header — got "