Debugging Runtime

How the Node server supervises ace-server

• Spawn: startAceServer() (index.ts:158-316) launches config.aceServer.exe with --models, --host, --port (default 8085, server/src/config.ts:102) plus optional --adapters, --keep-loaded, --noise-profile, --draft-lm, --vae-chunk, --vae-overlap. The exe is auto-detected among engine/ace-server.exe, engine/build/Release/ace-server.exe, engine/build/ace-server.exe, engine/build/Debug/ace-server.exe (config.ts:46-52); override with ACESTEPCPP_EXE in .env.
• Respawn: child exit with non-zero code (and signal not SIGTERM/SIGINT) → respawn after 3 s. Crash limiter: 3 crashes within a rolling 30 s window → give up, set engineReady=false with bootStatus Engine crashed 3 times — check logs for missing DLLs (index.ts:152-156, 284-309). Crashes spaced >30 s apart reset the window, so slow-cycle crashes (e.g. crash-on-first-request) still loop forever — hence Golden rule 1.
• Clean shutdown: Ctrl-C → shutdown() uses taskkill /PID <child> /T /F (index.ts:549). POST /api/shutdown kills ace-server by port 8085 via netstat (server/src/routes/shutdown.ts:24-44), then Vite (:3000), then its own process tree. POST /api/restart writes a .restart-requested marker (shutdown.ts:161); in dev mode server/restart-loop.cmd loops on it and relaunches Node (LAUNCH.bat is the end-user prod equivalent).
• dev-rebuild.bat = POST /api/shutdown → wait up to 10 s for ace-server.exe to die (force-kill at 10 s, abort at 15 s) → engine\build.cmd. It does not restart the app — run dev.bat afterwards (agents never use LAUNCH.bat).
• Bootstrap gate: on portable Windows CUDA builds, the server first downloads cuBLAS DLLs from HuggingFace before the engine is usable; until then POST /api/generate returns 503 Engine not ready: <bootStatus>. No internet → CPU-only start with a banner in node_console.log.

Generation orchestration facts that change your diagnosis

• Single-slot FIFO queue — only one generation runs at a time (enqueueGeneration, generate.ts:1291-1352), because engine log parsing is a global untagged pub/sub.
• The auto-retry loop is dead code for generation failures: enqueueGeneration has a retry-once-with-fresh-seed loop (generate.ts:1301-1339), but runGeneration's top-level catch (generate.ts:1271-1283) swallows every generation error without rethrowing, so the retry never fires (the only escapable throw, translateParams at :191, happens before any engine submission). One UI-visible failure = exactly ONE engine attempt, and the seed in the gen log's single params block is directly trustworthy for repro. Zero [Retry] lines exist across 118 logged sessions. If retry is ever intended behavior, the swallowed rethrow is the server bug to fix.
• Stall watchdog: pollUntilDone (generate.ts:102-170) polls the engine every 500 ms; stage/progress unchanged for 120 s → cancel + Generation stalled. First-run TensorRT builds are special-cased ([TRT-WARN] lines mutate the stage string, generate.ts:756) so a 5–10 min TRT compile doesn't trip it.
• Transient poll errors are normal: [Generate] Poll error ... (will retry) (generate.ts:165) just means the busy single-threaded engine missed a 30 s poll window.
• Fine-grained engine phase is exposed at GET /api/generate/status/:id as ace_phase — one of queued, loading_text_enc, encoding_text, loading_cond_enc, encoding_cond, loading_dit, loading_adapter, adapter_precompute, dit_inference, loading_vae, vae_decode, encoding_output, done, failed, cancelled (aceClient.ts:165-180) — plus ace_phase_progress ("step N/M"). These fields are optional on the wire for older engine builds; absence is not an error.
• Job TTL: the in-memory job map prunes terminal jobs after 1 h (generate.ts:83-94). A 404 from /status/:id on an old job is TTL cleanup, not data loss — the song row and gen log persist.
• Post-processing failures are non-fatal: mastering / Spectral Lifter / PP-VAE / Whisper / LRC / cover-art failures log a WARNING and the job still succeeds with raw audio. A "succeeded" job can still have WARNINGs worth reading in its gen log.

Correlating a gen_<uuid> log with engine + node logs

There are two job-ID namespaces: the server UUID (the gen_<uuid> filename; SSE lines show only its first 8 chars as [Gen:xxxxxxxx], logger.ts:118) and the engine's own job id (ace_job_id in /status/:id; returned by the engine's /lm and /synth). The engine id is NOT in the gen filename.

1. Open gen_<uuid>_*.log; note the ISO timestamps of the LM/synth submission lines and the final GENERATION FAILED line.
2. ace_engine.log has no timestamps, so pivot through node_console.log: search for [Generate] Job <uuid> — the submission line logs ditModel/synth_model/seed/source (generate.ts:199) and the failure logs [Generate] Job <uuid> failed: (generate.ts:1280). The block of [ace-server] ... lines between your job's submit and fail lines IS that generation's engine output.
3. If you need the exact region of ace_engine.log (e.g. to see noise-filtered lines), grep it for a distinctive engine line found in step 2 (the last [DiT] Step N/M, a model-load line) and read forward from there.
4. Repro comes from the params JSON at the top of the gen log — it is the fully resolved request actually sent to the engine.

Dev-mode note: under dev.bat (tsx watch), every source-change auto-restart creates a new session folder. A burst of near-identical logs/ folders seconds apart means tsx was restarting, not that the app was crashing.

Key files

Institutional knowledge

• VALIDATED: Killing ace-server externally while Node runs causes a respawn/file-lock loop. The crash limiter (3 crashes / 30 s) caps fast crash loops, but slow-cycle crashes (>30 s apart) reset the window and loop indefinitely — CLAUDE.md's "infinite respawn" warning predates the limiter but the rule stands: rebuild only via dev-rebuild.bat.
• VALIDATED: gen logs are lost if Node dies mid-generation (flush-on-completion only). Failed/cancelled generations DO get their gen log.
• VALIDATED: ace_engine.log and node_console.log carry no per-line timestamps; only gen logs do. All time correlation goes through node_console.log line ordering.
• VALIDATED: Process exited with code 1, signal null after a shutdown line is a taskkill artifact, not a crash (dozens in history). Respawns themselves are routine (Restarting in 3 seconds... (crash N/3) appears across many healthy sessions).
• VALIDATED: [Safetensors] Cannot open ...adapter_model.safetensors is usually benign filename probing.
• VALIDATED: one UI failure = exactly one engine attempt. The retry loop in enqueueGeneration is unreachable for generation failures (runGeneration swallows its own errors, generate.ts:1271-1283); no [Retry] line has ever appeared in log history. The gen-log params block is the real, only attempt.
• UNVALIDATED: exact engine-side text accompanying status:'failed' for every failure class — hot-step-server.cpp has many FATAL/failed emit sites; the tables above list only those with real log occurrences. Linux/macOS shutdown paths exist in code (shutdown.ts:45+) but are unconfirmed by logs. gen_<id>_unknown.log files have a code path but none have been observed on disk.
• Draft-LM speculative decoding is disabled by default (config.ts:117-122) — --draft-lm is only passed if ACESTEPCPP_DRAFT_LM is set. Don't chase it as a crash suspect unless that env var is set.

Deeper reading

• reference.md (this folder) — copy-paste triage command pack, status JSON shapes, engine phase reference, watchdog timing table.
• CLAUDE.md — build/git rules that constrain what you may do while debugging.
• engine/docs/ARCHITECTURE.md — engine internals, request JSON, generation modes.
• docs/dev/plugins-authoring.md — Lua solver/scheduler plugins (a bad plugin can wedge DiT inference).
• docs/plans/ — internal investigation docs. Gitignored, local-only — may be absent on a fresh clone.