Engine Dev Loop

C++ Engine Development Loop

How to safely edit, rebuild, and smoke-test the C++ engine (ace-server.exe and friends) without triggering the respawn/file-lock loop that this repo learned about the hard way.

Context for readers with zero prior exposure: the app has three tiers. A Node/Express server (port 3001) spawns the C++ inference engine ace-server.exe (port 8085) as a managed child process and serves the React UI. Agents run it in dev mode with dev.bat detached and work through the Vite dev server at http://localhost:3000; LAUNCH.bat/:3001 is the end-user prod path. The engine runs the generation pipeline LM → DiT → VAE (LM = language model that plans the song; DiT = Diffusion Transformer that denoises latent audio; VAE = decoder that turns latents into 48 kHz stereo audio). All commands below are Windows PowerShell (use ; to chain, never && in old Windows PowerShell 5; this repo's convention is ;).

When to use this skill

• You edited (or are about to edit) anything under engine/src/ or engine/tools/.
• A build fails with LNK1104 / "cannot open ace-server.exe".
• ace-server keeps respawning every ~3 seconds, or the linker can't replace the exe.
• A header change seems to be ignored by the build (stale object files).
• A build after an upstream sync fails with the hotstep_sampler_linked_ sentinel linker error. (For the full sync + hook-verification process itself, use the upstream-sync skill — it owns that procedure.)
• You need to confirm a rebuild actually worked and the engine is healthy.

Golden rules (hard constraints — each prevented real damage)

1. Recompile immediately after editing ANY engine/src/ or engine/tools/ file — do not batch up edits and wait. (Verbatim instruction from the departing lead engineer.) WHY: uncompiled edits pile up, and when the eventual build breaks you can't tell which edit did it; the engine also silently keeps serving stale behavior in the meantime.

2. Rebuild via dev-rebuild.bat at repo root, NEVER engine/build.cmd directly. WHY: you cannot reliably tell whether the app is running (a Node server or orphaned tsx-watch tree may be alive in another terminal). Node auto-respawns ace-server on abnormal exit (server/src/index.ts:284-309, retry after 3 s): if the linker deletes/replaces the exe while Node is alive, the respawned process re-locks ace-server.exe and the link fails with a file-lock error — an infinite respawn + file-lock loop. dev-rebuild.bat shuts everything down first, and is safe even when the app is already stopped (its shutdown request is a suppressed no-op, dev-rebuild.bat:10) — so there is no scenario where calling build.cmd directly is the right move.

3. NEVER cmake --build . --clean-first. WHY: CUDA kernel recompilation takes 20+ minutes (documented in CLAUDE.md; not independently timed). Only justified if the GGML/CUDA layer itself changed. For stale-object problems use the surgical recovery in Procedure 2 instead.

4. Don't switch between build scripts casually. WHY: buildcuda.cmd, buildvulkan.cmd, and buildall.cmd all run cmake .. unconditionally against the shared engine/build/ directory with different flags — reconfiguring the cache can invalidate large parts of the incremental build (CUDA-recompile territory). build.cmd only configures when CMakeCache.txt is absent (engine/build.cmd:140-148). Stick to build.cmd (via dev-rebuild.bat) unless you deliberately need a different backend set.

5. dev-rebuild.bat does NOT check the build result and does NOT restart the app. It prints "Done. Start the app with LAUNCH.bat" even if MSBuild failed. Read the build output yourself, then relaunch with dev.bat (detached) and check http://localhost:3000 — the script's message names LAUNCH.bat, which is the end-user prod path, not the one agents use.

Procedure 1 — Standard rebuild after a C++ edit

# 1. Rebuild (shuts down app gracefully, waits for ace-server to die, builds):
#    ONLY works from an interactive console. From an agent tool call or any piped/
#    redirected shell it silently does nothing and exits 0 — see the warning below,
#    and run the three phases manually instead.
& "D:\Ace-Step-Latest\hot-step-cpp\dev-rebuild.bat"

# 2. VERIFY the build yourself — dev-rebuild prints "Done" regardless of result, and the
#    wrapper ALWAYS exits 0 even when MSBuild failed (build.cmd's trailing `cd ..` resets
#    errorlevel) — $LASTEXITCODE is useless here. The ONLY valid checks are:
#    (a) no "error C..." or "LNK..." lines in the scrolled output, and
#    (b) a fresh LastWriteTime on the exe:
Get-Item "D:\Ace-Step-Latest\hot-step-cpp\engine\build\Release\ace-server.exe" | Select-Object LastWriteTime

# 3. Relaunch the app (dev-rebuild does NOT do this) — detached, then check http://localhost:3000:
& "D:\Ace-Step-Latest\hot-step-cpp\dev.bat"       # dev mode (Vite :3000 HMR + tsx watch :3001)
# LAUNCH.bat is the end-user prod launcher — agents don't run it

Note: dev-rebuild's shutdown also kills the Vite dev server on port 3000 (server/src/routes/shutdown.ts:72-98), so in dev mode you must restart with dev.bat, not just relaunch the engine — despite the script's final message naming only LAUNCH.bat.

What dev-rebuild.bat actually does (dev-rebuild.bat:1-35)

1. curl -s -X POST http://localhost:3001/api/shutdown — asks the Node server to shut everything down gracefully (errors suppressed, so it's safe when the app isn't running). The shutdown route (server/src/routes/shutdown.ts:134-153) kills, in order: ace-server (found via netstat on port 8085, then taskkill /PID <pid> /T /F), Vite (port 3000), and finally the Node server's own process tree via a detached killer — so nothing survives to respawn ace-server.
2. Polls tasklist for ace-server.exe once per second. At 10 retries it force-kills (taskkill /F /IM ace-server.exe /T); at 15 retries it aborts with FATAL: could not stop ace-server and exit code 1.
3. Once ace-server is gone: call engine\build.cmd.
4. Prints "Done." — no build-result check, no restart.

Why the graceful route matters: the kill is initiated by the server that owns the respawn logic, and then that server kills itself. Killing ace-server externally while Node lives triggers the respawn handler instead (server/src/index.ts:284-309; the crash limiter at index.ts:152-156 only gives up after 3 crashes within a 30-second window).

There is also POST /api/restart (shutdown.ts:156-190), which writes a .restart-requested marker that LAUNCH.bat / server/restart-loop.cmd loop on. That is for in-app restarts only — it does not rebuild anything.

⚠ dev-rebuild.bat does NOTHING from a non-interactive shell (agents: read this)

dev-rebuild.bat uses timeout /t 1 /nobreak in its wait loop (steps 2 above). timeout requires a real console: with stdin redirected to the null device — which is every agent tool call, every piped invocation, every CI-style run — it fails immediately with ERROR: Input redirection is not supported. The loop then falls straight through, so the script skips the shutdown AND the build, prints nothing, and exits 0.

This is maximally deceptive: exit 0, no error text, and (because it never shut anything down) the app is still running afterwards, so nothing looks wrong. Observed 2026-08-21 — the "build" took 2 seconds and ace-server.exe kept a five-hour-old timestamp.

Run the three phases yourself instead. This is the same sequence the script performs, and it is safe for the reason Procedure 1 explains — Node initiates the kill, so no respawn:

# 1. graceful shutdown (Node kills ace-server, Vite, then itself)
try { Invoke-RestMethod -Method Post -Uri "http://localhost:3000/api/shutdown" -TimeoutSec 10 | Out-Null } catch {}

# 2. wait for the binary lock to actually clear — never skip this, it is what
#    prevents LNK1104 and the respawn loop
$n=0; while ((Get-Process ace-server -ErrorAction SilentlyContinue) -and $n -lt 20) { Start-Sleep -Seconds 1; $n++ }
if (Get-Process ace-server -ErrorAction SilentlyContinue) { Stop-Process -Name ace-server -Force; Start-Sleep -Seconds 2 }

# 3. build, then VERIFY (see Procedure 1 step 2 — the exit code is meaningless)
& cmd.exe /c "engine\build.cmd"
Get-Item "engine\build\Release\ace-server.exe" | Select-Object LastWriteTime

Grep the build output for ace-server.vcxproj -> — that line is printed only on a successful link, and is a positive signal rather than the absence of a negative one.

Errors in a header you never edited = broken literal in the file included before it

A single unterminated string literal or comment makes the parser run on into the next header, so MSVC reports the damage at the wrong file entirely. Symptoms are surreal: (double) some_func() / 1048576.0 failing with "cannot convert from size_t (__cdecl *)(void) to double" plus "Context does not allow for disambiguation of overloaded function" — a call the compiler cannot see the () on, in code that is provably correct and unmodified.

Do not debug the reported file. Check the previous one in the include chain, at your edit.

The usual cause when an agent made the edit: a \n inside a C string written as a real newline. Shell heredocs and some scripting layers collapse \\n → \n → an actual line break, splitting the literal across two lines. Verify after any scripted C++ edit:

sed -n '<start>,<end>p' <file> | cat -A     # look for a line ending mid-string

Prefer the Edit tool over heredocs for C++ containing escapes. (Same class of bug bit a Windows path in a Markdown file the same day: \\2026 → octal escape → mangled text.)

How long builds take

• Incremental build.cmd after a single .cpp/.h edit: minutes-scale — configure is skipped when engine/build/CMakeCache.txt exists (unverified exact duration; no timing data in repo).
• Full clean CUDA build / --clean-first: 20+ minutes (CLAUDE.md claim). Avoid.
• First generation after a rebuild is slow: LoKr adapter precompute takes ~17 s per DiT+adapter combo unless --keep-loaded is set (server/src/index.ts:177-180). That slowness is normal, not a regression.

Procedure 2 — Stale-.obj recovery (surgical, NOT clean)

Use when a header change isn't picked up, or you see weird ODR/linker behavior. This recompiles only the core lib, leaving the expensive GGML/CUDA objects intact:

Remove-Item -Recurse -Force "D:\Ace-Step-Latest\hot-step-cpp\engine\build\acestep-core.dir"
Remove-Item -Force "D:\Ace-Step-Latest\hot-step-cpp\engine\build\Release\acestep-core.lib"
& "D:\Ace-Step-Latest\hot-step-cpp\dev-rebuild.bat"

(Both paths verified to exist in the current tree.)

Procedure 3 — After an upstream sync: verify fork hooks

The engine is a patched fork of acestep.cpp. Three upstream files carry HOT-Step #include hooks that break if a sync overwrites them:

powershell -File "D:\Ace-Step-Latest\hot-step-cpp\engine\verify-hooks.ps1"   # exit 0 = all hooks intact

It checks 5 things (engine/verify-hooks.ps1:15-68): the three hooks above, hot-step-server.cpp → hot-step-params.h, and the linker sentinel below.

Linker sentinel (silent-hook tripwire): engine/src/hot-step-sampler.h:1311-1315 defines hotstep_sampler_linked_ (selectany/weak, external linkage); engine/tools/hot-step-server.cpp:51-52 references it via extern. If a sync reverts pipeline-synth-ops.cpp to dit-sampler.h, the symbol vanishes and the link fails with unresolved external symbol hotstep_sampler_linked_. That error means "re-hook hot-step-sampler.h", not a real linker problem. It converts the otherwise-silent hook loss into a build failure.

Procedure 4 — Smoke-testing after a rebuild

Relaunch with dev.bat (detached), confirm the app answers at http://localhost:3000/api/health, then probe the engine directly. ace-server takes a few seconds to come up after dev.bat returns (the script just starts detached windows) — retry /health for up to ~30 s; connection refused immediately after launch is normal, not a build failure. (Use curl.exe to be safe — in legacy Windows PowerShell 5.1 bare curl is an alias for Invoke-WebRequest; in PowerShell 7 it resolves to the real curl.exe.)

# 1. Engine alive? Returns {"status":"ok"}  (engine/tools/hot-step-server.cpp:2709-2711)
curl.exe -s http://localhost:8085/health

# 2. Models discovered + config sane?
curl.exe -s http://localhost:8085/props

# 3. Lua plugin registry loaded? (solvers/schedulers/guidance — the thing silent hook loss kills)
curl.exe -s http://localhost:8085/plugins

# 4. CUDA context up? Returns used/total/free VRAM MB; HTTP 500 if CUDA is broken
#    (hot-step-server.cpp:2722-2741)
curl.exe -s http://localhost:8085/vram

Full endpoint surface, if you need deeper probes (hot-step-server.cpp:2704-2852): POST /lm /synth /understand /vae /warm /job /models/unload /pp-vae-reencode, GET /health /props /logs /jobs /job /plugins /vram /models/loaded, plus /supersep/* and /spectral-lifter routes.

Real smoke test: kick a short generation from the UI — ask the human user to do this and report back; do NOT use a browser agent for visual verification (project rule). Watch the newest session folder under logs/ (folders are named YYYY-MM-DD_HH-MM-SS, so name-sorted = time-sorted): ace_engine.log (C++ stdout/stderr), node_console.log, and generations/gen_<uuid>_<task>.log. Never delete generated test audio or other generation outputs — even ones you predict are bad. The user verifies results by ear; leave all artifacts in place.

Standalone engine (without the Node server): engine/server.cmd runs ace-server.exe --host 0.0.0.0 --port 8085 --models .\models --adapters .\adapters --max-batch 1. Caveat: its --models .\models is relative to engine/, but the real model directory is at repo root (models/ — the Node server passes the absolute path at server/src/index.ts:166-170). Treat server.cmd as a template, not a turnkey smoke test.