Validating Changes

Validating Changes to the Required Standard

HOT-Step CPP is a 3-tier app: a C++17/CUDA/GGML engine (engine/, spawned as ace-server.exe on port 8085), a Node/TypeScript/Express server (server/src/, port 3001), and a React 19 UI (ui/src/, Vite dev server on port 3000). Agents run the app with dev.bat, detached, and verify through http://localhost:3000; :3001 is the end-user prod address. A change is NOT done until it clears the bar for every tier it touches. This skill defines those bars, how to run a headless end-to-end smoke generation, and how to report results honestly.

All commands are Windows PowerShell (; as separator, never &&). Deep API/log detail lives in reference.md in this folder.

When to use this skill

• You just finished a code change and need to decide whether it is "done".
• You need to verify a C++ engine change, a server/UI TypeScript change, or a Lua plugin change actually works.
• You need to run an end-to-end generation without the UI (headless smoke test).
• You are about to report validation results to the user.

Golden rules (hard constraints — each prevents expensive damage)

1. ONLY the human can judge audio quality, by ear. Never claim audio "sounds fine" or "is broken" from metrics, file size, duration, RMS, spectrograms, log silence, or the pipeline's quality-evaluator scores (stored in the quality_scores DB column — advisory telemetry only). Say: "generation completed mechanically; audio at <path> awaits your ear." WHY: metrics have repeatedly failed to predict what the user hears; false verdicts destroy trust and mislead debugging.

2. PRESERVE EXPERIMENT ARTIFACTS. Never delete generated test audio, latents, or other outputs because you predict they are bad. The human verifies by ear and has lost artifacts to premature cleanup before. WHY: a generation costs GPU time and a seed; a deleted artifact may be unreproducible evidence.

3. Report outcomes faithfully. Failed tests are reported as FAILED with the actual output/error pasted. Skipped steps are reported as SKIPPED with the reason. Never write hedged "should work" / "probably fine". Distinguish what you ran and saw vs what you inferred vs what you did not verify. WHY: the user makes release and debugging decisions off your report.

4. Rebuild C++ only via dev-rebuild.bat at repo root — never engine\build.cmd directly, under any circumstances. WHY: you cannot reliably tell whether the app is running; the Node server auto-respawns ace-server 3 s after a non-clean exit (server/src/index.ts:284-308), and the respawned exe file-locks the linker output — infinite loop. dev-rebuild.bat is safe even when the app is stopped (its shutdown request is a suppressed no-op).

5. Never push a v* tag to "test" a build. WHY: any pushed v* tag triggers the full multi-platform Release workflow and drafts a GitHub Release. Throwaway compile checks use a -CI-Test suffix; any push still requires explicit user approval.

6. Never cmake --build . --clean-first unless the GGML/CUDA layer itself changed. WHY: CUDA kernel recompilation takes 20+ minutes. For stale-.obj symptoms delete only engine/build/acestep-core.dir/ and engine/build/Release/acestep-core.lib.

7. No npm run build during dev. Type-check with tsc only (§ below). WHY: builds are slow and unnecessary until the user tests a prod (LAUNCH.bat) flow.

8. Never visually verify UI with a browser agent — ask the human; they provide screenshots. Browser/Invoke-RestMethod IS fine for non-visual API checks. WHY: the browser agent is too slow/unreliable in this environment.

9. A feature is not done until a USER can run it. Anything the app resolves at runtime — model weights, a data file, a binary — must be reachable by someone who downloaded a release, not just present on this machine. Run node server/scripts/check-release-prereqs.mjs before any push (§ Tier 6). WHY: this has shipped broken twice. MM3 training asked for two GGUFs that existed only on the dev box (#137), and the MM3 caption corpus was committed but never copied into the archives (#139). Both passed tsc, both built green, both were dead on arrival for every user.

10. Don't clobber the user's generation queue. Jobs run one at a time on a single GPU. Check GET /api/generate/queue before submitting a smoke generation, and never call POST /api/generate/reset-queue (force-drains everything) without asking. WHY: it cancels the user's in-flight work.

Tier 1 — TypeScript (server and UI)

Bar: type-check clean. Zero errors.

Node 20 to 24 LTS, 24 recommended (engines: >=20.0.0 <25.0.0 in server/package.json). If tsc/tsx/npm fails oddly, check node --version BEFORE blaming the change — a wrong Node version produces false validation verdicts.

cd D:\Ace-Step-Latest\hot-step-cpp\server; npx tsc --noEmit
cd D:\Ace-Step-Latest\hot-step-cpp\ui; npx tsc -b

• server/package.json defines "typecheck": "tsc --noEmit". The UI uses project references (ui/tsconfig.json → tsconfig.app.json + tsconfig.node.json, both noEmit-safe), so npx tsc -b type-checks without emitting JS.
• In dev mode (dev.bat) the server runs under tsx watch (server/package.json "dev": "tsx watch src/index.ts") — it auto-restarts on save. Server-side bar = passing tsc plus a clean restart visible in the newest logs/<session>/node_console.log (no stack trace after the restart).

Tier 2 — C++ engine

Bar: rebuild compiles AND the engine starts clean after relaunch (log check).

1. Recompile immediately after editing any engine/src/ or engine/tools/ file — don't wait to be asked:

   D:\Ace-Step-Latest\hot-step-cpp\dev-rebuild.bat

   What it does (verified in the script): POSTs http://localhost:3001/api/shutdown (kills ace-server, Vite, and the Node server cleanly), waits up to 10 s for ace-server.exe to exit, force-kills at 10 s, aborts at 15 s, then runs engine\build.cmd.

2. The rebuild does NOT restart the app. Relaunch with dev.bat (detached) and check http://localhost:3000 before any runtime verification. LAUNCH.bat is the end-user prod launcher — agents don't run it.

3. Clean-start check — read the newest logs/<session>/ace_engine.log and confirm:

   • model registry scan ran ([Server] Scanning models in ... / [Registry] ... lines),
   • [Server] Listening on 127.0.0.1:8085 appears,
   • no crash/exception before the Listening line,
   • node_console.log has no [ace-server] Restarting in 3 seconds... (that exact string = crash respawn, server/src/index.ts:304).

   $sess = (Get-ChildItem D:\Ace-Step-Latest\hot-step-cpp\logs | Sort-Object Name -Descending | Select-Object -First 1).FullName
   Select-String -Path "$sess\ace_engine.log" -Pattern 'Listening on'
   Select-String -Path "$sess\node_console.log" -Pattern 'Restarting in 3 seconds'

4. After any upstream sync (the engine is a patched fork of acestep.cpp):

   powershell -File D:\Ace-Step-Latest\hot-step-cpp\engine\verify-hooks.ps1

   Exit 0 = good. Note: CLAUDE.md says "3 hook files" but the script checks 5 things (the script is the truth): pipeline-synth-ops.cpp → hot-step-sampler.h (loss is SILENT — compiles fine but all solvers/guidance/schedulers go dead), model-store.h → hot-step-params.h, dit.h → adapter-merge.h + adapter-runtime.h, tools/hot-step-server.cpp → hot-step-params.h, and a linker sentinel hotstep_sampler_linked_ in hot-step-sampler.h.

5. Lua plugins (engine/plugins/) need no rebuild, but they load at engine start — relaunch the app for a new/edited plugin to appear.

Tier 3 — UI visuals

Bar: npx tsc -b clean + Vite HMR applies the change without console errors + the human confirms the visuals. You cannot close this tier yourself. Ask the user to look and wait for their screenshot/feedback.

Tier 4 — Audio quality

Bar: the human's ear. There is no mechanical substitute — see Golden rules 1–2. Your job ends at: generation succeeded mechanically, here is the file path (server/data/audio/<uuid>.wav or http://localhost:3000/audio/<uuid>.wav in dev), here are the seed and params for reproduction. Then wait.

Tier 5 — End-to-end headless smoke generation

Bar: job reaches status: "succeeded", a WAV exists on disk, and the per-generation log ends with GENERATION COMPLETED. (Mechanical success only — quality is Tier 4.)

Full request/response/log detail: reference.md.

1. Pre-flight health — require engine.ready -eq $true:

   Invoke-RestMethod http://localhost:3000/api/health

2. Check the queue is free (don't queue behind/ahead of the user silently):

   Invoke-RestMethod http://localhost:3000/api/generate/queue

3. Get an auth token (POST /api/generate returns 401 without a Bearer token; tokens are in-memory and reset on server restart):

   $auth = Invoke-RestMethod http://localhost:3000/api/auth/auto
   $hdr = @{ Authorization = "Bearer $($auth.token)" }

4. Submit a fast smoke request — skipLm: $true skips the language-model phase (the "LM" that writes audio codes/metadata before the DiT diffusion transformer runs). Always pass duration explicitly: with skipLm the server defaults an unset duration to 120 s (generate.ts:222).

   $body = @{
     prompt         = 'minimal ambient techno, warm pads'
     instrumental   = $true
     skipLm         = $true
     duration       = 30
     inferenceSteps = 8
     seed           = 42      # fixed seed = reproducible; or randomSeed = $true
   } | ConvertTo-Json
   $job = Invoke-RestMethod -Method Post -Uri http://localhost:3000/api/generate -Headers $hdr -ContentType 'application/json' -Body $body
   $job.jobId

   Omit skipLm if your change touches the LM phase — then the smoke must exercise it.

5. Poll to completion (status values: pending | lm_running | synth_running | saving | succeeded | failed | cancelled):

   do { Start-Sleep 3; $s = Invoke-RestMethod "http://localhost:3000/api/generate/status/$($job.jobId)";
        "{0} {1}% {2}" -f $s.status, $s.progress, $s.stage
   } while ($s.status -notin 'succeeded','failed','cancelled')
   $s | ConvertTo-Json -Depth 5

   Built-in watchdogs (pollUntilDone, generate.ts:102-170): 2 min with no stage/progress change ⇒ "Generation stalled"; wall-clock default 45 min. A first-ever TensorRT run legitimately shows Building TRT engine ... (first run only, ~5-10 min) — do not panic-cancel it. There is 1 automatic retry with a randomized seed on transient failure.

6. Confirm outputs. On success $s.result.audioUrls is like ["/audio/<uuid>.wav"]. **The live data dir is server\data, NOT the repo-root data\** (verified: config.data.dir resolves relative to server/src, server/src/config.ts:191-199; root data/ is stale/legacy — never "verify" outputs there). Files land in server\data\audio\; DB row in the songs table of server\data\hotstep.db with the full request in generation_params.

7. Confirm the log. Per-generation logs are buffered in memory and flushed only at finish/fail — mid-run, tail node_console.log (lines prefixed [Gen:<jobId8>]) or ace_engine.log instead.

   $sess = (Get-ChildItem D:\Ace-Step-Latest\hot-step-cpp\logs | Sort-Object Name -Descending | Select-Object -First 1).FullName
   Get-Content (Get-ChildItem "$sess\generations" | Sort-Object LastWriteTime -Descending | Select-Object -First 1).FullName -Tail 40

   Success ends GENERATION COMPLETED.; failure ends GENERATION FAILED: <error> (server/src/services/logger.ts:145,167).

8. Hand off to the human (Tier 4): file path + seed + params. Do not delete the artifacts.