Server Feature Dev

Node Server Feature Development

The Node server (server/src/) is the middle tier of HOT-Step CPP, a local AI music-generation app. It serves the React UI, owns the SQLite database, and orchestrates the C++ inference engine (ace-server.exe, spawned as a child process). This skill covers how to add or change a server feature the way this codebase does it.

Glossary (no prior context assumed):

• Engine / ace-server — the C++ binary doing actual AI inference, HTTP API on port 8085. The Node server spawns it and talks to it over HTTP.
• DiT — Diffusion Transformer, the engine's main music-synthesis model. LM — the language model that generates "audio codes" before synthesis. VAE — decodes latents to audio. Pipeline: LM → DiT → VAE.
• AceRequest — the canonical JSON request shape the engine accepts, defined in server/src/services/aceClient.ts:38-151.
• aceReq — the AceRequest instance built for a generation job (from UI params via translateParams).
• Adapter — a LoRA/LoKr fine-tune applied to the DiT at generation time.

UI (fetch /api/*) → Express :3001 (dev: Vite on :3000 proxies /api to it) → routes/*.ts → services/*.ts
                                        ├→ getDb()  (better-sqlite3, synchronous)
                                        └→ aceClient (HTTP → ace-server :8085)

When to use this skill

• Adding a new /api/* endpoint or a whole feature (route + service).
• Modifying SQLite schema or queries (server/src/db/database.ts).
• Adding a config value or a Settings-UI-exposed .env key (server/src/config.ts).
• Making the server call the engine (new AceRequest field, new engine endpoint).
• Debugging server-side failures (jobs stalling, engine timeouts, DB errors).

Golden rules (hard constraints — each prevents expensive damage)

1. Type-check with npx tsc --noEmit during dev; only npm run build before user testing. (Institutional rule, verbatim.) tsx watch runs TypeScript directly, so a build is wasted work mid-dev — but tsx also tolerates errors that tsc rejects, so type-check before committing.

   cd d:\Ace-Step-Latest\hot-step-cpp\server; npx tsc --noEmit

2. Node 20 to 24 LTS, 24 recommended. server/package.json:5-7 enforces >=20.0.0 <25.0.0. better-sqlite3 is a native module; postinstall runs npm rebuild better-sqlite3 (package.json:14).
3. ESM everywhere — relative imports MUST end in .js even though sources are .ts. server/package.json:8 sets "type": "module". import { config } from '../config.js' — omitting .js may run under tsx but fails tsc. There is no __dirname; the pattern is const __dirname = path.dirname(fileURLToPath(import.meta.url)) (config.ts:7, index.ts:46).
4. Never block a request on long work. The engine is single-threaded; generations take minutes. Return a job id immediately and let the UI poll (see the worked example). The generation queue is deliberately serialized to one job at a time (generate.ts:1286-1292).
5. Never call getDb() at module import time. initDb() runs at index.ts:60; a query executed during module load throws Database not initialized. Call initDb() first. (database.ts:17-22). Query only inside handlers/functions.
6. Declare static routes before /:id params. Express matches in order — GET /recent after GET /:id returns a 404 for id "recent". See the explicit warning at songs.ts:47 and seeds.ts (/favorites, /random at lines 131/143 precede /:name at 156).
7. Git: all work on master, stage explicit paths (never git add -A, never git add -f on gitignored paths — data/, logs/, docs/plans/, checkpoints/ are gitignored on purpose), commit locally often, push only with explicit user approval. Pushing any v* tag triggers a full multi-platform CI release build.
8. PowerShell syntax: ; not && for command chaining on this machine.
9. Don't visually verify the UI with a browser agent — ask the human user; they provide screenshots. (Hitting API endpoints programmatically is fine.)
10. If your change touches C++ (engine/src/), rebuild via dev-rebuild.bat at repo root — never engine/build.cmd directly (you cannot reliably tell whether the app is running; the Node server auto-respawns ace-server on crash, index.ts:284-309 → infinite respawn + file-lock loop). Never cmake --clean-first (20+ min CUDA recompile); for stale .obj issues delete only engine/build/acestep-core.dir/ and engine/build/Release/acestep-core.lib.
11. Never delete generated audio or test outputs, even ones you predict are bad — the human verifies results by ear. Leave all artifacts in place unless the user explicitly asks for cleanup.

Dev loop

1. Start dev mode: d:\Ace-Step-Latest\hot-step-cpp\dev.bat — runs Vite (UI, :3000, HMR) plus server\restart-loop.cmd, which loops npx tsx watch src/index.ts (restart-loop.cmd:6). Develop against http://localhost:3000. LAUNCH.bat (Node :3001 serving prebuilt ui/dist/) is the end-user prod path — agents don't run it.
2. Save any server/src/**/*.ts file → tsx watch restarts the whole Node process, which kills and respawns ace-server too (it's a child process). Expect a few seconds of engine downtime after every save; a generation in flight will die.
3. The restart loop re-enters if a .restart-requested marker file exists at repo root — this is how the in-app "restart server" works without killing Vite.
4. Type-check before committing: cd d:\Ace-Step-Latest\hot-step-cpp\server; npx tsc --noEmit.
5. Logs land in logs/<YYYY-MM-DD_HH-MM-SS>/ at repo root: node_console.log (server), ace_engine.log (engine), generations/gen_<jobId>_<task>.log (per generation). Newest folder = current session.

Procedure: add a new feature (route + service)

1. Create the route file server/src/routes/myFeature.ts. Convention: open with a comment block stating the mount point and a route table (best example: seeds.ts:1-20).

   // myFeature.ts — one-line purpose
   // Mounts at: /api/my-feature
   // Routes:
   //   GET  /api/my-feature        — list things
   import { Router } from 'express';
   import { config } from '../config.js';   // note the .js extension
   
   const router = Router();
   
   router.get('/', (req, res) => {
     try {
       res.json({ ok: true });
     } catch (err: any) {
       console.error('[MyFeature] list failed:', err.message);
       res.status(500).json({ error: err.message });
     }
   });
   
   export default router;

2. Register it in server/src/index.ts — two edits: add the import to the block at index.ts:21-44 (import myFeatureRoutes from './routes/myFeature.js';) and mount it in the block at index.ts:72-95 (app.use('/api/my-feature', myFeatureRoutes);). URL segments are kebab-case (/api/model-manager, /api/cover-art, /api/stem-studio). One legacy exception: songBuilder.ts mounts at /api/builder.
3. Put non-trivial logic in a service: server/src/services/myFeature.ts. When a feature grows, use a sub-folder — services/generation/ holds translateParams.ts, lmCache.ts, postProcessing.ts, sourceAudio.ts, etc., all extracted from the once-monolithic generate.ts.
4. Auth-scope if the data is per-user: import { getUserId } from './auth.js'; then

   const userId = getUserId(req);
   if (!userId) { res.status(401).json({ error: 'Unauthorized' }); return; }

   (exact pattern: songs.ts:19-20). Auth is deliberately lightweight: an in-memory Map<token, userId> that resets on restart, with a single auto-created local user "Producer" (auth.ts:13-26). getUserId is auth.ts:98-102.
5. Express 5 handler style: end early with res.status(...).json(...); return; — do not return res... from typed handlers. The SPA fallback uses Express-5 wildcard syntax app.get('/{*splat}', ...) (index.ts:136) — Express 4 '*' patterns do not work.
6. Log with a bracketed feature tag — console.log('[MyFeature] ...'). Existing tags: [Server], [DB], [Config], [Generate], [Settings], [Logger], [ace-server]. This convention is what makes node_console.log greppable. To also surface lines in the UI's live log panel, import { pushLog } from './logs.js'; pushLog('text', 'server');.
7. Type-check (npx tsc --noEmit), test via the running dev server, then commit explicit paths:

   cd d:\Ace-Step-Latest\hot-step-cpp; git add server/src/routes/myFeature.ts server/src/services/myFeature.ts server/src/index.ts; git commit -m "feat(server): my feature"

Not every feature needs SQLite. routes/seeds.ts stores JSON files under server/data/seeds/ on purpose (ComfyUI-compatible, import/export-friendly) with a filename-sanitization regex at seeds.ts:44. Use seeds.ts as the template for small self-contained features; use generate.ts for engine-orchestrating ones.

Procedure: SQLite schema change

All DB work lives in server/src/db/database.ts. One connection, synchronous better-sqlite3 API, module singleton (initDb() once at startup, getDb() everywhere else, closeDb() on shutdown at index.ts:564). Pragmas at open: WAL, synchronous = NORMAL, foreign_keys = ON (database.ts:32-34).

1. New table: add a CREATE TABLE IF NOT EXISTS block inside the big db.exec(...) at database.ts:37-209. Schema is re-run idempotently at every startup — there is no migration framework and no version table.
2. New column on an existing table: add an entry to the checked-migration list (the songsMigrations array pattern, database.ts:213-259):

   {
     check: `SELECT COUNT(*) as c FROM pragma_table_info('songs') WHERE name='my_col'`,
     alter: `ALTER TABLE songs ADD COLUMN my_col TEXT DEFAULT ''`,
   },

   The loop runs ALTER only when the column is absent and logs [DB] Migration: .... Use this idiom for new columns. (A second, older idiom — try { db.exec("ALTER TABLE ...") } catch {} — exists for the lireek tables at database.ts:262-278; don't add to it.)
3. Column conventions: TEXT primary keys are UUIDs (uuidv4()) for core tables; lireek (Lyric Studio) tables use INTEGER PRIMARY KEY AUTOINCREMENT; timestamps TEXT DEFAULT (datetime('now')); booleans are INTEGER 0/1, re-hydrated as !!row.is_public (songs.ts:38); complex values are JSON-in-TEXT parsed with JSON.parse(s.tags || '[]').
4. Query style: inline getDb().prepare(sql).get/all/run(...params) with ? placeholders — no statement-caching layer. Dynamic filters append to the SQL string and a params array (songs.ts:24-32), including JSON queries like json_extract(generation_params, '$.source') = ?.
5. Don't imitate the two-database pattern. db/lireekDb.ts is a query-helper module over the same connection; its initLireekDb/closeLireekDb are deprecated no-ops (lireekDb.ts:12-23). The old separate lireek.db was consolidated into server/data/hotstep.db (one-time ATTACH DATABASE migration at database.ts:293-386, a good reference if you ever need bulk data migration).

Procedure: add a config / .env setting

server/src/config.ts builds one exported config object (config.ts:96-257) grouping aceServer (engine exe autodetected from 4 candidate paths at config.ts:46-52; port default 8085), server (port 3001), data (getter dbPath → server/data/hotstep.db; DATA_DIR in .env resolves relative to server/, NOT the repo root — config.ts:192. A stale legacy data/ dir may exist at repo root with an outdated hotstep.db; ignore it), lireek (LLM keys/endpoints), vst, whisper, essentia. .env lives at repo root and is auto-bootstrapped from .env.example on first launch (config.ts:22-30).

To expose a new env setting in the Settings UI, you must touch three places in config.ts or it silently won't work:

1. Add the key to the EXPOSED_ENV_KEYS whitelist (config.ts:265-286) — "nothing else leaks" to the UI.
2. If it only takes effect at process/engine spawn time, also add it to RESTART_REQUIRED_KEYS (config.ts:289-294) so the UI shows "restart required".
3. Add an apply('MY_KEY', setter, getter) line inside reloadEnvConfig() (config.ts:300-395) — without this the value saves to .env but never hot-patches the live config.

The Settings route (POST /api/settings/env, settings.ts:109-174 — note: the header comment says PUT, the code is POST; code wins) rewrites .env preserving comments and line endings, calls reloadEnvConfig(), and returns restartRequired.

Procedure: call the engine (aceClient)

server/src/services/aceClient.ts is the typed wrapper over ace-server's HTTP API (BASE = config.aceServer.url, i.e. http://127.0.0.1:8085 by default).

Critical constraint (aceClient.ts:6-8, verbatim comment): "ace-server uses single-threaded httplib. During heavy compute (DiT generation, adapter merge, VAE decode) it cannot respond to HTTP requests." Hence three timeout tiers via AbortSignal.timeout (aceClient.ts:17-19) — pick one deliberately for any new call:

• Async-job pattern: submitLm / submitSynth / submitUnderstand / warm POST and return a job id string; poll with pollJob(id) → { status: 'running'|'done'|'failed'|'cancelled', phase?, phase_step?, phase_total? } (phases like adapter_precompute, dit_inference — aceClient.ts:165-180); fetch the result with getJobResult(id) (GET /job?id=N&result=1); cancel with cancelJob(id). Sync endpoints (/spectral-lifter, /pp-vae-reencode) return processed WAV bytes directly.
• A poll timeout is NOT a failure. The engine may simply be busy mid-DiT-step. Catch transient poll errors and retry; only treat explicit failed/cancelled statuses as terminal (see generate.ts:158-166).
• New engine parameter? Add the field to the AceRequest interface first (aceClient.ts:38-151), then plumb it through services/generation/translateParams.ts (UI params → AceRequest). Known gotcha (institutional, from the LM-echo bug): server-only fields do not survive the engine's /lm round trip — synth requests are rebuilt from the original aceReq plus only the LM-generated fields (audio_codes, caption, lyrics, bpm, duration, keyscale, timesignature — generate.ts:237-246). Never assume a field you sent to /lm comes back.
• File upload to the engine: multipart is hand-rolled with a manual boundary because the engine's parser expects a request JSON part plus named binary parts (audio, ref_audio, src_latents, ref_latents, seed_latents). The multipart path takes a single JSON object, not an array (aceClient.ts:374-378). Copy submitSynthMultipart (aceClient.ts:347-424).
• Error convention: non-OK responses throw new Error('ace-server POST /x failed (status): body') (aceClient.ts:266-270); soft-fail helpers return null/false (getJobLatent, isReachable).
• Readiness gate: routes that hit the engine should return 503 while the engine is booting. import { engineReady, engineBootStatus } from '../engineState.js'; — this module exists solely to break a circular import between index.ts and routes (engineState.ts:1-5). Pattern: generate.ts:1356-1363.