Subwave LLM Bench

SUB/WAVE LLM model benchmark

Two harnesses, one job: measure how well a provider + model handles the station's real LLM calls before trusting it on air.

• llm-bench (controller/scripts/llm-bench/, npm run llm-bench) — the primary harness. A matrix runner over every on-air call kind: pool picks (pickNextTrack), agent picks (djAgentPick), pool + agent segments (generateSegment / djAgentSegment), request matching (matchRequest / djAgentRequest), the free-text scripts (intro, link, station ID, hourly), banter, and the programme family (plan, beats, exchanges). Scores reliability + deterministic rule checks, prints a per-model comparison table, writes a diffable JSON report.
• picker-test.mjs — the legacy picker-only deep-dive. Still useful for high-iteration picker runs comparable with historical results, and it has a bundled orchestration script (see the legacy section at the end).

Both import the live prompts and schemas from src/ (never copies), fake everything external (library, tools, weather/news data), and override provider/model only inside their own short-lived process — the live controller's configured model is never touched.

The two things that surprise people

1. Routing matters as much as the model. The same model can pass through one provider and fail through another, because each @ai-sdk/* provider translates tools / structured output differently. Canonical case: deepseek-v4-flash scored 0/4 via the deepseek direct provider but 4/4 via openrouter. Always benchmark through the routing you'll actually deploy — "is model X good?" is the wrong question; "is provider+X good?" is the right one.
2. The stress scenarios are where truth lives. Easy cells make every model look fine. The verdict cells are: djAgentPick/long-context (full prompt + tool loop), generateSegment/dull-weather (the model must decline to air — small models botch the silence encoding; a huge wall-clock here means the structured-output retry rescued a failed first attempt), and the multi-voice JSON kinds (generateBanter, generateProgrammeExchange).

When NOT to use this

• Changing the live model — that's an admin Settings change, not this skill.
• Diagnosing general runtime behaviour — subwave-log-analysis.
• Starting/stopping the stack — subwave-control.

Running llm-bench

Runs on the host from a repo clone (it imports TS source via tsx; prod containers ship only compiled dist/). The running stack is not required — keys resolve from state/secrets.env and controller/.env automatically (existing env vars win). If a key exists only inside a prod container's env, copy it out first: export OPENROUTER_API_KEY=$(docker exec sub-wave-controller printenv OPENROUTER_API_KEY).

cd <repo>/controller
npm run llm-bench -- \
  --models openrouter:google/gemma-4-31b-it,ollama:qwen3:8b \
  --iterations 3

Flags:

• --models (required) — comma list of provider:model, split on the FIRST colon, so Ollama tags keep theirs (ollama:qwen3:8b). Providers: ollama | openai-compatible | anthropic | openai | google | deepseek | openrouter | requesty | gateway.
• --kinds — groups (pick,segment,request,scripts,banter,programme) or exact kind names. Default: all.
• --modes — pool,agent (default both). Mode-independent kinds run once either way. If the operator runs pool mode live (Agentic picker off), --modes pool is the honest benchmark — agent cells they'll never use just add noise and cost.
• --iterations — per scenario. 1 to smoke, 3 to screen (default), 5+ to confirm a winner. Full matrix ≈ 32 scenario cells, so ~100 calls per model at 3 iterations.
• --out — report path (default scripts/llm-bench/reports/<ts>.json, gitignored).

A full run takes many minutes (every call is a real model call, agent cells can take 45 s each). Run it in the background with a long timeout and don't poll; OLLAMA_URL=http://localhost:11434 translates the container-internal Ollama address for host runs.