Verify Production

Verify production

The dashboard deploys itself: .github/workflows/cloudflare.yml builds, ships, and runs bun scripts/verify-deploy.ts on every push to main. So CI proves the deploy path. It does not prove the product works, because the checks CI runs stop at "the endpoints answered".

This skill is the knowledge for the second pair of eyes: what to probe, what a green probe actually means, and what to do when it is red.

Script contract, gotchas, and the assertion list: scripts/verify-deploy.ts (apps/dashboard/). The desk job that runs all of this on a schedule is local:prod — see docs/herdr-desk/prod-watch.md.

What each signal is worth

Anonymous /api/health returns {status, timestamp} only. Deployment metadata is deliberately withheld from anonymous callers (#1768), so you cannot compare gitSha to origin/main without a token. With CHM_API_KEY_SECRET in the env, verify-deploy.ts does the authenticated call for you.

--skip-auth passing does not mean ClickHouse is reachable

This is the trap, and it cost a full day of a broken guest experience before anyone noticed. The authenticated half of verify-deploy.ts is the only check that proves the Worker can query ClickHouse, and it needs CHM_API_KEY_SECRET. Without it, run the compensating check:

curl -sS -o /dev/null -w '%{http_code}\n' \
  'https://dash.chmonitor.dev/api/v1/host-status?hostId=0'
curl -sS https://dash.chmonitor.dev/api/healthz | head -c 400

• 500 with {"error":"error code: 1016"} — Cloudflare cannot resolve the origin domain. The signature of a ClickHouse host that only resolves on a tailnet: a *.ts.net MagicDNS name has no public DNS record, so it answers from a tailnet-connected box and nowhere else. Workers resolve through public DNS, so the edge gets 1016 while the host itself is alive and healthy.
• 503 with hosts[0].status: "down" — the Worker is up; upstream is not. /api/healthz returns 503 for the whole deployment when one host is down, so read it as a host signal, not a Worker signal.

In cloud mode the env host list is the public demo (docs/knowledge/cloud-saas-mode.md) and an anonymous visitor is shown nothing else — so one unreachable demo host means every guest data read fails while /api/health, /overview, the Clerk-key check and the agent gateway all stay green. Check cloudMode.mismatch in the healthz payload to rule out a build/runtime split-brain.

If you are fixing the host: note that filterToDemoHosts (lib/cloud/demo-hosts.ts) fails open — an allowlist matching zero hosts shows all env hosts rather than emptying the demo — so renaming allowlist entries neither breaks nor fixes this. Fix the host, then reconcile the names.

Run it

cd apps/dashboard

# anonymous only — no secret needed
bun scripts/verify-deploy.ts --skip-auth

# full, including ClickHouse connectivity
CHM_API_KEY_SECRET=… bun scripts/verify-deploy.ts --hosts 0

# a preview deploy
bun scripts/verify-deploy.ts \
  --base-url https://preview.dash.chmonitor.dev --json

Flags: --base-url <url> (default https://dash.chmonitor.dev), --hosts 0,1, --json, --skip-auth. Exit 0 = all pass, 1 = failures, 2 = harness error. The secret lives in apps/dashboard/.env.local and the GitHub secret of the same name — source it into the env, never print it.

When --skip-auth is all you can run (the secret is unset locally), add the host-status probe below. Four green unauthenticated checks and a broken guest product is a combination that actually happens.

Agent and guest-model probes

A dead model id is invisible in every health check, and that is exactly how it broke before: the guest default is the constant GUEST_DEFAULT_AGENT_MODEL in apps/dashboard/src/lib/billing/guest-ai.ts. That id can go stale while other surfaces still answer.

curl -sS https://dash.chmonitor.dev/api/v1/agents/config-check | head -c 400
curl -sS https://dash.chmonitor.dev/api/v1/agents/models     | head -c 400

configured.apiKey: true with a model list that 404s upstream is a routing / account fact, not a code bug. Record it; do not file a repo issue against the code for it, and never change a model default or a quota as a drive-by. A default id is a product decision: it needs a human.