Test Weave Router with Codex

Testing the router locally with Codex

For an automated pre-merge regression net (fixture-driven, asserts caching/streaming/decision-headers against real Anthropic), run make smoke — see docs/SMOKE.md. This skill is the interactive counterpart for the Codex CLI (codex exec): stand the stack up by hand and drive it via a throwaway CODEX_HOME so ~/.codex/config.toml is never edited.

Stand up the router in docker compose, point a one-off codex exec at it via CODEX_HOME, and read the local server logs to confirm behavior. Two upstream modes: the real provider API (needs a working key + credits, and for native GPT models a ChatGPT OAuth login) or a mock upstream that emits an exact SSE shape (deterministic, no credits).

The sibling skill test-claude-locally is the Claude Code (claude -p) counterpart. Stack bring-up and seeding are identical; only the client driver differs.

Critical gotchas (read first)

• Never edit ~/.codex/config.toml or ~/.codex/auth.json. Those are the user's live Codex install (prod Weave Router + ChatGPT OAuth). Redirect with a throwaway CODEX_HOME directory that contains its own config.toml (and a copy of auth.json when ChatGPT OAuth is required). CODEX_HOME is how Codex finds config; there is no --settings flag equivalent to Claude Code.
• requires_openai_auth = true is required. Codex 0.149 hangs or never issues a /v1/responses request if the custom provider uses requires_openai_auth = false or env_key. Copy ~/.codex/auth.json into the throwaway CODEX_HOME so Codex can attach the ChatGPT JWT + ChatGPT-Account-ID. The mock-router experiment that dropped OAuth never even hit the mock.
• Codex always sends a ChatGPT JWT + ChatGPT-Account-ID. The local router will classify that as a Codex subscription (codexResponsesRequest → native Responses passthrough for OpenAI decisions). That is the path this skill is usually exercising (routing markers on verbatim GPT frames). To force the prepaid/BYOK translation path instead, flip subscription_routing_disabled=true on the seeded installation (see step 3).
• codex exec is one-shot. A standalone /force-model call does not persist to the next codex exec (new session id every time, unless you codex exec resume). Put /force-model <model> as the first line of the SAME prompt. The leading space is load-bearing — Codex consumes unknown slash tokens as local commands; a leading space makes it a normal user message the router can parse.
• Pin via config.toml, not codex -c. On Codex 0.149.1, -c 'model_providers.weave-local.http_headers."x-weave-force-model"="…"' did not merge into the outgoing request (no x-weave-force-model applied in server logs; the scorer served something else). Put "x-weave-force-model" = "<id>" on the same http_headers table in the throwaway config.toml (step 4). Confirm with docker compose logs server | grep 'x-weave-force-model applied' before drawing conclusions.
• The router ignores the request's model field for routing (cluster scorer / pins win). Codex's -m gpt-5.5 only changes what Codex requests; /force-model or x-weave-force-model is the only way to pin a specific routed model. Native GPT family (gpt-5.6-sol / terra / luna, plus whatever -m Codex defaulted to) still goes through the scorer unless forced.
• No GNU timeout on macOS. Drive codex exec directly. If a run hangs (Reading additional input from stdin... with no banner), Codex is waiting on a TTY/stdin — do not pipe into it, and make sure -s read-only (or --dangerously-bypass-approvals-and-sandbox) is set.
• Docker may not be running. docker compose fails with a unix://…/.docker/run/docker.sock connect error if Desktop is stopped. open -a Docker and wait for docker info before step 1.
• Other agents in the same workspace can docker compose down and delete /tmp scratch. If health suddenly 404s or CODEX_HOME vanishes mid-run, re-up the stack and reseed — do not reuse a key you can no longer /validate.
• Session key is apiKeyID + first user message. Reusing the same curl/prompt body on the same key reuses the pin slot. routingMarkerFor then returns empty when PriorServedModel == ServedIdentity() (sticky same-model). For first-turn marker tests, seed a new key or change the first user text. Isolated /validate first: curl -sS -o /dev/null -w '%{http_code}\n' -H "X-Weave-Router-Key: $KEY" http://localhost:8080/validate must be 200.
• Native GPT passthrough + tool-only emits a synthetic badge item. SetPassthroughBadge rewrites existing response.output_text.* events and, when a ChatGPT-subscription turn has no assistant text, inserts a leading assistant message item before native tool/reasoning output. Verify the badge in the Responses SSE/JSON; the Codex TUI can bury the first-turn line under tool chatter.
• GET /v1/models 501 from a mock is fine. Codex probes <base_url>/models first, logs an HTML 501, then continues to POST /v1/responses. The real local router implements GET /v1/models as Anthropic passthrough, so this only shows up against a Python mock.
• Port 8085 conflict. The monorepo's pubsub emulator may already own host port 8085. Drop the router's host binding with a docker-compose.override.yml (see workflow). The server still reaches the emulator over the compose network.
• No credits / no key = no reproduction. If the real upstream returns an error (e.g. OpenRouter "Insufficient credits"), use the mock-upstream path instead.

Workflow

- [ ] 1. Bring up the stack (handle port 8085)
- [ ] 2. Seed an API key
- [ ] 3. Choose upstream + subscription vs prepaid path
- [ ] 4. Write a throwaway CODEX_HOME (copy auth.json)
- [ ] 5. Drive with `CODEX_HOME=... codex exec`, forcing the target model
- [ ] 6. Read local logs to verify behavior
- [ ] 7. Clean up

1. Bring up the stack

cd <router-repo>
# Drop the pubsub host-port binding to avoid an 8085 conflict:
cat > docker-compose.override.yml <<'EOF'
services:
  pubsub-emulator:
    ports: !reset []
EOF
docker compose up -d --build server   # --build picks up code changes
until curl -sf http://localhost:8080/health >/dev/null; do sleep 2; done

The override file is gitignored-by-intent scaffolding — delete it in cleanup.

2. Seed an API key

docker compose run --rm seed

Copy the rk_... key it prints — only the line under Weave Router key (shown once — store it now):. Seed stdout repeats the same token in curl examples and installer snippets; grep -oE 'rk_[A-Za-z0-9]+' can grab a stale key from an earlier seed in the same log. Always /validate before driving Codex:

curl -sS -o /dev/null -w '%{http_code}\n' \
  -H "X-Weave-Router-Key: rk_REPLACE_ME" http://localhost:8080/validate
# expect 200

A 401 mid-session usually means the stack was torn down and reseeded (new installation, old token dead).

3. Choose the upstream

Real provider — set the provider key in .env.local (e.g. OPENAI_API_KEY=..., FIREWORKS_API_KEY=...) and restart docker compose up -d server. Confirm the boot log shows <Provider> provider enabled with the real base_url. Use this to confirm a model genuinely produces the behavior.

For an actual Codex subscription response, do not set ROUTER_CODEX_BASE_URL and do not run either mock. Remove that environment entry from docker-compose.override.yml, then rebuild the server:

unset ROUTER_CODEX_BASE_URL  # also remove it from docker-compose.override.yml
docker compose up -d --build server

Use a throwaway CODEX_HOME containing a copy of the real ~/.codex/auth.json, with requires_openai_auth = true and X-App = "codex". The router detects the OAuth bearer plus ChatGPT-Account-ID and calls the real https://chatgpt.com/backend-api/codex/responses endpoint. This consumes the user's ChatGPT quota and requires a valid login; the mock is only for repeatable tests when that endpoint is unavailable or too expensive.

Mock upstream — for a deterministic, credit-free repro of a precise SSE shape. Point the provider's base URL at a local mock and restart. Codex talks to the router (POST /v1/responses); the mock sits behind the router as the upstream the router dispatches to:

python3 .claude/skills/test-claude-locally/scripts/mock_openai_upstream.py >/tmp/mock.log 2>&1 &   # serves :8099
# In docker-compose.override.yml under `server:`, add:
#   environment:
#     FIREWORKS_BASE_URL: http://host.docker.internal:8099/v1
#     FIREWORKS_API_KEY: sk-mock
#   extra_hosts: ["host.docker.internal:host-gateway"]
docker compose up -d server

To exercise the production-equivalent native Codex subscription path, use a native Responses mock instead of the Chat Completions mock:

python3 .claude/skills/test-codex-locally/scripts/mock_codex_upstream.py >/tmp/mock-codex.log 2>&1 &

Add this to the server service in docker-compose.override.yml:

environment:
  ROUTER_CODEX_BASE_URL: http://host.docker.internal:8099/v1
extra_hosts: ["host.docker.internal:host-gateway"]

This preserves real Codex subscription detection (OAuth bearer plus ChatGPT-Account-ID) and native Responses passthrough, while replacing only the outbound ChatGPT backend with a deterministic tool-only stream. The mock emits a function call, so the expected result is a synthetic marker message at output_index: 0, followed by the native tool at output_index: 1.

Edit the mock's emitted chunks to match the upstream shape you're reproducing. Provider→env-var names live in internal/providers/provider.go; base-URL overrides are read in cmd/router/main.go (<PROVIDER>_BASE_URL). For local testing, ROUTER_CODEX_BASE_URL overrides only the subscription backend; the default remains https://chatgpt.com/backend-api/codex.

Subscription vs prepaid (important for marker / passthrough tests):

Codex will attach a ChatGPT JWT. The local router then treats the turn as a Codex subscription:

• OpenAI-family decision → verbatim Responses passthrough (SetPassthrough / SetPassthroughBadge) to the Codex backend (or ROUTER_CODEX_BASE_URL in the native mock setup). This is the path the routing-marker-on-Codex work exercises.
• Non-OpenAI decision → Chat Completions translation + ResponsesWriter.SetBadgeText.

To force the prepaid/BYOK translation path (no ChatGPT backend, uses OPENAI_API_KEY / other provider keys):

docker compose exec postgres psql -U router -d router -c \
  "SET search_path TO router; UPDATE model_router_installations SET subscription_routing_disabled=true WHERE external_id='__router_admin__';"

Revert with subscription_routing_disabled=false when done.