Codex Session Debugging

Debugging a Codex session

Given a Codex session ID, pull the local rollout transcript (what the client saw) and the corresponding production cloud logs (which model/provider served it), then correlate them. The rollout .jsonl is ground truth for what rendered; the cloud logs confirm what the router decided and what the upstream sent; internal/translate/responses*.go + internal/proxy/service.go explain why the wire shape looks that way.

The sibling skill debug-claude-session is the Claude Code counterpart. The workflow is the same shape; the transcript format, the id conventions, and the correlation key are all different — do not carry Claude assumptions over.

Setup: Cloud deployment config

Before starting, create a gitignored config file with your deployment's cloud logging details:

cat > .claude/skills/debug-codex-session/.deployment.json <<'EOF'
{
  "cloud_provider": "gcp",
  "project_id": "your-project-id",
  "region": "us-central1",
  "service_name": "router",
  "log_command_template": "gcloud logging read ... --project {project_id} --format=json"
}
EOF

If .deployment.json is missing, prompt the user for these details and walk them through creating it. The file is gitignored and contains no secrets — just the service/project/region names needed to construct cloud log queries.

Critical gotchas (read first)

• Correlate by client_session_id, not by time. Codex 0.149+ sends the session ID as the Session-Id header (and Thread-Id with the same value on the main thread); sessionIDFromHeaders (internal/proxy/client_identity.go) picks it up and bindRequestLogger binds it as client_session_id on every log line. So the Codex session ID from the transcript filename is a direct log filter. This is the big difference from the Claude skill, which has to correlate by time + model.
• The transcript does NOT record the served model. turn_context.model and thread_settings_applied.model are the model Codex requested (e.g. gpt-5.6-sol). The router ignores the request's model field for routing. The served model appears only in (a) the injected Weave routing marker, when one was emitted, and (b) decision_model in cloud logs. Never report turn_context.model as the model that served the turn.
• Empty summary: [] on a reasoning item is not lost thinking. Native Codex reasoning items carry opaque encrypted_content (replay state for the next Responses request) and a public summary array. With summary: "auto" the upstream frequently returns zero summary text while still billing reasoning_output_tokens. Check event_msg/token_count → info.total_token_usage.reasoning_output_tokens before concluding thinking was dropped — nonzero tokens with empty summaries means the model reasoned and the upstream chose not to surface a summary.
• Codex subscription turns take the native passthrough path — the router does not translate them. codexResponsesRequest (internal/proxy/service.go) detects a ChatGPT JWT + ChatGPT-Account-ID; the original Responses body is preserved and ResponsesWriter.SetPassthrough() / SetPassthroughBadge() (internal/translate/responses.go) forward upstream bytes verbatim. On that path, "the router dropped it" is almost never the answer — verify passthrough before blaming translation.
• ProxyOpenAIChatCompletion in the logs does not mean Codex used chat completions. ProxyOpenAIResponses delegates into the shared chat proxy for routing/billing/telemetry, so the completion log line is named ProxyOpenAIChatCompletion complete and ingress="openai_chat_completions" even for /v1/responses traffic. Check path on the access-log line to see the real surface.
• Short base36 item ids are router-minted; long hex ids are upstream. newResponsesID produces msg_23aljo-style ids, so a synthetic routing-badge assistant item looks like msg_s6w20w, while a genuine upstream assistant message looks like msg_09cf01fda402.... Same for rs_, ctc_, fc_ prefixes. This is the fastest way to tell injected content from served content.
• The routing marker is a separate assistant item, and it is stripped on the way back up. StripRoutingBadgeFromResponsesInput removes the badge from replayed input so router text never reaches the model. A missing marker on a turn is often expected (sticky same-model turns emit nothing) — see test-codex-locally for the PriorServedModel == ServedIdentity() rule.
• The rollout file mixes client-side and upstream events. response_item entries are conversation items (both directions); event_msg entries are client-side UI/telemetry events. developer/user role messages with UUID-shaped ids are locally constructed, not served.

Workflow

- [ ] 1. Locate the local rollout transcript
- [ ] 2. Get the session shape (event histogram + session_meta)
- [ ] 3. Read turn_context: requested model, effort, summary mode
- [ ] 4. Extract the items showing the symptom
- [ ] 5. Check token accounting before calling anything "missing"
- [ ] 6. Fetch cloud logs filtered by client_session_id
- [ ] 7. Correlate transcript + cloud logs
- [ ] 8. Trace to the responses code path

1. Locate the local rollout transcript

Codex writes one rollout file per session, date-partitioned, with the session ID as the filename suffix:

find ~/.codex/sessions -name "rollout-*<SESSION_ID>.jsonl" -type f

You get ~/.codex/sessions/<YYYY>/<MM>/<DD>/rollout-<ISO8601>-<SESSION_ID>.jsonl — one JSON object per line, no sibling spillover directory (unlike Claude Code). wc -l it; a working session is typically hundreds of lines.

Note the ISO timestamp in the filename is local time; the timestamp field inside each line is UTC (Z-suffixed). Use the in-file timestamps for any time math.

2. Get the session shape

Every line is {"timestamp", "type", "payload"}. The type is the outer kind; payload.type is the specific event. Histogram first — it tells you at a glance whether the session compacted, aborted, or ran tools:

python3 - <<'EOF'
import json, collections
F = "<path>/rollout-...-<SESSION_ID>.jsonl"
c = collections.Counter()
for line in open(F):
    try:
        o = json.loads(line)
    except Exception:
        continue
    c[(o.get("type"), (o.get("payload") or {}).get("type"))] += 1
for k, v in c.most_common():
    print(v, k)
EOF

The kinds you'll see:

model_provider on session_meta tells you whether the session went through the router at all — it's the provider id from ~/.codex/config.toml (e.g. weave). If it isn't the router's provider id, stop: this session never hit the router.

3. Read turn_context

python3 - <<'EOF'
import json
F = "<path>/rollout-...-<SESSION_ID>.jsonl"
for i, line in enumerate(open(F), 1):
    o = json.loads(line)
    if o.get("type") != "turn_context":
        continue
    p = o["payload"]
    print(f"line {i}: turn_id={p.get('turn_id')} requested_model={p.get('model')} "
          f"effort={p.get('effort')} summary={p.get('summary')}")
EOF

effort and summary are the two knobs that decide how much reasoning is produced and how much of it is displayable. summary: "auto" is the usual cause of a session with lots of reasoning tokens and no visible thinking. Again: model here is the request, not the route.

4. Extract the items showing the symptom

Adapt this to your symptom. The template below covers the common "no thinking rendered" case plus assistant-message provenance:

python3 - <<'EOF'
import json
F = "<path>/rollout-...-<SESSION_ID>.jsonl"
reasoning = empty_summary = 0
for i, line in enumerate(open(F), 1):
    o = json.loads(line)
    p = o.get("payload") or {}
    t = p.get("type")
    if t == "reasoning":
        reasoning += 1
        summary = p.get("summary") or []
        enc = p.get("encrypted_content") or ""
        if not summary:
            empty_summary += 1
        if reasoning <= 3:  # sample the first few
            print(f"line {i}: rs id={p.get('id')} summary_parts={len(summary)} "
                  f"encrypted_len={len(enc)}")
    elif t == "message" and p.get("role") == "assistant":
        mid = p.get("id") or ""
        # short base36 tail => router-minted (badge); long hex => upstream
        origin = "router" if len(mid) < 20 else "upstream"
        text = "".join(b.get("text", "") for b in (p.get("content") or []))
        print(f"line {i}: assistant id={mid} origin={origin} {text[:80]!r}")
print(f"\nreasoning items: {reasoning}, with empty summary: {empty_summary}")
EOF

To see exactly what the TUI rendered (including markers), read the agent_message events instead — those are the client-side render records:

python3 - <<'EOF'
import json
F = "<path>/rollout-...-<SESSION_ID>.jsonl"
for i, line in enumerate(open(F), 1):
    o = json.loads(line)
    p = o.get("payload") or {}
    if p.get("type") == "agent_message":
        print(f"line {i}: {(p.get('message') or '')[:120]!r}")
EOF

The routing marker renders with invisible provenance characters (``) prefixed by EnableCodexBadgeProvenance — that prefix is how the router recognizes and strips its own badge on replay. Seeing those codepoints confirms the marker came from the router, not the model.