Troubleshoot

/troubleshoot <error description>   <- This skill (diagnosis & fix planning)
    | After approval
/team-execute                       <- Parallel fix implementation (Phase 1)
    | After completion
    Phase 2 REVIEW                  <- Parallel review (regression check)

Phase 1: REPRODUCE & UNDERSTAND (Opus 1M context + Codex Initial Analysis + Claude Lead)
  Opus subagent analyzes the error context, Codex generates initial hypotheses,
  Claude gathers details from the user
    |
Phase 2: DIAGNOSE (Agent Teams -- Parallel, Codex-driven)
  Root Cause Analyst (Codex mandatory) <-> Impact Investigator (Opus + Codex) communicate bidirectionally
  Both teammates consult Codex for deep reasoning throughout analysis
    |
Phase 3: FIX PLAN & APPROVE (Codex Validation + Claude Lead + User)
  Integrate diagnosis results, validate fix plan with Codex, get user approval

python3 .claude/skills/_shared/workspace.py --skill troubleshoot --title "{short English title}" --create

python3 .claude/skills/troubleshoot/repro.py "<repro-command>" \
  --label {slug}-initial [--file <path-from-stack-trace>] [--timeout 120]

Task tool:
  subagent_type: "general-purpose-opus"
  prompt: |
    Analyze this reproduced error (already captured by repro.py):

    Error: {error message / stack trace}
    repro.py JSON: {exit_code, traceback, recent_commits, log_file}

    Tasks:
    1. Read all files mentioned in the traceback; trace the execution flow
       leading to the error and identify the immediate cause (what line fails).
    2. Look for related tests and whether they pass/fail.
    3. Check if similar patterns exist elsewhere in the codebase.

    Use Glob, Grep, and Read tools to investigate thoroughly.

    Save analysis to `{paths.context}` (from Phase 1 Step 0).
    Return concise summary (5-7 key findings).

Objective: Analyze this error and generate initial hypotheses for root cause.
Context:
- Error: {error message / stack trace}
- Failing location: {file:line from Opus subagent analysis}
- Execution flow: {call chain from Opus subagent analysis}
Constraints:
- Focus on root cause categories (state mutation, boundary, concurrency, dependency, type/contract)
- Rank hypotheses by likelihood
- Suggest specific code areas to investigate for each hypothesis
Output format:
## Error Pattern Recognition
## Hypotheses (ranked by likelihood)
## Investigation Plan (per hypothesis)
## Known Similar Patterns

python3 .claude/skills/_shared/codex_consult.py --prompt-file .claude/logs/codex/prompt-troubleshoot-initial.md --label troubleshoot-initial

python3 .claude/skills/_shared/validate_doc.py --contract bug-report --file {paths.bug_report}

Create an agent team named `{team_name}` for troubleshooting: {slug}

Spawn two teammates:

1. **Root Cause Analyst** — Uses Codex CLI as PRIMARY analysis engine for deep code reasoning
   Prompt: "You are the Root Cause Analyst for bug: {slug}.

   Your job: Identify the definitive root cause of this error through deep code analysis.
   Codex CLI is your PRIMARY tool for reasoning about code behavior.

   Bug Report: read `{paths.bug_report}` (written and validated in Phase 1 Step 3).

   Tasks:
   1. Trace the execution flow step by step from entry point to error
   2. Evaluate each hypothesis from the Bug Report:
      - Gather evidence FOR and AGAINST each hypothesis
      - Eliminate hypotheses that contradict the evidence
   3. Identify the root cause (not just the symptom):
      - What is the underlying defect?
      - Why does it manifest as this specific error?
      - Under what conditions does it trigger?
   4. Propose fix approaches (at least 2 alternatives):
      - Approach A: {description, pros, cons}
      - Approach B: {description, pros, cons}
      - Recommended approach with rationale

   ## Codex Analysis Protocol (MANDATORY)

   You MUST consult Codex for EACH of the following analysis tasks.
   Do NOT skip Codex consultation — it is the primary reasoning engine for this role.
   Each consultation below follows the same shape: write the prompt to a file,
   then run `python3 .claude/skills/_shared/codex_consult.py --prompt-file <path> --label <label>`
   and read the JSON `response_file`.

   ### 1. Execution Flow Tracing
   For complex control flow, write the prompt below to a file, then consult Codex:

   Objective: Trace the execution flow from {entry point} to {error location}.
   Context:
   - Entry point: {file:function}
   - Error location: {file:line}
   - Key intermediate functions: {list}
   Constraints:
   - Track state transformations at each step
   - Identify where assumptions are violated
   Output format:
   ## Execution Flow (step by step)
   ## State Transformations
   ## Assumption Violations
   ## Critical Decision Points

   python3 .claude/skills/_shared/codex_consult.py --prompt-file .claude/logs/codex/prompt-troubleshoot-flow.md --label troubleshoot-flow

   ### 2. Hypothesis Evaluation
   For each hypothesis, write the prompt below to a file, then consult Codex to evaluate evidence:

   Objective: Evaluate hypothesis "{hypothesis}" against collected evidence.
   Context:
   - Hypothesis: {description}
   - Evidence FOR: {list}
   - Evidence AGAINST: {list}
   - Code context: {relevant code snippets}
   Constraints:
   - Apply logical reasoning, not pattern matching
   - Consider alternative explanations for the evidence
   Output format:
   ## Verdict (CONFIRMED / ELIMINATED / INCONCLUSIVE)
   ## Reasoning
   ## Remaining Unknowns

   python3 .claude/skills/_shared/codex_consult.py --prompt-file .claude/logs/codex/prompt-troubleshoot-hypothesis.md --label troubleshoot-hypothesis

   ### 3. Fix Approach Design
   Write the prompt below to a file, then consult Codex for trade-off analysis of fix alternatives:

   Objective: Design and compare fix approaches for root cause: {root cause description}.
   Context:
   - Root cause: {description}
   - Affected code: {file:line}
   - Current behavior: {description}
   - Desired behavior: {description}
   Constraints:
   - Propose at least 2 approaches
   - Evaluate: correctness, minimal invasiveness, maintainability, performance
   - Consider backward compatibility
   Output format:
   ## Approach A: {name}
   ## Approach B: {name}
   ## Comparison Matrix
   ## Recommendation with Rationale

   python3 .claude/skills/_shared/codex_consult.py --prompt-file .claude/logs/codex/prompt-troubleshoot-fix-design.md --label troubleshoot-fix-design

   ### 4. Fix Correctness Verification
   Before finalizing, write the prompt below to a file, then consult Codex to verify the proposed fix:

   Objective: Verify that the proposed fix correctly resolves the root cause.
   Context:
   - Root cause: {description}
   - Proposed fix: {description}
   - Edge cases identified: {list}
   Constraints:
   - Check that the fix addresses the root cause, not just symptoms
   - Verify behavior under all identified trigger conditions
   - Check for new failure modes introduced by the fix
   Output format:
   ## Correctness Assessment (CORRECT / INCOMPLETE / INCORRECT)
   ## Edge Case Coverage
   ## New Failure Modes (if any)
   ## Confidence Level

   python3 .claude/skills/_shared/codex_consult.py --prompt-file .claude/logs/codex/prompt-troubleshoot-fix-verify.md --label troubleshoot-fix-verify

   Save analysis to `{paths.root_cause}` (from Phase 1 Step 0).

   Communicate with Impact Investigator teammate:
   - Share root cause findings that expand the affected scope
   - Request context about specific code paths or history
   - Confirm or refute hypotheses based on shared evidence

   IMPORTANT — Work Log:
   When ALL your tasks are complete, write your work log to
   {paths.team_dir}root-cause-analyst.md per the shared
   format: .claude/skills/_shared/work-log-format.md
   Keep all five core sections, `## Tasks Completed` included -- the Lead
   validates this log with `validate_doc.py --contract work-log`, which
   rejects a log that drops it.
   Role-specific sections (between Tasks Completed and Communication with
   Teammates) for this role:
   ## Hypotheses Evaluated
   - [confirmed/eliminated] {hypothesis}: {evidence}
   ## Root Cause
   - Defect: {description}
   - Location: {file:line}
   - Trigger condition: {when it occurs}
   ## Proposed Fixes
   - Approach A: {description} — {pros/cons}
   - Approach B: {description} — {pros/cons}
   - Recommended: {which and why}
   ## Codex Consultations
   - {question asked to Codex}: {key insight from response}
   "

2. **Impact Investigator** — Uses Opus with Git history, codebase search, WebSearch, and Codex for risk analysis
   Prompt: "You are the Impact Investigator for bug: {slug}.

   Your job: Determine the full scope and impact of this bug, and gather context for the fix.
   Consult Codex for regression risk reasoning and fix safety analysis.

   Bug Report: read `{paths.bug_report}` (written and validated in Phase 1 Step 3).

   Tasks:
   1. Trace the bug's origin in git history:
      - git log / git bisect to find the introducing commit
      - What change caused this? Was it intentional?
   2. Assess blast radius:
      - What other code paths call the affected function?
      - What features/users are impacted?
      - Are there related bugs or similar patterns elsewhere?
   3. Research external context:
      - Is this a known issue in a dependency? (WebSearch)
      - Are there upstream fixes or workarounds?
      - Check issue trackers, changelogs, migration guides
   4. Evaluate regression risk:
      - What tests cover the affected area?
      - What could break if we change this code?
      - Are there downstream consumers to consider?

   How to research:
   - Use Git commands (git log, git blame, git bisect) for history
   - Use Grep/Glob for codebase impact analysis
   - Use WebSearch for external known issues:
     WebSearch: '{library} {error message} issue fix'

   ## Codex Risk Analysis Protocol (MANDATORY)

   You MUST consult Codex for regression risk reasoning and fix safety analysis.
   Each consultation below follows the same shape: write the prompt to a file,
   then run `python3 .claude/skills/_shared/codex_consult.py --prompt-file <path> --label <label>`
   and read the JSON `response_file`.

   ### Regression Risk Reasoning
   Write the prompt below to a file, then consult Codex to evaluate what could break if the proposed change is applied:

   Objective: Evaluate regression risk if {proposed change} is applied to {file:line}.
   Context:
   - Current behavior: {description}
   - Proposed change: {description}
   - Callers of affected function: {list}
   - Existing test coverage: {description}
   Constraints:
   - Consider all callers and downstream consumers
   - Identify implicit contracts that may be violated
   - Assess backward compatibility impact
   Output format:
   ## Risk Assessment (HIGH / MEDIUM / LOW)
   ## Affected Code Paths
   ## Implicit Contracts at Risk
   ## Recommended Safeguards

   python3 .claude/skills/_shared/codex_consult.py --prompt-file .claude/logs/codex/prompt-troubleshoot-regression.md --label troubleshoot-regression

   ### Fix Safety Analysis
   Write the prompt below to a file, then consult Codex to verify the proposed fix does not introduce new issues:

   Objective: Analyze whether the proposed fix introduces new issues or side effects.
   Context:
   - Root cause: {from Root Cause Analyst}
   - Proposed fix: {description}
   - Blast radius: {affected code paths}
   - Dependencies: {upstream/downstream}
   Constraints:
   - Check for new edge cases created by the fix
   - Verify thread safety if applicable
   - Check for performance implications
   Output format:
   ## Safety Assessment (SAFE / CAUTION / UNSAFE)
   ## New Issues Identified
   ## Side Effects
   ## Mitigation Recommendations

   python3 .claude/skills/_shared/codex_consult.py --prompt-file .claude/logs/codex/prompt-troubleshoot-fix-safety.md --label troubleshoot-fix-safety

   Save findings to `{paths.impact}` (from Phase 1 Step 0).

   Communicate with Root Cause Analyst teammate:
   - Share git history context that informs root cause
   - Share external findings (known issues, upstream fixes)
   - Request clarification on which code paths to investigate

   IMPORTANT — Work Log:
   When ALL your tasks are complete, write your work log to
   {paths.team_dir}impact-investigator.md per the shared
   format: .claude/skills/_shared/work-log-format.md
   Keep all five core sections, `## Tasks Completed` included -- the Lead
   validates this log with `validate_doc.py --contract work-log`, which
   rejects a log that drops it.
   Role-specific sections (between Tasks Completed and Communication with
   Teammates) for this role:
   ## Git History
   - Introducing commit: {hash} — {description}
   - Related commits: {list}
   ## Blast Radius
   - Affected code paths: {list}
   - Affected features/users: {list}
   ## External Research
   - {source}: {finding and relevance}
   ## Regression Risk
   - Existing test coverage: {description}
   - Risk areas: {what could break}
   ## Codex Risk Analysis
   - Regression risk assessment: {Codex's verdict and reasoning}
   - Fix safety assessment: {Codex's verdict and reasoning}
   "

Wait for both teammates to complete their tasks.

Example interaction flow:

Root Cause Analyst: "The error occurs because parse_config() returns None when key is missing"
    -> Impact Investigator: "Checking git blame -- this was changed in commit abc123"
    -> Impact Investigator: "Found 5 other callers of parse_config() that don't handle None"
    -> Root Cause Analyst: "Expanding fix scope -- need to either fix callers or fix parse_config()"
    -> Root Cause Analyst: "Codex recommends: fix parse_config() to raise KeyError instead of returning None"
    -> Impact Investigator: "Codex risk analysis confirms: all 5 callers already have try/except for KeyError"
    -> Root Cause Analyst: "Root cause confirmed. Codex verified fix correctness. Fix approach: restore KeyError in parse_config()"

python3 .claude/skills/_shared/workspace.py --skill troubleshoot --slug {slug} --verify
python3 .claude/skills/_shared/validate_doc.py --contract work-log \
  --dir {paths.team_dir} --expect-files 2

Objective: Validate this fix plan for completeness and correctness.
Context:
- Root cause: {from Root Cause Analyst}
- Proposed fix: {recommended approach}
- Blast radius: {from Impact Investigator}
- Fix tasks: {task list}
Constraints:
- Check for missing edge cases
- Verify the fix addresses the root cause (not just symptoms)
- Identify potential new issues the fix could introduce
- Suggest additional test cases if needed
Output format:
## Validation Result (PASS / NEEDS_REVISION)
## Missing Coverage
## Potential New Issues
## Additional Test Cases Recommended
## Revised Task List (if needed)

python3 .claude/skills/_shared/codex_consult.py --prompt-file .claude/logs/codex/prompt-troubleshoot-plan-validation.md --label troubleshoot-plan-validation