Investigate

Investigate

Systematic methodology for finding the root cause of bugs, failures, and unexpected behavior. Cycle through characterize-isolate-hypothesize-test steps, with oracle escalation for hard problems. Diagnose the root cause — do not apply fixes.

Optional: $ARGUMENTS contains the problem description or error message.

Step 1: Characterize

Gather the symptom and establish what is actually happening:

1. Collect evidence — error message, stack trace, test output, log entries, or user description of unexpected behavior
2. Classify the problem type:

3. Establish reproduction — run the failing command, test, or operation. If the problem cannot be reproduced (intermittent, environment-specific), document the constraints and proceed with historical evidence.

Record the exact reproduction command and its output for verification. For intermittent or long-running reproductions, tail logs in a background shell, filtered for relevant signals (errors, stack traces, specific identifiers) so failures surface live while you work.

Step 2: Isolate

Narrow from "something is wrong" to "the problem is in this area." Read references/problem-type-playbooks.md for type-specific first moves and tool sequences.

Git Archeology

For all problem types, check what changed recently near the failure point:

git log --oneline -20 -- <file>
git blame -L <start>,<end> <file>

If a known-good state exists (e.g., "this worked yesterday"), consider git bisect to pinpoint the breaking commit.

When the problem description names a version, tag, or build that differs from the ref under investigation, resolve it to a ref and read what changed on the failing path between the two before generating hypotheses:

git diff <reported-ref>..HEAD -- <failing-path-files>

Read the full diff rather than its --stat summary. Carry each difference that could produce the symptom forward as a ranked hypothesis.

Upstream Issue Search

When the failure surfaces inside a third-party dependency, search its issue tracker for a distinctive string from the error before reading deeper into the dependency's code. An issue whose symptom matches often names the cause and the fix outright. Carry a match forward as a ranked hypothesis and test it.

Scope Narrowing

• Stack traces: Read the throwing function and its callers — full functions, not just the flagged line
• Test failures: Read both the test and the system under test
• Build errors: Read the config file and the referenced source
• Unexpected behavior: Trace the data flow from input to the unexpected output

Before treating a record, file, or build artifact as evidence of the system's behavior, confirm the system under test produced it: check creator, source metadata, or generation time. Suspect imported, seeded, hand-edited, and leftover data from an earlier run, which reads identically to generated output. A checkout of another repository is the same trap: confirm it is current before reading it as evidence, since a stale one reads identically to the authoritative source.

Step 3: Hypothesize

Before forming a hypothesis about the machinery around a failure, such as a toolchain version, a configuration policy, or an environment difference, read the failing line, identify every path, package, symbol, or resource it names, and confirm each one resolves. Error text often names the site that consumed a missing input rather than the input itself, so the surrounding machinery looks responsible when it is not. Rank a machinery hypothesis only after every named reference checks out.

Once every named reference resolves and the operation has never once succeeded, rank a refusal ahead of any race or resource-exhaustion hypothesis: a denied permission, a firewall rule, an allowlist, an expired or missing credential. Intermittent failure is what a race or a contended resource usually looks like, so a run of attempts with zero successes ranks both below a refusal. Carry the hard blocks on the failing path into Step 4 as the first hypotheses to test, ahead of any measurement or instrumentation.

Generate 2-4 hypotheses ranked by likelihood. Each hypothesis must be falsifiable — specify what evidence would confirm or refute it.

Format:

H1 (most likely): [description] — confirmed if [X], refuted if [Y]
H2: [description] — confirmed if [X], refuted if [Y]
H3: [description] — confirmed if [X], refuted if [Y]

Check that the observed case can discriminate: when confirming and refuting evidence would look identical in it, the case is degenerate and any verdict drawn from it is inconclusive. Degenerate cases hide the difference they are supposed to reveal, such as a scaling factor of 1, a single-element collection, or an identity transform. Find a non-degenerate case, or construct one as a Step 4 experiment.

Parallel Investigation

For complex problems with 3+ hypotheses and a non-obvious root cause, spawn parallel investigators simultaneously.

Spawn condition: 3+ hypotheses AND the problem is not a simple typo, missing import, or syntax error.

Skip when 1-2 hypotheses are obvious (e.g., stack trace points directly to the bug).

Before dispatching, read the project's test configuration and CI workflow to identify any test tier that resets a shared external resource between tests, such as a database, a fixed port, or a cache. Such tiers have no cross-process interlock, so branches running them concurrently wipe each other's state and return failures that look like real defects. Name any such tier to every branch as off-limits.

When the evidence lives in a repository other than this one, including a submodule or a vendored clone with its own remote, establish the authoritative ref before dispatching and bring it up to date, fetching it or reading it through the forge API. A local checkout may be behind its remote, and an investigation branch reading a stale one returns findings the current code has already resolved. Name that ref and how to read it in every branch prompt, including the text the Claude consultation branch forwards.

Launch all investigation branches with spawn_agent / wait_agent using inherited model defaults, issuing every call in one batch. Do not issue one and await its result before issuing the rest. Expect one branch per hypothesis plus one Claude consultation branch. Every branch prompt must direct it to treat the shared working tree and its git index as read-only and to gather evidence by reading and reasoning; experiments that mutate code wait for Step 4, where they run one at a time. HEAD stays where it is: read other refs with git show <ref>:<path> rather than git checkout or git switch.

• Hypothesis branch (one per hypothesis): Each receives the hypothesis, relevant file paths, what evidence to look for, and instructions to report confirmed / refuted / inconclusive with evidence. Budget: max 5 tool calls per branch.
• Claude consultation branch: Run the $consult-claude skill with a focused prompt describing the problem, reproduction, and files examined. The external perspective can dig into patterns the hypothesis-driven branches miss. Run the $evaluate-findings skill on its output after the consultation returns.

Once wait_agent has returned every hypothesis branch's report and only the Claude consultation branch has not reported, run the Step 4 actions that change nothing in the working tree, its git index, or anything the consultation's prompt points it at, and state each result as it lands. Then call wait_agent until the consultation's report arrives. The merge, every other Step 4 action, the Iteration decision, and the Investigation Report wait for that report.

After all investigators complete, merge results. Claude findings that overlap with a confirmed hypothesis reinforce confidence. Novel Claude findings become additional hypotheses to test in Step 4.