Structured Debugging

Structured Debugging

When debugging hard issues, the natural instinct is to form a theory and immediately apply a fix. This fails more often than it works. The fix addresses the wrong cause, adds complexity, creates false confidence, and obscures the real issue. Worse, after several failed attempts you lose track of what's been tried and start guessing randomly.

This methodology replaces guessing with a disciplined cycle that converges on the root cause. Each iteration narrows the search space. It's slower per attempt but dramatically faster overall because you stop wasting runs on wrong theories.

The Cycle

1. Hypothesize

Before touching code, write down what you think is happening and why. Be specific about the expected state at each step in the execution path.

Bad: "Something is wrong with the wait loop." Good: "The leader hangs because hasActiveTeammates() returns true after all agents have reported completed, likely because terminal status isn't being set on the agent object after the backend process exits."

For bugs you expect to take more than one round, create a side note file for the investigation in whichever location the project uses for such notes.

Write your hypothesis there. This file persists across conversation turns and even across sessions — it's your investigation journal.

2. Design Instrumentation

Add targeted debug logs or assertions at the exact decision points that would confirm or reject your hypothesis. Think about what data you need to see.

Don't scatter console.log everywhere. Identify the 2-3 places where your hypothesis makes a testable prediction, and instrument those.

Prefer logging values (return codes, payload contents, stream types, message bodies, env state) over presence checks ("was this function called?", "was this branch taken?"). Code-path traces tell you what ran; data traces tell you what it ran on. Most non-trivial bugs are correct code processing wrong data.

Ask yourself: "If my hypothesis is correct, what will I see at point X? If it's wrong, what will I see instead?"

3. Verify Data Collection

Before running, confirm that your instrumentation output will actually be captured and accessible.

Common traps:

• stderr discarded by 2>/dev/null in the test command
• Process killed before flush (logs lost)
• Logging to a file in a directory that doesn't exist
• Output piped through something that truncates it
• Looking at log files from a previous run, not the current one

A test run that produces no data is wasted.

4. Run and Observe

Execute the test. Read the actual output — every line of it. Don't assume what it says.

When the data contradicts your hypothesis, believe the data. Don't rationalize it away. The whole point of this step is to let reality override your theory.

5. Document Findings

Update the side note with:

• What the data showed (quote specific log lines)
• What was confirmed vs. disproved
• Updated hypothesis for the next iteration

This is critical for not losing context across attempts. Hard bugs typically take 3-5 rounds. Without notes, you'll forget what you ruled out and waste runs re-checking things.

6. Iterate

Update the hypothesis based on the new evidence. Go back to step 2. Each round should narrow the search space.

If you're not making progress after 3 rounds, step back and question your assumptions. The bug might be in a layer you haven't considered.

Failure Modes to Avoid

These are the specific traps this methodology is designed to prevent. When you notice yourself drifting toward any of them, stop and return to the cycle.

Jumping to fixes without evidence

The most common failure. You have a plausible theory, so you "fix" it and run again. If the theory was wrong, you've added complexity, wasted a test run, and possibly introduced a new bug. The side note should always show "hypothesis verified by [specific data]" before any fix is applied.

Blaming external systems

"The model is hallucinating." "The API is flaky." "The library has a bug." These conclusions feel satisfying because they put the problem outside your control. They're also usually wrong.

Before blaming an external system, inspect what it actually received. A model that appears to hallucinate may be responding rationally to stale data you didn't know was there. An API that appears flaky may be receiving malformed requests. Look at the inputs, not just the outputs.