Root Cause Debugging

Root-Cause Debugging

Overview

Random fixes waste time and create new bugs. Quick patches mask underlying issues.

Core principle: ALWAYS find the root cause before attempting fixes. Symptom fixes are failure.

The Iron Law

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST

Use this for ANY technical issue: a task returned to need_revision, a failing test, a red pipeline, unexpected behavior. Use it ESPECIALLY under time pressure — systematic is faster than guess-and-check thrashing.

The Four Phases

Complete each phase before the next.

Phase 1 — Build a feedback loop, then investigate

This is the skill: before anything else, name ONE command you have already run that goes red on THIS symptom — a failing test, a curl against the running service, a browser_read_dom that doesn't find what it should, a replayed payload. No loop, no hypothesis. If none exists, write the smallest one that reproduces it (a test is the default; a curl/DOM read is the fallback when the bug is only visible live) before reading another line of code.

1. Read the feedback completely: the reviewer/QA/pipeline comment on the task, the full error output, stack traces, exact files/lines/messages. They often contain the exact answer.
2. Reproduce consistently in the task workspace with the loop from above: does it happen every time? Not reproducible → gather more data, don't guess.
3. Check recent changes: git diff, recent commits on the task branch, config changes.
4. Stage/prod symptom (a runtime error, not a local failure): before reading code, get_environment for the environment, list_runtime_errors with new: true to see whether the group started with the latest deploy, then query_runtime_logs with text: set to the error message. You hold these tools; use them before guessing from the code alone.
5. Multi-component systems (API → service → database): add diagnostic logging at each component boundary — what enters, what exits — run once, and locate WHICH layer breaks before touching anything. Tag every line you add with a unique marker, e.g. [DEBUG-a4f2], so cleanup is one grep_code for the tag before you finish — an untagged debug log left behind is a review finding.
6. Trace the bad value backward from where the error appears to where it originates. Fix at the source, not at the symptom.

Phase 2 — Pattern analysis

• Find working examples of the same pattern in the codebase (codebase_search, grep_code).
• Read the reference implementation completely — don't skim.
• List every difference between working and broken; don't assume "that can't matter".

Phase 3 — Hypothesis and testing

• List 3–5 hypotheses, ranked by likelihood. Each must be falsifiable: "If X is the cause, then changing Y will make the symptom disappear."
• Test the top one with the SMALLEST possible change. One variable at a time.
• Didn't work? Move to the next ranked hypothesis. Do NOT stack more fixes on top.

Phase 4 — Implementation

1. Write a failing test that reproduces the issue (see tdd-workflow).
2. Implement the single fix that addresses the root cause. No bundled refactoring.
3. Verify: the test passes, no other test breaks, the original symptom is gone.
4. Address EVERY point from the revision comment explicitly — partial fixes come straight back.

Was the test red before you started?

Before fixing a test you didn't write that's failing on your branch, check whether it failed on the base commit too: git stash -u && <focused test command>; git stash pop. If it was already red there, it's pre-existing, not something your change broke.

• Small and adjacent to your change → fix it in its own commit, separate from your feature commit.
• Otherwise → one comment naming the test, the failure, and the base commit SHA: that's a blocker a person must see, not something to silently carry or silently fix.

A regression that appeared somewhere in recent history (not clearly your change): git bisect run <focused test> to find the exact commit before theorizing about the cause.