Recursive Debugging

recursive-debugging (Phase 1.5)

Overview

When a requirement involves fixing a bug or investigating unexpected behavior, ad-hoc fixes waste time and create new bugs. Systematic debugging finds the root cause before any fix is attempted.

Core Principle: ALWAYS find root cause before attempting fixes. Symptom fixes are failure.

The Iron Law for recursive-mode debugging:

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST

Trigger examples

• Tests are failing after the last change; debug it
• Fix crash on empty input
• Investigate why the API returns wrong data
• Do a root cause analysis before making changes

When to Use

Mandatory Phase 1.5 when:

• Requirement is a bug fix
• Test failures need investigation
• Unexpected behavior reported
• Performance problems
• Integration issues

Use ESPECIALLY when:

• Under time pressure (emergencies make guessing tempting)
• "Just one quick fix" seems obvious
• Previous fix attempts failed
• You don't fully understand the issue

Don't skip when:

• Issue seems simple (simple bugs have root causes too)
• You're in a hurry (rushing guarantees rework)
• Manager wants it fixed NOW (systematic is faster than thrashing)

Phase 1.5 Insertion

Phase 1.5 is inserted between Phase 1 (AS-IS) and Phase 2 (TO-BE Plan) when debugging is required:

Phase 0: 00-requirements.md
    ->
Phase 1: 01-as-is.md (captures current behavior)
    ->
Phase 1.5: 01.5-root-cause.md <- NEW (this skill)
    ->
Phase 2: 02-to-be-plan.md (includes fix plan based on root cause)

The Four Phases of Systematic Debugging

digraph debugging_phases {
    rankdir=TB;
    
    phase1 [label="Step 1:\nRoot Cause Investigation", shape=box, style=filled, fillcolor="#ffcccc"];
    phase2 [label="Step 2:\nPattern Analysis", shape=box, style=filled, fillcolor="#ffffcc"];
    phase3 [label="Step 3:\nHypothesis & Testing", shape=box, style=filled, fillcolor="#ccffcc"];
    phase4 [label="Step 4:\nFix Implementation", shape=box, style=filled, fillcolor="#ccccff"];
    
    phase1 -> phase2 -> phase3 -> phase4;
    
    // Feedback loops
    phase3 -> phase1 [label="hypothesis\nfailed", style=dashed];
    phase4 -> phase1 [label="fix\nfailed", style=dashed];
}

Step 1: Root Cause Investigation

BEFORE attempting ANY fix:

1.1 Read Error Messages Carefully

• Don't skip past errors or warnings
• They often contain the exact solution
• Read stack traces completely
• Note line numbers, file paths, error codes

Record in Phase 1.5 artifact:

## Error Analysis

**Error Message:** [verbatim]
**Stack Trace:** [key frames]
**File:Line:** [locations]
**Error Code:** [if applicable]
**Key Insight:** [what the error is telling you]

1.2 Reproduce Consistently

• Can you trigger it reliably?
• What are the exact steps?
• Does it happen every time?
• If not reproducible -> gather more data, don't guess

Record in Phase 1.5 artifact:

## Reproduction Verification

**Steps:**
1. [exact step]
2. [exact step]
3. [exact step]

**Reproducible:** Yes / No / Intermittent
**Frequency:** [X out of Y attempts]
**Deterministic:** Yes / No

1.3 Check Recent Changes

• What changed that could cause this?
• Git diff, recent commits
• New dependencies, config changes
• Environmental differences

Record in Phase 1.5 artifact:

## Recent Changes Analysis

**Git History:** [relevant commits]
**Dependency Changes:** [package.json, requirements.txt, etc.]
**Config Changes:** [relevant files]
**Environment:** [OS, runtime versions]
**Likely Culprit:** [most suspicious change]

1.4 Gather Evidence in Multi-Component Systems

WHEN system has multiple components (CI -> build -> signing, API -> service -> database):

BEFORE proposing fixes, add diagnostic instrumentation:

For EACH component boundary:

• Log what data enters component
• Log what data exits component
• Verify environment/config propagation
• Check state at each layer

Run once to gather evidence showing WHERE it breaks, THEN analyze evidence.

Record in Phase 1.5 artifact:

## Multi-Layer Evidence

**Layer 1: [Component Name]**
- Input: [data]
- Output: [data]
- Status: WORKING / BROKEN

**Layer 2: [Component Name]**
- Input: [data from Layer 1]
- Output: [data]
- Status: WORKING / BROKEN

**Failure Boundary:** Layer X -> Layer Y
**Root Cause Location:** [specific component]

1.5 Trace Data Flow

WHEN error is deep in call stack:

Trace backward:

• Where does bad value originate?
• What called this with bad value?
• Keep tracing up until you find the source
• Fix at source, not at symptom

Record in Phase 1.5 artifact:

## Data Flow Trace

**Error Location:** [file:line - function]
**Bad Value:** [what was wrong]

**Call Stack Trace:**
1. [deepest] `functionA()` at fileA:line - received [value]
2. `functionB()` at fileB:line - passed [value]
3. `functionC()` at fileC:line - passed [value]
4. [source] `functionD()` at fileD:line - ORIGIN of bad value

**Root Cause:** [source location] - [explanation]

Step 2: Pattern Analysis

Find the pattern before fixing:

2.1 Find Working Examples

• Locate similar working code in same codebase
• What works that's similar to what's broken?

2.2 Compare Against References

• If implementing pattern, read reference implementation COMPLETELY
• Don't skim - read every line
• Understand the pattern fully before applying

2.3 Identify Differences

• What's different between working and broken?
• List every difference, however small
• Don't assume "that can't matter"

2.4 Understand Dependencies

• What other components does this need?
• What settings, config, environment?
• What assumptions does it make?

Record in Phase 1.5 artifact:

## Pattern Analysis

**Working Example:** [file:location]
**Broken Code:** [file:location]

**Key Differences:**
| Aspect | Working | Broken |
|--------|---------|--------|
| [X] | [value] | [value] |
| [Y] | [value] | [value] |

**Likely Cause:** [difference that explains the bug]
**Dependencies:** [what the code needs to work]

Step 3: Hypothesis and Testing

Scientific method:

3.1 Form Single Hypothesis

• State clearly: "I think X is the root cause because Y"
• Write it down
• Be specific, not vague

3.2 Test Minimally

• Make the SMALLEST possible change to test hypothesis
• One variable at a time
• Don't fix multiple things at once

3.3 Verify Before Continuing

• Did it work? Yes -> Phase 4
• Didn't work? Form NEW hypothesis
• DON'T add more fixes on top

3.4 When You Don't Know

• Say "I don't understand X"
• Don't pretend to know
• Ask for help
• Research more

Record in Phase 1.5 artifact:

## Hypothesis Testing

### Hypothesis 1
**Statement:** [clear hypothesis]
**Rationale:** [why you think this]
**Test:** [minimal change to verify]
**Result:** [confirmed/rejected]
**Evidence:** [output/observation]

### Hypothesis 2 (if needed)
[...]

**Confirmed Root Cause:** [final hypothesis]

Step 4: Fix Summary (Handoff to Phase 2 Planning)

Fix the root cause, not the symptom:

4.1 Create Failing Test Case

• Simplest possible reproduction
• Automated test if possible
• One-off test script if no framework
• MUST have before fixing
• This becomes part of Phase 2's test strategy