Systematic Debugging: A Four-Phase Methodology for Root Cause Fixes

This reference document defines a structured debugging methodology that mandates root cause investigation before any fix, using a tight feedback loop and four sequential phases to eliminate guesswork and prevent symptom-driven patches.

Purpose and When to Use

The methodology enforces root cause investigation before any fix attempt. It provides a tight feedback loop construction guide to reproduce the exact symptom, defines four sequential phases (Root Cause Investigation, Pattern Analysis, Hypothesis & Testing, Implementation), includes a Rule of Three (after 3 failed fixes, stop and question the architecture), and lists red flags and common rationalizations to prevent process violations.

Use this methodology for any technical issue: test failures, production bugs, unexpected behavior, performance problems, build failures, and integration issues. It is especially important under time pressure, when a quick fix seems obvious, after multiple failed fixes, when a previous fix did not work, or when the issue is not fully understood. Do not skip the process for simple issues, when in a hurry, or when someone demands an immediate fix.

Prerequisites

Before starting, ensure you have:

• Access to the codebase and relevant source files.
• Ability to run commands (terminal, tests, scripts).
• Ability to search files and read source code (search_files, read_file).
• For multi-component systems, ability to add diagnostic instrumentation (logs, state checks).
• Familiarity with git for checking recent changes.
• For flaky bugs, ability to run high-repetition loops or add stress/timing control.

Phase 1: Root Cause Investigation

This phase enforces the Iron Law of debugging: no fixes without root cause investigation first.

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST

Step 1: Read Error Messages Carefully

Note line numbers, file paths, error codes. Use read_file and search_files on relevant source to understand the context of the error.

Step 2: Build a Tight Feedback Loop

A tight feedback loop is a single command that goes red on the exact symptom and green when fixed. It must be fast, deterministic, agent-runnable, and specific. If the bug is hard to reproduce, spend disproportionate effort on the loop. For non-deterministic bugs, aim for a high reproduction rate (e.g., 50%) before debugging; a 1% flake is usually not debuggable.

# Run a specific failing test
pytest tests/test_module.py::test_name -v

# Or run a scripted repro
python scripts/repro_bug.py

# Or run a high-repetition flaky repro
for i in {1..100}; do pytest tests/test_flake.py::test_name -q || break; done

Step 3: Check Recent Changes

Review recent commits and changes to identify what might have introduced the bug.

# Recent commits
git log --oneline -10

# Uncommitted changes
git diff

# Changes in specific file
git log -p --follow src/problematic_file.py | head -100

For regressions introduced by a specific commit, use git bisect run to automate the search.

Step 4: Add Diagnostic Instrumentation in Multi-Component Systems

In multi-component systems, add diagnostic instrumentation at each component boundary: log inputs, outputs, environment/config, state. Run once to gather evidence, then analyze to identify the failing component.

Step 5: Trace Data Flow Upstream

Trace data flow upstream from the error. Find where bad values originate, what called the function, and continue until the source is found. Use search_files to trace references.

# Find where the function is called
search_files("function_name(", path="src/", file_glob="*.py")

# Find where the variable is set
search_files("variable_name\\s*=", path="src/", file_glob="*.py")

Phase 1 Checklist

Complete this checklist before moving to Phase 2:

• Errors read and understood.
• Tight loop exists and is red-capable.
• Loop is deterministic, or flaky bug has high reproduction rate.
• Recent changes reviewed.
• Evidence gathered from instrumentation.
• Problem isolated to specific component/code.
• Root cause hypotheses can be stated and tested.

Phase 2: Pattern Analysis

Step 1: Minimize the Reproduction

Shrink the repro to the smallest scenario that still goes red by cutting inputs, callers, config, data, and steps one at a time, re-running the loop after each cut. Stop when removing any remaining element makes the loop green.

Step 2: Find Working Examples

Locate similar working code in the same codebase using search_files.

search_files("similar_pattern", path="src/", file_glob="*.py")

Step 3: Compare Against References

If implementing a pattern, read the reference implementation completely, every line. Partial understanding of a reference pattern can lead to new bugs.

Step 4: Identify Differences

List every difference between working and broken code, no matter how small. Do not assume "that can't matter."

Step 5: Understand Dependencies

Identify what other components, settings, config, environment, and assumptions are needed for the code to work correctly.

Phase 3: Hypothesis and Testing

Step 1: Form 3–5 Ranked Hypotheses

Before testing any single hypothesis, form 3–5 ranked falsifiable hypotheses. Rank by likelihood and cheapness to falsify. Each must state a testable prediction. If a hypothesis does not make a testable prediction, discard or sharpen it. If the user is present, show the ranked list before testing.

Step 2: Test the Highest-Ranked Hypothesis

Test with the smallest possible probe. Change one variable at a time. Prefer debugger/REPL inspection; if adding logs, tag each temporary line with a unique prefix like [DEBUG-a4f2].

Step 3: Verify

If the fix works, proceed to Phase 4. If not, form a new hypothesis. Do not add more fixes on top. If you don't know, say "I don't understand X," ask the user, or research more using web_search and web_extract.

Phase 4: Implementation

Step 1: Create a Failing Test Case

Write the simplest possible automated reproduction of the bug before fixing. Use the test-driven-development skill.

Step 2: Implement a Single Fix

Implement a single fix addressing the root cause. One change at a time. No "while I'm here" improvements or bundled refactoring.

Step 3: Verify the Fix

Run the specific regression test, then run the full test suite to check for regressions.

# Run the specific regression test
pytest tests/test_module.py::test_regression -v

# Run full suite — no regressions
pytest tests/ -q

Step 4: Apply the Rule of Three

If the fix does not work, count the number of distinct fix attempts made. If fewer than 3, return to Phase 1. If 3 or more, stop and question the architecture. Do not attempt Fix #4 without architectural discussion.

When questioning the architecture, look for patterns like:

• Each fix revealing new shared state/coupling.
• Fixes requiring massive refactoring.
• Each fix creating new symptoms elsewhere.

Discuss with the user before attempting more fixes.

Multi-Component Debugging with Delegate Task

For multi-component debugging, use delegate_task to investigate issues in other components without fixing them prematurely.

delegate_task(
    goal="Investigate why [specific test/behavior] fails",
    context="""
    Follow systematic-debugging skill:
    1. Read the error message carefully
    2. Reproduce the issue
    3. Trace the data flow to find root cause
    4. Report findings — do NOT fix yet

    Error: [paste full error]
    File: [path to failing code]
    Test command: [exact command]
    """,
    toolsets=['terminal', 'file']
)

Red Flags and Rationalizations

Be alert for these red flags that indicate the process is being violated:

• "Quick fix for now."
• "Just try changing X."
• "Skip the test."
• "We'll fix it properly later."

When you encounter these, stop and return to Phase 1. Common rationalizations to avoid include:

• "This is too simple for the full process."
• "We're under too much time pressure."
• "Someone demands an immediate fix."

Failure Modes

The following failure modes are common when the methodology is not followed:

• Guessing without a red-capable loop.
• Symptom fixes that mask underlying issues.
• Multiple fixes at once, preventing isolation of what worked.
• Skipping the process under time pressure, leading to thrashing and rework.
• Continuing to fix after 3+ failures without questioning the architecture.
• Partial understanding of a reference pattern, leading to new bugs.
• Assuming "that can't matter" when comparing working vs. broken code.
• Adding logs without unique prefixes, making cleanup difficult.
• Failing to create a regression test before fixing, so the fix is not proven.

Expected Outcomes

When followed correctly, the methodology produces:

• Systematic approach: 15-30 minutes to fix; random fixes: 2-3 hours of thrashing.
• First-time fix rate: 95% vs 40%.
• New bugs introduced: near zero vs common.

Summary of Parameters

The methodology relies on these key parameters:

• Tight feedback loop command: A single command that reproduces the exact symptom (red) and passes only when the bug is fixed (green). Must be built in Phase 1 before any fix.
• Number of failed fix attempts: Count of distinct fix attempts made. Used in the Rule of Three to decide whether to question the architecture.
• Hypothesis list: 3–5 ranked falsifiable hypotheses with testable predictions, generated before testing any single one.
• Failing test case: Simplest possible automated reproduction of the bug, written before the fix.

Constraints and Caveats

• No fixes without root cause investigation first (Iron Law).
• Must complete each phase before proceeding to the next.
• Do not skip the process for simple issues, emergencies, or when someone demands an immediate fix.
• Do not propose solutions before tracing data flow.
• Do not make multiple changes at once; change one variable at a time.
• Do not add "while I'm here" improvements or bundled refactoring.
• After 3 failed fixes, stop and question the architecture; do not attempt Fix #4 without discussion.
• For non-deterministic bugs, aim for a high reproduction rate (e.g., 50%) before debugging; 1% flake is usually not debuggable.
• If a hypothesis does not make a testable prediction, discard or sharpen it.
• If you don't understand something, say so and ask for help.
• Red flags (e.g., "quick fix for now", "just try changing X", "skip the test") mean stop and return to Phase 1.