skills/diagnose-systematically/root-cause-tracing.md

# Root Cause Tracing

## Overview

Bugs often manifest deep in the call stack (git init in wrong directory, file created in wrong location, database opened with wrong path). Your instinct is to fix where the error appears, but that's treating a symptom.

**Core principle:** Trace backward through the call chain until you find the original trigger, then fix at the source.

## When to Use

When a bug appears deep in the stack and the trail can be followed backwards, trace it to the original trigger and fix there. Fix at the point where the error appears only when the backward trail truly dead-ends. After fixing at the source, consider adding layered validation as well (see `defense-in-depth.md`).

**Use when:**
- Error happens deep in execution (not at entry point)
- Stack trace shows long call chain
- Unclear where invalid data originated
- Need to find which test/code triggers the problem

## The Tracing Process

### 1. Observe the Symptom
```
Error: git init failed in ~/project/packages/core
```

### 2. Find Immediate Cause
**What code directly causes this?**
```typescript
await execFileAsync('git', ['init'], { cwd: projectDir });
```

### 3. Ask: What Called This?
```typescript
WorktreeManager.createSessionWorktree(projectDir, sessionId)
  → called by Session.initializeWorkspace()
  → called by Session.create()
  → called by test at Project.create()
```

### 4. Keep Tracing Up
**What value was passed?**
- `projectDir = ''` (empty string!)
- Empty string as `cwd` resolves to `process.cwd()`
- That's the source code directory!

### 5. Find Original Trigger
**Where did empty string come from?**
```typescript
const context = setupCoreTest(); // Returns { tempDir: '' }
Project.create('name', context.tempDir); // Accessed before beforeEach!
```

## Adding Stack Traces

When you can't trace manually, add instrumentation:

```typescript
// Before the problematic operation
async function gitInit(directory: string) {
  const stack = new Error().stack;
  console.error('DEBUG git init:', {
    directory,
    cwd: process.cwd(),
    nodeEnv: process.env.NODE_ENV,
    stack,
  });

  await execFileAsync('git', ['init'], { cwd: directory });
}
```

**Critical:** Use `console.error()` in tests (not logger - may not show)

**Run and capture:**
```bash
npm test 2>&1 | grep 'DEBUG git init'
```

**Analyze stack traces:**
- Look for test file names
- Find the line number triggering the call
- Identify the pattern (same test? same parameter?)

## Locating the Failing Layer in Multi-Component Systems

When the path runs through several components (CI → build → deploy, API → service → database), instrument every boundary before proposing fixes: log what enters and what exits each component, and confirm config/environment propagation at each hop. One instrumented run shows which layer the data dies in; investigate that layer.

```bash
# Layer 1: entry point - is the value present at all?
if [ -n "${API_KEY:-}" ]; then echo "API_KEY: SET"; else echo "API_KEY: UNSET"; fi

# Layer 2: intermediate script - did it propagate?
if [ "${API_KEY+x}" = x ]; then echo "API_KEY propagated: SET"; else echo "API_KEY propagated: UNSET"; fi

# Layer 3: consumer - what does the application actually see?
node -e "console.log('key seen by app:', process.env.API_KEY ? 'SET' : 'UNSET')"
```

This reveals which hop drops the value (entry ✓, intermediate ✗) instead of guessing at the whole chain.
Log only presence or absence. Never print the credential itself while tracing propagation.

## Finding Which Test Causes Pollution

If something appears during tests but you don't know which test:

Use the polluter-finder script `find-polluter.sh` in this directory:

```bash
./find-polluter.sh '.git' 'src/**/*.test.ts'
```

Runs tests one-by-one, stops at first polluter. See script for usage.

## Real Example: Empty projectDir

**Symptom:** `.git` created in `packages/core/` (source code)

**Trace chain:**
1. `git init` runs in `process.cwd()` ← empty cwd parameter
2. WorktreeManager called with empty projectDir
3. Session.create() passed empty string
4. Test accessed `context.tempDir` before beforeEach
5. setupCoreTest() returns `{ tempDir: '' }` initially

**Root cause:** Top-level variable initialization accessing empty value

**Fix:** Made tempDir a getter that throws if accessed before beforeEach

**Also added defense-in-depth:**
- Layer 1: Project.create() validates directory
- Layer 2: WorkspaceManager validates not empty
- Layer 3: NODE_ENV guard refuses git init outside tmpdir
- Layer 4: Stack trace logging before git init

## Key Principle

From the immediate cause, keep asking what called this, and with what value, one level at a time, until the answer is the source rather than another caller. Fix at the source, then add validation at each layer the bad value passed through so the bug becomes structurally impossible.

**NEVER fix just where the error appears.** Trace back to find the original trigger.

## Stack Trace Tips

**In tests:** Use `console.error()` not logger - logger may be suppressed
**Before operation:** Log before the dangerous operation, not after it fails
**Include context:** Directory, cwd, environment variables, timestamps
**Capture stack:** `new Error().stack` shows complete call chain