@qball-inc/the-bulwark 1.0.0

package/skills/issue-debugging/SKILL.md

@@ -0,0 +1,385 @@
+---
+name: issue-debugging
+description: Systematic methodology for issue debugging including root cause analysis, impact mapping, tiered validation plans, and confidence assessment. Use when analyzing bugs, fixing issues, or validating fixes.
+user-invocable: false
+---
+
+# Issue Debugging
+
+Systematic methodology for debugging that prevents the "first error fix and declare done" anti-pattern. Ensures root cause analysis, impact assessment, and verified fixes.
+
+---
+
+## When to Use This Skill
+
+**Load this skill when the user request matches ANY of these patterns:**
+
+| Trigger Pattern | Example User Request |
+|-----------------|---------------------|
+| Bug investigation | "Why is X failing?", "Debug this error", "Investigate this issue" |
+| Test failures | "Tests are failing", "Fix the broken tests", "Why did CI fail?" |
+| Error analysis | "Getting this error...", "Exception in...", "Stack trace shows..." |
+| Regression hunting | "This used to work", "Something broke after...", "Regression in..." |
+| Fix validation | "Did the fix work?", "Verify this change", "Make sure it's fixed" |
+| Root cause requests | "What's causing this?", "Find the root cause", "Why does this happen?" |
+
+**DO NOT use for:**
+- New feature implementation (no existing bug)
+- Refactoring without reported issues
+- General code questions
+
+---
+
+## Overview
+
+### The Problem
+
+AI (and developers) often fix the first visible error and declare victory without verifying end-user functionality works. This leads to:
+- Symptom fixes instead of root cause fixes
+- Regressions in seemingly unrelated areas
+- False confidence in fix completeness
+
+### The Solution
+
+This skill provides:
+1. **Root Cause Analysis** - Find the actual cause, not just symptoms
+2. **Impact Analysis** - Map affected areas and dependencies
+3. **Validation Planning** - Tiered test priorities (P1/P2/P3)
+4. **Confidence Assessment** - Objective fix reliability rating
+5. **Escalation Triggers** - When manual testing is required
+
+---
+
+## Root Cause Analysis
+
+### The 5 Whys Method
+
+Iteratively ask "Why?" to drill past symptoms to root cause:
+
+```
+Problem: Login fails with 500 error
+├─ Why? → API returns null for user profile
+├─ Why? → Database query found no records
+├─ Why? → Migration didn't run on deploy
+├─ Why? → Deployment script skipped migration step
+└─ Why? → Environment variable missing
+
+Root Cause: Missing environment variable (not the 500 error)
+```
+
+**Key Principle**: Focus on HOW and WHY the defect occurred, not just WHERE.
+
+### Hypothesis-Driven Debugging
+
+Apply scientific method:
+
+| Step | Action |
+|------|--------|
+| **Observe** | Gather data: errors, logs, stack traces, reproduction steps |
+| **Hypothesize** | Form a **falsifiable** hypothesis about the cause |
+| **Experiment** | Design tests to prove/disprove hypothesis |
+| **Conclude** | If confirmed, proceed. If rejected, form new hypothesis |
+| **Document** | Write down hypotheses tested (prevents circular investigation) |
+
+### Binary Search / Bisection
+
+For isolating bugs in time or space:
+1. Place checkpoint at midpoint
+2. Determine if bug is before or after
+3. Repeat on remaining half
+4. Continue until exact location found
+
+**Efficiency**: O(log n) - approximately 7 tests for 100 commits.
+
+---
+
+## Impact / Dependency Analysis
+
+### Mapping Procedure
+
+For each affected file, identify:
+
+```
+Affected Code: src/auth/login.ts
+
+Upstream (What calls this?):
+├─ src/api/auth-routes.ts → POST /login
+├─ src/middleware/session.ts → validateSession()
+└─ src/pages/login.tsx → handleSubmit()
+
+Downstream (What does this affect?):
+├─ User dashboard → fetches profile on load
+├─ Settings page → assumes profile exists
+└─ API responses → include user context
+```
+
+### Risk Scope Assessment
+
+| Scope | Definition | Validation Approach |
+|-------|------------|---------------------|
+| **Isolated** | Self-contained, no affected areas | Unit tests only |
+| **Medium** | 1-5 affected areas | Integration tests |
+| **Broad** | >5 affected areas, cross-cutting | Integration + E2E + manual |
+
+---
+
+## Debug Report Format
+
+### Location
+
+Debug reports are written to: `logs/debug-reports/{issue-id}-{timestamp}.yaml`
+
+This location is separate from standard agent logs to:
+- Enable easy reference by downstream pipeline stages
+- Keep debug context grouped by issue
+- Allow FixValidator to read validation plans
+
+### Schema Overview
+
+```yaml
+debug_report:
+  metadata:
+    issue_id: "{id}"
+    timestamp: "{ISO-8601}"
+    analyzer: "bulwark-issue-analyzer"
+    complexity: low | medium | high
+
+  analysis:
+    symptom: "{user-visible problem}"
+    root_cause: "{underlying reason}"
+    fix_approach: "{recommended approach}"
+
+  impact_analysis:
+    affected_files: [...]
+    upstream_dependencies: [...]
+    downstream_effects: [...]
+    risk_scope: isolated | medium | broad
+
+  validation_plan:
+    tests_to_execute:
+      - path: "{test file}"
+        reason: "{why this test}"
+        priority: 1  # P1=must, P2=should, P3=nice-to-have
+    functionalities_to_validate: [...]
+
+  confidence_criteria:
+    high: [...]
+    medium: [...]
+    low: [...]
+
+  debug_journey:  # Required for medium/high complexity
+    hypotheses_tested: [...]
+```
+
+**Full schema**: See `references/debug-report-schema.md`
+
+---
+
+## Validation Plan
+
+### Tiered Test Priorities
+
+| Priority | Definition | Action |
+|----------|------------|--------|
+| **P1 (Must)** | Direct tests of affected function | Always run |
+| **P2 (Should)** | Integration tests for upstream/downstream | Run if time/tokens allow |
+| **P3 (Nice-to-have)** | E2E tests, edge cases | Run if available |
+
+### Selection Criteria
+
+**P1 Tests** (always run):
+- Tests in the same file/module as the fix
+- Tests named after the affected function
+- Tests that were failing before the fix
+
+**P2 Tests** (run if possible):
+- Tests for upstream callers
+- Tests for downstream consumers
+- Tests for related functionality
+
+**P3 Tests** (run if available):
+- E2E tests that include the affected flow
+- Performance tests
+- Edge case tests
+
+### Functionality Validation
+
+Beyond tests, list user-level verifications:
+- "User can log in successfully"
+- "Dashboard displays correct data after login"
+- "Session persists across page refresh"
+
+---
+
+## Confidence Rubric
+
+### Assessment Criteria
+
+| Level | Criteria |
+|-------|----------|
+| **High** | All P1-P2 tests pass, no regression, at least one functionality verified |
+| **Medium** | P1 tests pass, some P2-P3 skipped or not applicable |
+| **Low** | Tests cannot reliably validate, broad risk scope with untested paths |
+
+### Determining Complexity
+
+| Complexity | Indicators |
+|------------|------------|
+| **Low** | Single file, isolated change, clear cause |
+| **Medium** | 2-5 files, some dependencies, requires investigation |
+| **High** | >5 files, cross-cutting, unclear cause, multiple hypotheses needed |
+
+---
+
+## Escalation Triggers
+
+Manual testing is **required** when ANY of:
+- Confidence level is `low`
+- Risk scope is `broad` AND confidence is not `high`
+- Any functionality cannot be validated via automated tests
+- Security-related fix without security test coverage
+
+### Escalation Message Format
+
+When escalation is triggered, the Orchestrator must inform the user:
+
+```
+Manual testing required for:
+- [functionality 1]
+- [functionality 2]
+
+Reason: [why automated validation insufficient]
+
+Recommended manual test steps:
+1. [step]
+2. [step]
+```
+
+---
+
+## Debug Journey Logging
+
+### When Required
+
+| Complexity | Debug Journey Required? |
+|------------|------------------------|
+| Low | Optional |
+| Medium | **Required** |
+| High | **Required** |
+
+### Format
+
+```yaml
+debug_journey:
+  started_at: "{timestamp}"
+  completed_at: "{timestamp}"
+
+  hypotheses_tested:
+    - hypothesis: "Database connection timeout causing failure"
+      tested_at: "{timestamp}"
+      method: "Checked DB logs and connection pool stats"
+      result: rejected
+      evidence: "DB logs show successful queries, pool not exhausted"
+
+    - hypothesis: "Null profile object when user has no data"
+      tested_at: "{timestamp}"
+      method: "Added logging at profile access point"
+      result: confirmed
+      evidence: "Stack trace shows NPE at profile.name access"
+
+  investigation_path:
+    - "Started with error log analysis"
+    - "Ruled out infrastructure issues"
+    - "Narrowed to application code"
+    - "Identified profile access as failure point"
+```
+
+---
+
+## Anti-Patterns
+
+### What NOT to Do
+
+| Anti-Pattern | Description | Correct Approach |
+|--------------|-------------|------------------|
+| **Shotgun Debugging** | Random changes hoping bug disappears | Hypothesis-driven approach |
+| **Fix Without Verify** | Declaring "fixed" without running tests | Always verify at code + user level |
+| **Symptom Fixing** | Adding workarounds instead of root cause fix | Use 5 Whys |
+| **Blind AI Trust** | Accepting suggestions without verification | Test the failing scenario |
+| **Full Regression** | Running 4000+ tests after every fix | Use tiered validation |
+| **Circular Investigation** | Repeating tested hypotheses | Document debug journey |
+
+**Full checklist**: See `references/anti-patterns.md`
+
+---
+
+## Quick Reference
+
+### Before Fixing
+
+- [ ] Symptom identified and documented
+- [ ] Root cause analysis completed (not just symptom)
+- [ ] Impact analysis completed (upstream/downstream)
+- [ ] Risk scope assessed (isolated/medium/broad)
+- [ ] Complexity determined (low/medium/high)
+
+### After Fixing
+
+- [ ] Fix implemented
+- [ ] P1 tests pass
+- [ ] P2 tests pass (or documented why skipped)
+- [ ] Confidence level assessed
+- [ ] Manual testing escalated if required
+- [ ] Debug journey documented (if medium/high complexity)
+- [ ] Debug report written to `logs/debug-reports/`
+
+### Complexity → Actions
+
+| Complexity | Debug Journey | Validation | Escalation |
+|------------|---------------|------------|------------|
+| Low | Optional | P1 tests | Only if low confidence |
+| Medium | Required | P1 + P2 tests | If broad scope |
+| High | Required | P1 + P2 + P3 | Almost always |
+
+---
+
+## Integration
+
+### Pipeline Integration
+
+This skill is used by the Fix Validation pipeline:
+
+```fsharp
+IssueAnalyzer (loads this skill, produces debug report)
+|> FixWriter (reads debug report for context)
+|> TestWriter (reads validation plan)
+|> FixValidator (executes validation plan, assesses confidence)
+|> CodeReviewer (reviews all, makes approval decision)
+```
+
+### Agent Integration
+
+| Agent | Usage |
+|-------|-------|
+| `bulwark-issue-analyzer` (P1.2) | Loads via `skills: issue-debugging`, produces debug report |
+| `bulwark-fix-validator` (P1.3) | Loads via `skills: issue-debugging`, executes validation plan |
+
+### Artifact Flow
+
+```
+IssueAnalyzer
+    └─> logs/debug-reports/{issue-id}.yaml
+              │
+              ├─> FixWriter (reads root cause, fix approach)
+              ├─> TestWriter (reads validation plan)
+              ├─> FixValidator (executes tests, assesses confidence)
+              └─> CodeReviewer (reviews everything)
+```
+
+---
+
+## Related Skills
+
+- `subagent-prompting` (P0.1) - 4-part template for agent invocation
+- `subagent-output-templating` (P0.2) - Output format for logs
+- `pipeline-templates` (P0.3) - Fix Validation pipeline definition

package/skills/issue-debugging/references/anti-patterns.md

@@ -0,0 +1,245 @@
+# Debugging Anti-Patterns
+
+Common debugging mistakes that waste time and reduce fix reliability. Avoid these patterns.
+
+---
+
+## Anti-Pattern Checklist
+
+Before debugging, review this checklist to avoid common traps:
+
+### Pre-Fix Anti-Patterns
+
+| Anti-Pattern | Symptoms | Correct Approach |
+|--------------|----------|------------------|
+| **Shotgun Debugging** | Making random changes, "try this and see" | Form hypothesis first, then test it |
+| **Premature Fixing** | Fixing before understanding root cause | Complete 5 Whys before writing code |
+| **Tunnel Vision** | Only looking at obvious locations | Map upstream/downstream dependencies |
+| **Stack Overflow Copy-Paste** | Applying solutions without understanding | Understand WHY the solution works |
+
+### During-Fix Anti-Patterns
+
+| Anti-Pattern | Symptoms | Correct Approach |
+|--------------|----------|------------------|
+| **Symptom Fixing** | Adding workarounds, null checks without understanding why | Find and fix root cause |
+| **Fix Pile-Up** | Multiple unrelated changes in one fix | One fix per root cause |
+| **Magic Numbers** | Adding arbitrary delays, retries without analysis | Understand timing/resource issue |
+| **Commented Code** | Leaving "backup" code commented out | Use version control, delete cleanly |
+
+### Post-Fix Anti-Patterns
+
+| Anti-Pattern | Symptoms | Correct Approach |
+|--------------|----------|------------------|
+| **Fix Without Verify** | Declaring "fixed" without running tests | Always run P1 tests minimum |
+| **Blind AI Trust** | Accepting AI suggestions without testing | Verify fix addresses the failing scenario |
+| **Full Regression Compulsion** | Running 4000+ tests for every fix | Use tiered validation (P1/P2/P3) |
+| **Silent Merge** | Merging without review for "simple" fixes | All fixes need verification |
+
+### Investigation Anti-Patterns
+
+| Anti-Pattern | Symptoms | Correct Approach |
+|--------------|----------|------------------|
+| **Circular Investigation** | Testing same hypotheses repeatedly | Document debug journey |
+| **Heisenberg Debugging** | Adding debug code that changes behavior | Use non-invasive logging |
+| **Blame Game** | Assuming it's someone else's code/library | Verify assumptions with tests |
+| **Environment Blame** | "Works on my machine" | Test in consistent environment |
+
+---
+
+## Detailed Anti-Pattern Analysis
+
+### Shotgun Debugging
+
+**Description**: Making random changes hoping the bug will disappear.
+
+**Example**:
+```typescript
+// BAD: Random changes without hypothesis
+function login(user) {
+  // Try adding timeout?
+  await sleep(100);
+  // Maybe add null check?
+  if (user == null) return;
+  // What if we try-catch?
+  try {
+    return doLogin(user);
+  } catch (e) {
+    // Maybe retry?
+    return doLogin(user);
+  }
+}
+```
+
+**Why It's Bad**:
+- Wastes time on irrelevant changes
+- May mask the real bug without fixing it
+- Makes code harder to maintain
+- No understanding = bug will recur
+
+**Correct Approach**:
+1. Observe: What exactly fails?
+2. Hypothesize: "The null user case isn't handled"
+3. Experiment: Add logging to confirm user is null
+4. Fix: Handle the specific case
+5. Verify: Test the scenario that was failing
+
+---
+
+### Fix Without Verify
+
+**Description**: Declaring a fix complete without running tests or verifying the scenario.
+
+**Example Conversation**:
+```
+Developer: "I fixed the login bug, the null check was missing"
+Reviewer: "Did you test it?"
+Developer: "The code looks right, it should work"
+```
+
+**Why It's Bad**:
+- "Should work" ≠ "Does work"
+- Untested fixes often introduce regressions
+- User may still experience the bug
+- Erodes trust in "fixed" status
+
+**Correct Approach**:
+1. Before fixing: Reproduce the bug
+2. After fixing: Verify the scenario that was failing
+3. Run P1 tests (direct coverage)
+4. Run P2 tests if time allows
+5. Document verification in debug report
+
+---
+
+### Symptom Fixing
+
+**Description**: Adding workarounds instead of fixing the root cause.
+
+**Example**:
+```typescript
+// SYMPTOM: API returns 500 when profile is null
+
+// BAD: Workaround without understanding
+function getProfile(user) {
+  try {
+    return fetchProfile(user);
+  } catch (e) {
+    return {}; // Just return empty, "fixes" the 500
+  }
+}
+
+// GOOD: Fix root cause
+function getProfile(user) {
+  if (!user.hasProfile) {
+    return createDefaultProfile(user);  // Handle the actual case
+  }
+  return fetchProfile(user);
+}
+```
+
+**Why It's Bad**:
+- Bug still exists, just hidden
+- Data inconsistency downstream
+- Users may see incorrect behavior
+- Future bugs harder to diagnose
+
+**Correct Approach**:
+1. Use 5 Whys to find root cause
+2. Fix at the appropriate level
+3. Consider: Why was the symptom happening?
+4. Add proper handling, not just suppression
+
+---
+
+### Full Regression Compulsion
+
+**Description**: Running all 4000+ tests after every small fix.
+
+**Why It's Bad**:
+- Wastes CI resources
+- Slows down development
+- Creates analysis paralysis
+- Often triggered by fear, not necessity
+
+**When Full Regression IS Appropriate**:
+- Before major release
+- After architecture changes
+- When risk scope is "broad"
+- When confidence is "low"
+
+**Correct Approach**:
+Use tiered validation:
+- P1 tests: Always run (direct coverage)
+- P2 tests: Run for medium+ complexity
+- P3 tests: Run for high complexity or broad scope
+
+---
+
+### Circular Investigation
+
+**Description**: Testing the same hypotheses multiple times without documenting.
+
+**Example Timeline**:
+```
+10:00 - "Maybe it's a timing issue" - added delay, didn't help
+10:30 - "Let me check the database" - queries look fine
+11:00 - "What if it's timing?" - added delay again (forgot)
+11:30 - "Database seems slow" - checked queries again
+```
+
+**Why It's Bad**:
+- Wastes time on already-ruled-out theories
+- Context lost between attempts
+- Frustration increases, efficiency drops
+- No learning accumulates
+
+**Correct Approach**:
+Document debug journey:
+```yaml
+hypotheses_tested:
+  - hypothesis: "Timing issue"
+    result: rejected
+    evidence: "Delays don't fix it"
+  - hypothesis: "Database slow"
+    result: rejected
+    evidence: "Query logs show <10ms"
+```
+
+---
+
+## Quick Reference: What to Do Instead
+
+| Instead of... | Do This |
+|---------------|---------|
+| Random changes | Form and test hypotheses |
+| Declaring "fixed" | Verify with P1 tests |
+| Workarounds | Find and fix root cause |
+| Full regression every time | Use tiered validation |
+| Forgetting investigations | Document debug journey |
+| Copying solutions | Understand why they work |
+| Assuming library bugs | Verify with isolated test |
+| "Works on my machine" | Test in CI environment |
+
+---
+
+## Red Flags During Code Review
+
+When reviewing a fix, watch for these indicators of anti-patterns:
+
+1. **No tests changed/added** - Fix may not be verified
+2. **Generic try-catch added** - May be symptom fixing
+3. **Magic numbers or delays** - May be shotgun debugging
+4. **"Cleanup" mixed with fix** - May obscure actual fix
+5. **No explanation of root cause** - May not understand issue
+6. **"This should work"** - Not verified
+
+---
+
+## Self-Check Before Declaring Fixed
+
+- [ ] Can I explain the root cause in one sentence?
+- [ ] Did I verify the scenario that was failing?
+- [ ] Did I run P1 tests (direct coverage)?
+- [ ] Is this fix addressing root cause, not just symptoms?
+- [ ] Did I document my investigation (if medium/high complexity)?
+- [ ] Would I bet $100 this is actually fixed?