@qball-inc/the-bulwark 1.0.0

package/skills/mock-detection/SKILL.md

@@ -0,0 +1,511 @@
+---
+name: mock-detection
+description: Deep mock appropriateness analysis for Test Audit pipeline
+user-invocable: false
+---
+
+# Mock Detection
+
+Prompt template for deep mock appropriateness analysis using call graph tracing. Designed for a Sonnet sub-agent to detect T1-T4 violations and track violation scope.
+
+---
+
+## When to Use This Skill
+
+**This is an internal skill loaded by the orchestrator during Test Audit pipeline.**
+
+| Context | Action |
+|---------|--------|
+| `/test-audit` invoked | Orchestrator loads this skill for Stage 2 |
+| Test Audit pipeline triggered by hook | Orchestrator loads this skill for Stage 2 |
+| Need deep mock analysis | Load directly as prompt template for Sonnet |
+| Files flagged by test-classification | Analyze only `needs_deep_analysis: true` files |
+
+**DO NOT use for:**
+- Direct user invocation (not user-invocable)
+- Surface-level classification (use `test-classification` skill)
+- Full audit synthesis (use `test-audit` skill)
+
+---
+
+## Role in Test Audit Pipeline
+
+This skill provides the **second stage** prompt template:
+
+```
+test-audit (P0.8) orchestrates:
+  Stage 1: test-classification (Haiku) → classification YAML
+  Stage 2: mock-detection (Sonnet) → violations YAML    ← THIS SKILL
+  Stage 3: synthesis (Sonnet) → audit report
+```
+
+The orchestrator loads this skill and constructs a 4-part prompt for a general-purpose Sonnet sub-agent.
+
+---
+
+## 4-Part Prompt Template
+
+### GOAL
+
+Analyze flagged test files for T1-T4 violations using mock appropriateness rubric and call graph analysis. Track the full scope of each violation for test effectiveness calculation.
+
+### CONSTRAINTS
+
+- Do NOT modify any files
+- Only analyze files with `needs_deep_analysis: true` from classification
+- Use call graph analysis to detect broken integration chains
+- Track violation scope (all affected lines, not just violation line)
+- Provide full context for each violation (line, snippet, reason, fix)
+- Complete within 50 tool calls
+
+### CONTEXT
+
+**Classification output:** `{classification_yaml_path}`
+
+**Files to analyze:** List of files with `needs_deep_analysis: true`
+
+**Mock appropriateness rubric:** See "Mock Appropriateness Rubric" section below
+
+**T1-T4 detection patterns:** See "T1-T4 Detection Patterns" section below
+
+**Violation scope tracking:** See "Violation Scope Tracking" section below
+
+**Extended stub/fake patterns:** See `references/stub-patterns.md` — Meszaros taxonomy, class hierarchy detection, factory function classification
+
+**False positive prevention:** See `references/false-positive-prevention.md` — Two-tier allowlist (Universal Safe / Context-Dependent) and decision tree. Consult BEFORE flagging borderline patterns.
+
+### OUTPUT
+
+Write violations to: `logs/mock-detection-{YYYYMMDD-HHMMSS}.yaml`
+
+Write diagnostics to: `logs/diagnostics/mock-detection-{YYYYMMDD-HHMMSS}.yaml`
+
+Use the schema specified in "Output Schema" section below.
+
+---
+
+## Mock Appropriateness Rubric
+
+Determine whether mocks are appropriate based on test type:
+
+| Test Type | Expected Mocks (OK) | Inappropriate Mocks (VIOLATION) |
+|-----------|---------------------|--------------------------------|
+| **Unit** | External deps (DB, HTTP, fs) to isolate unit | Mocking function/module under test (T1) |
+| **Integration** | Unrelated systems only | Mocking integration boundaries (T3), broken chain (T3+) |
+| **E2E** | Almost never | Any mock breaking end-to-end flow |
+
+### Mixed-Type Files (MANDATORY)
+
+Test files commonly contain multiple test types in different describe blocks (e.g., unit tests at top, integration tests at bottom). You MUST evaluate mock appropriateness **per describe block/section**, not per file. A `jest.fn()` that is safe in a unit test section is a T3 violation in an integration test section of the same file.
+
+Classification signals (language-agnostic — apply to TypeScript, Python, Java, Go, Ruby, etc.):
+- Block/suite name containing keywords: `integration`, `e2e`, `end-to-end`, `acceptance`, `system`
+- Preceding comments or section headers: `// INTEGRATION TESTS`, `# E2E`, `/* system tests */`
+- Setup patterns within the block (real DB connections = integration, browser launch = e2e)
+
+If AST integration-mock metadata is available (from `just integration-mocks`), use it as ground truth for section boundaries and mock locations. Validate AST leads and add any the AST missed.
+
+**BINDING: AST classification is final.** When the AST script classifies a section as integration or e2e, that classification is NOT subject to LLM override. You MUST evaluate mocks in that section against integration/e2e rules — even if you believe the section is "actually" a unit test. Dismissing an AST T3 lead by re-classifying the section as a different test type is a rule violation. If you believe the section is mislabeled, note it as advisory — but still flag T3 violations against the classified type.
+
+See `references/false-positive-prevention.md` § "Worked Example: Mixed-Type File" for a concrete demonstration.
+
+### Key Principle
+
+A mock is inappropriate when it **defeats the purpose of the test**:
+- Unit test claims to test function X but mocks function X → T1
+- Integration test claims to verify HTTP calls but mocks HTTP → T3
+- Test verifies function was called but not what it produced → T2
+
+---
+
+## T1-T4 Detection Patterns
+
+### T1: Mocking System Under Test
+
+**Severity:** Critical
+**Priority:** P0 (False Confidence)
+
+**Detection patterns:**
+- `jest.spyOn(ModuleUnderTest, 'functionBeingTested')`
+- `jest.mock('./file-being-tested')`
+- `vi.mock()` on the module imported in the test's subject
+- Mock intercepts the exact function the test claims to verify
+
+**Call graph check:** Trace from test assertion back to setup. If mock sits between "action" and "assertion" for the claimed behavior, it's T1.
+
+### T2: Verifying Calls Not Results
+
+**Severity:** High
+**Priority:** P1 (Incomplete Verification)
+
+**Detection patterns:**
+- `expect(fn).toHaveBeenCalled()` without result assertion
+- `expect(fn).toHaveBeenCalledWith(...)` without verifying the effect
+- `verify(mock).someMethod()` without outcome check
+
+**Call graph check:** After the `toHaveBeenCalled` assertion, is there a result/state assertion for the same operation?
+
+### T3: Mock at Integration Boundary
+
+**Severity:** Critical
+**Priority:** P1 (Incomplete Verification)
+
+**Detection patterns:**
+- In `.integration.*` file: `jest.mock('node-fetch')`, `jest.mock('fs')`, `jest.mock('http')`
+- In integration test: Mocking the very boundary being integrated
+
+**Call graph check:** Does the test claim to verify "integration with X" while mocking X?
+
+### T3+: Broken Integration Chain
+
+**Severity:** Critical
+**Priority:** P0 (False Confidence)
+
+**Detection patterns:**
+- `mockData` used where real function output should flow
+- Data manually constructed instead of flowing from upstream function
+- Integration test with hardcoded intermediate values
+- Property access on manually-constructed objects used in downstream calls (e.g., `mockOrder.id` passed into new objects)
+
+**Class hierarchy signals** (see `references/stub-patterns.md` for full taxonomy):
+- Classes named Fake*/Stub*/Mock*/InMemory*/Test* that implement interfaces or extend base classes
+- Manual stub objects without naming conventions (all methods are no-ops or return hardcoded values)
+- Factory functions prefixed with `buildMock*` or `createFake*`
+
+**Call graph check:** Trace data flow. If Component A should output to Component B, but test injects `mockAOutput` into B, the chain is broken.
+
+### T4: No Test Execution Verification
+
+**Severity:** Medium
+**Priority:** P2 (Pattern Issues)
+
+**Detection patterns:**
+- Test file exists but `just test` not run
+- Tests pass in isolation but skip in suite
+- Manual notation: "Requires V3 manual verification"
+
+**Note:** T4 is primarily a process check. Flag for manual review.
+
+---
+
+## Violation Scope Tracking
+
+Track the **full scope** of each violation - not just the violation line, but all lines affected by it. This enables accurate test effectiveness calculation.
+
+### Scope Calculation Rules
+
+| Violation Type | Scope Definition |
+|----------------|------------------|
+| **T1 (Mock SUT)** | All lines that use the mock: assertions depending on mock, calls using mock return value |
+| **T2 (Call-only)** | The assertion line itself (single line) |
+| **T3 (Mock boundary)** | All lines using the mocked boundary (similar to T1) |
+| **T3+ (Broken chain)** | All lines using the incorrect/mocked data downstream |
+
+### Example
+
+```javascript
+// Line 15: Mock setup (violation line)
+const mockSpawn = jest.spyOn(child_process, 'spawn')
+  .mockReturnValue(mockProcess);
+
+// Lines 20-95: All use mockSpawn results
+const proxy = startProxy();      // Line 20 - uses mock
+await proxy.waitForReady();      // Line 21 - uses mock
+expect(proxy.port).toBe(8080);   // Line 25 - assertion on mock
+// ... more lines using mock ...
+expect(proxy.isRunning()).toBe(true);  // Line 95 - still mock
+
+// violation_scope: [15, 95]
+// affected_lines: 80
+```
+
+---
+
+## Call Graph Analysis Approach
+
+For each flagged file, perform systematic analysis:
+
+### Step 1: Identify Test Claims
+
+What does each test claim to verify? Look at:
+- Test name/description (`it('starts proxy correctly', ...)`)
+- Assertion statements (what is being expected?)
+
+### Step 2: Trace Data Flow
+
+How does data flow from setup → action → assertion?
+- Where is input created?
+- What transforms the input?
+- What does the assertion check?
+
+### Step 3: Locate Mock Interception
+
+Where do mocks intercept this flow?
+- Is mock between action and the claimed verification?
+- Does mock replace real behavior the test should exercise?
+
+### Step 4: Evaluate Appropriateness
+
+Does the mock defeat the test's purpose?
+- If test claims "proxy starts" but mocks spawn → T1
+- If test claims "API integration" but mocks fetch → T3
+
+---
+
+## Output Schema
+
+```yaml
+metadata:
+  skill: mock-detection
+  timestamp: "{ISO-8601}"
+  classification_source: logs/test-classification-{YYYYMMDD-HHMMSS}.yaml
+  model: sonnet
+  files_analyzed: 5
+
+violations:
+  - file: tests/proxy.test.ts
+    line: 15
+    violation_scope: [15, 95]
+    affected_lines: 80
+    rule: T1
+    severity: critical
+    priority: P0
+    pattern: "jest.spyOn(child_process, 'spawn')"
+    code_snippet: |
+      const mockSpawn = jest.spyOn(child_process, 'spawn')
+        .mockReturnValue(mockProcess);
+    reason: |
+      Test claims to verify "proxy starts correctly" but mocks spawn().
+      This provides false confidence - mock always succeeds.
+      Lines 15-95 all use this mock, making them ineffective.
+    suggested_fix: |
+      Replace mock with real spawn. Use port check to verify proxy started.
+
+  - file: tests/api.integration.ts
+    line: 8
+    violation_scope: [8, 45]
+    affected_lines: 37
+    rule: T3
+    severity: critical
+    priority: P1
+    pattern: "jest.mock('node-fetch')"
+    code_snippet: |
+      jest.mock('node-fetch');
+      // ... later in test
+      const response = await fetchUserData(userId);
+    reason: |
+      Integration test should verify real HTTP communication.
+      Mocking fetch defeats the purpose of integration testing.
+    suggested_fix: |
+      Remove jest.mock('node-fetch'). Use test server or MSW.
+
+  - file: tests/workflow.integration.ts
+    line: 42
+    violation_scope: [42, 78]
+    affected_lines: 36
+    rule: T3+
+    severity: critical
+    priority: P0
+    pattern: "Broken integration chain"
+    code_snippet: |
+      const result = await processOrder(mockOrderData);
+      // mockOrderData should come from createOrder() output
+    reason: |
+      Test uses mockOrderData instead of real createOrder() output.
+      This breaks the integration chain - no real integration tested.
+    suggested_fix: |
+      Replace mockOrderData with: const order = await createOrder(input);
+
+  - file: tests/config.test.ts
+    line: 42
+    violation_scope: [42, 42]
+    affected_lines: 1
+    rule: T2
+    severity: high
+    priority: P1
+    pattern: "expect(db.save).toHaveBeenCalled()"
+    code_snippet: |
+      await saveConfig(newConfig);
+      expect(db.save).toHaveBeenCalled();
+    reason: |
+      Verifies db.save was called but not what was saved.
+      Call verification without result verification is incomplete.
+    suggested_fix: |
+      Add result verification: expect(saved.value).toBe(newConfig.value);
+
+totals:
+  critical: 3
+  high: 1
+  medium: 0
+  low: 0
+  total_affected_lines: 154
+
+file_summaries:
+  # For each file, compute affected_lines as the UNION of all violation_scope ranges
+  # (merge overlapping/identical ranges). Do NOT sum individual affected_lines values.
+  # Example: two violations both scoped to [228, 269] = 42 affected lines, not 84.
+  - file: tests/proxy.test.ts
+    verification_lines: 95
+    affected_lines: 80
+    test_effectiveness: 16%
+  - file: tests/api.integration.ts
+    verification_lines: 55
+    affected_lines: 37
+    test_effectiveness: 33%
+  - file: tests/workflow.integration.ts
+    verification_lines: 50
+    affected_lines: 36
+    test_effectiveness: 28%
+  - file: tests/config.test.ts
+    verification_lines: 40
+    affected_lines: 1
+    test_effectiveness: 98%
+
+summary: |
+  Analyzed 5 flagged files. Found 4 violations affecting 154 lines.
+  3 files below 95% test effectiveness threshold.
+  P0 violations (false confidence): proxy.test.ts, workflow.integration.ts
+  P1 violations (incomplete): api.integration.ts, config.test.ts
+```
+
+---
+
+## Diagnostic Output
+
+Write diagnostic output to `logs/diagnostics/mock-detection-{YYYYMMDD-HHMMSS}.yaml`:
+
+```yaml
+diagnostic:
+  skill: mock-detection
+  timestamp: "{ISO-8601}"
+  model: sonnet
+
+execution:
+  tool_calls: 35
+  files_analyzed: 5
+  analysis_depth: "call graph tracing"
+
+decisions:
+  - file: tests/proxy.test.ts
+    decision: T1_violation
+    call_graph_analysis: |
+      Test claims: "proxy starts correctly"
+      Action: startProxy() calls child_process.spawn()
+      Mock: jest.spyOn intercepts spawn()
+      Result: Assertion verifies mock behavior, not real spawn
+    confidence: high
+
+  - file: tests/config.test.ts
+    decision: T2_violation
+    call_graph_analysis: |
+      Assertion: toHaveBeenCalled() on db.save
+      Missing: No assertion on saved data value
+      Scope: Single assertion line (minimal impact)
+    confidence: high
+
+errors: []
+```
+
+---
+
+## Priority Classification
+
+### P0: False Confidence
+
+Tests that pass but provide no real assurance:
+- **T1**: Mock hides real failures - test always passes regardless of SUT behavior
+- **T3+**: Broken chain means integration is never actually tested
+
+### P1: Incomplete Verification
+
+Tests that run real code but don't fully verify:
+- **T2**: Call happened but effect not verified
+- **T3**: Integration boundary mocked (partial integration)
+
+### P2: Pattern Issues
+
+Style and organization issues:
+- Minor mock patterns
+- Test structure recommendations
+
+---
+
+## Integration Notes
+
+### Orchestrator Usage
+
+The orchestrator (P0.8) constructs the full prompt by:
+
+1. Loading this skill content
+2. Including classification YAML path in CONTEXT
+3. Spawning: `Task(subagent_type="general-purpose", model="sonnet", prompt=...)`
+4. Reading output from `logs/mock-detection-{YYYYMMDD-HHMMSS}.yaml`
+
+### Upstream Input
+
+From P0.6 (test-classification):
+- `needs_deep_analysis: true` file list
+- `verification_lines` count per file
+- `mock_indicators` as analysis starting points
+
+### Downstream Output
+
+To P0.8 (test-audit synthesis):
+- Violation details with scope tracking
+- `affected_lines` per file
+- Pre-calculated `test_effectiveness` per file
+
+---
+
+## Batching for Scale
+
+When processing many flagged files (>10), the orchestrator must batch detection to avoid context limits.
+
+### Batching Instructions
+
+```
+IF flagged_file_count > 10:
+    Split flagged files into batches of 10-15
+    FOR each batch:
+        Spawn Sonnet sub-agent with batch file list
+        Include verification_lines from classification for each file
+        Collect violations YAML for batch
+    Merge all batch results into single detection output
+ELSE:
+    Process all flagged files in single sub-agent call
+```
+
+### Batch Merge Strategy
+
+When merging batch results:
+1. Combine all `violations` arrays
+2. Recalculate `totals` across all batches
+3. Combine all `file_summaries`
+4. Preserve individual violation details exactly
+
+### Parallel Execution
+
+For optimal performance, spawn batch sub-agents in parallel:
+
+```
+Task(subagent_type="general-purpose", model="sonnet", prompt=batch1_prompt, run_in_background=true)
+Task(subagent_type="general-purpose", model="sonnet", prompt=batch2_prompt, run_in_background=true)
+...
+```
+
+Read all outputs after completion, then merge.
+
+---
+
+## References
+
+| Document | Purpose |
+|----------|---------|
+| `references/stub-patterns.md` | Meszaros test double taxonomy, class hierarchy detection, factory function classification |
+| `references/false-positive-prevention.md` | Two-tier allowlist (Universal Safe / Context-Dependent), decision tree for violation evaluation |
+
+## Related Skills
+
+- `test-classification` (P0.6) - Surface classification (upstream)
+- `test-audit` (P0.8) - Orchestration and synthesis (downstream)
+- `pipeline-templates` (P0.3) - Test Audit pipeline definition