@qball-inc/the-bulwark 1.0.0

package/skills/test-audit/references/prompts/synthesis.md

@@ -0,0 +1,57 @@
+# Synthesis Prompt Template
+
+Use this template for Stage 3 (Synthesis). Unified for both Deep and Scale modes.
+
+## GOAL
+
+Synthesize classification and violation findings into prioritized audit report with test effectiveness metrics and REWRITE_REQUIRED directive.
+
+## CONSTRAINTS
+
+- Do NOT modify any files
+- Use AST verification_lines as ground truth (not heuristic estimates)
+- Calculate test effectiveness per file: `(verification_lines - affected_lines) / verification_lines`
+- When multiple violations affect the same file, compute affected_lines as the UNION of all
+  violation_scope ranges (merge overlapping/identical ranges), NOT the sum of individual
+  affected_lines values. Example: two violations both scoped to [228, 269] = 42 affected
+  lines total, not 84.
+- Include T4 violations from AST skip-detect markers (these are deterministic — no LLM re-evaluation needed)
+- Apply two-gate REWRITE_REQUIRED logic exactly as specified
+- Priority by impact: P0 (false confidence), P1 (incomplete), P2 (pattern)
+- Provide directional rewrite guidance (orchestrator figures out specifics)
+- Complete within 20 tool calls
+
+## CONTEXT
+
+**Mode:** {deep or scale}
+
+**Classification output:** `{classification_yaml_path}` (Scale mode only; Deep mode: "N/A — detection self-classified")
+
+**Detection output:** `{detection_yaml_path}`
+
+**AST skip-detect output:** `{skip_detect_json or "no markers found"}`
+Include T4 violations from skip markers in the audit report. Each skip/only/todo marker is a T4 violation:
+- `.skip` / `xdescribe` / `xit` → severity: medium, "Test disabled"
+- `.only` / `fdescribe` / `fit` → severity: high, "Focus marker — other tests not running in CI"
+- `.todo` → severity: low, "Test placeholder — not implemented"
+
+**AST verify-count output:** `{verify_count_json}`
+Use `metrics.test_logic_lines` as the authoritative `verification_lines` for effectiveness calculation.
+
+**Two-gate REWRITE_REQUIRED logic:**
+- Gate 1: Any P0 violation → REWRITE_REQUIRED regardless of percentage
+- Gate 2: P1 violations + any file <95% effectiveness → REWRITE_REQUIRED
+- Advisory: P1 >=95% or P2 only → recommendations only
+
+**Priority classification:**
+- P0: False confidence (T1, T3+) - test passes but provides no assurance
+- P1: Incomplete verification (T2, T3) - runs real code but doesn't fully verify
+- P2: Pattern issues (T4 skip/only/todo) - style, organization, disabled tests
+
+## OUTPUT
+
+Write audit report to: `logs/test-audit-{YYYYMMDD-HHMMSS}.yaml`
+
+Write diagnostics to: `logs/diagnostics/test-audit-{YYYYMMDD-HHMMSS}.yaml`
+
+Use the schema in `references/schemas/audit-output.yaml`.

package/skills/test-audit/references/rewrite-instructions.md

@@ -0,0 +1,46 @@
+# Rewrite Instructions
+
+When `REWRITE_REQUIRED == true`, follow this procedure for each file in `files_to_rewrite` (ordered by priority, then effectiveness):
+
+```
+FOR each file in files_to_rewrite:
+    1. Read the file
+    2. Load `assertion-patterns` skill (P2.1) for T1-T4 transformation patterns
+    3. Identify component type from code analysis
+    4. Load `component-patterns` skill (P2.2) for verification templates
+    5. Load `bug-magnet-data` context file matching component type:
+       - CLI: context/cli-args.md
+       - HTTP Server: context/http-body.md
+       - File Parser: context/file-contents.md
+       - Database: context/db-query.md
+       - Process Spawner: context/process-spawn.md
+    6. Load T0 + T1 edge cases from bug-magnet-data/data/ for the component:
+       - T0 (Always): strings/boundaries, numbers/boundaries, booleans/boundaries
+       - T1 (Common): strings/injection (if input handling), strings/unicode
+    7. Select applicable patterns for the violation type:
+       - T1 fix: Apply "Function call" or "Process spawn" patterns from P2.1
+       - T2 fix: Apply "Add result assertion" pattern from P2.1
+       - T3 fix: Apply "HTTP Server" or appropriate boundary pattern from P2.2
+       - T3+ fix: Apply chain patterns from P2.1
+    8. Generate verification script as intermediate artifact:
+       - Location: tmp/verification/{test-name}-verify.{ext}
+       - Purpose: Validate rewrite works before modifying test
+       - REQUIRED: Include edge cases from bug-magnet-data in verification
+    9. Rewrite test file using structured patterns
+   10. Run verification script to confirm fix
+   11. Run original test to verify it now passes
+```
+
+If `REWRITE_REQUIRED == false`: Display recommendations without auto-rewrite.
+
+## Bug-Magnet-Data Integration (REQUIRED)
+
+When generating verification scripts (step 8), you MUST include edge cases from bug-magnet-data:
+
+1. **Read the context file** for the detected component type (step 5)
+2. **Load applicable data files** based on context file's "Applicable Categories" section
+3. **Include at minimum**:
+   - Empty string / zero / null boundary values (T0)
+   - Length extremes for any string inputs (T0)
+   - Injection patterns if component handles external input (T1)
+4. **Mark destructive patterns** (`safe_for_automation: false`) as manual-only in script comments

package/skills/test-audit/references/schemas/audit-output.yaml

@@ -0,0 +1,100 @@
+# Audit Output Schema
+# Used by the synthesis sub-agent (Stage 3) to structure the final audit report.
+
+metadata:
+  skill: test-audit
+  timestamp: "{ISO-8601}"
+  mode: deep | scale
+  threshold: 5
+  sources:
+    classification: logs/test-classification-{YYYYMMDD-HHMMSS}.yaml | "N/A (deep mode)"
+    detection: logs/mock-detection-{YYYYMMDD-HHMMSS}.yaml
+    ast_verify_count: /tmp/claude/ast-verify-count.json
+    ast_skip_detect: /tmp/claude/ast-skip-detect.json
+    ast_data_flow: /tmp/claude/ast-data-flow.json
+  model: sonnet
+
+audit:
+  overview:
+    total_files: 25
+    files_analyzed: 5
+    total_verification_lines: 1250
+    total_affected_lines: 154
+    overall_effectiveness: 88%
+
+  file_analysis:
+    - file: tests/proxy.test.ts
+      verification_lines: 95
+      affected_lines: 80
+      test_effectiveness: 16%
+      priority: P0
+      violations: [T1]
+      impact: "False confidence - spawn mock hides startup failures"
+      rewrite_direction: |
+        Use real child_process.spawn(). Verify with port check.
+        Test should actually start proxy and verify it's listening.
+
+    - file: tests/workflow.integration.ts
+      verification_lines: 50
+      affected_lines: 36
+      test_effectiveness: 28%
+      priority: P0
+      violations: [T3+]
+      impact: "False confidence - broken integration chain"
+      rewrite_direction: |
+        Chain real function outputs. Remove mock data injection.
+        Replace mockOrderData with: const order = await createOrder(input);
+
+    - file: tests/api.integration.ts
+      verification_lines: 55
+      affected_lines: 37
+      test_effectiveness: 33%
+      priority: P1
+      violations: [T3]
+      impact: "Incomplete verification - mocked HTTP"
+      rewrite_direction: |
+        Use MSW or test server instead of jest.mock('node-fetch').
+        Test should verify actual HTTP request/response cycle.
+
+    - file: tests/config.test.ts
+      verification_lines: 40
+      affected_lines: 1
+      test_effectiveness: 98%
+      priority: P1
+      violations: [T2]
+      impact: "Incomplete verification - call-only assertion"
+      rewrite_direction: |
+        Add result assertion after toHaveBeenCalled().
+        Verify the actual saved value matches expected.
+
+  recommendations:
+    - "Establish test harness for proxy testing"
+    - "Create shared fixtures for integration tests"
+    - "Consider test utilities for common verification patterns"
+
+directive:
+  REWRITE_REQUIRED: true
+  gate_triggered: "Gate 1: Impact (P0 violations exist)"
+  files_to_rewrite:
+    - path: tests/proxy.test.ts
+      priority: P0
+      test_effectiveness: 16%
+    - path: tests/workflow.integration.ts
+      priority: P0
+      test_effectiveness: 28%
+    - path: tests/api.integration.ts
+      priority: P1
+      test_effectiveness: 33%
+  files_advisory:
+    - path: tests/config.test.ts
+      priority: P1
+      test_effectiveness: 98%
+      reason: "Above 95% threshold - advisory only"
+  threshold: 95%
+  rationale: "Gate 1 triggered: 2 P0 violations (false confidence). Gate 2 would also trigger: 1 P1 file below threshold."
+
+summary: |
+  Audit complete: 5 files analyzed, 4 with violations.
+  REWRITE_REQUIRED: true (Gate 1 - P0 violations)
+  Priority rewrites: proxy.test.ts (P0, 16%), workflow.integration.ts (P0, 28%), api.integration.ts (P1, 33%)
+  Advisory: config.test.ts (P1, 98% - above threshold)

package/skills/test-audit/references/schemas/diagnostic-output.yaml

@@ -0,0 +1,49 @@
+# Diagnostic Output Schema
+# Written by the synthesis sub-agent to logs/diagnostics/test-audit-{YYYYMMDD-HHMMSS}.yaml
+
+diagnostic:
+  skill: test-audit
+  timestamp: "{ISO-8601}"
+  model: sonnet
+
+mode_selection:
+  mode: deep | scale
+  file_count: 5
+  threshold: 5
+  threshold_source: default | override
+  safety_cap_triggered: false
+
+stage_0_ast:
+  verify_count: success | failed
+  skip_detect: success | failed
+  ast_analyze: success | failed
+  graceful_degradation: false
+  degradation_details: ""
+
+execution:
+  stages_completed: 3
+  stages_skipped: ["classification"]  # Deep mode skips classification
+  total_tool_calls: 12
+  classification_source: logs/test-classification-{ts}.yaml | "N/A (deep mode)"
+  detection_source: logs/mock-detection-{ts}.yaml
+
+gate_evaluation:
+  gate_1_check: true
+  gate_1_result: "triggered (2 P0 violations)"
+  gate_2_check: false
+  gate_2_result: "not evaluated (Gate 1 triggered)"
+
+decisions:
+  - file: tests/proxy.test.ts
+    decision: rewrite_required
+    reason: "P0 violation (T1) - false confidence"
+    effectiveness: 16%
+    verification_lines_source: ast  # ast or heuristic
+
+  - file: tests/config.test.ts
+    decision: advisory_only
+    reason: "P1 violation but 98% effectiveness >= 95% threshold"
+    effectiveness: 98%
+    verification_lines_source: ast
+
+errors: []