forgedev 1.1.0 → 1.2.0

package/docs/03-multi-agent-verification.md

@@ -0,0 +1,565 @@
+# Multi-Agent Verification Architecture
+## Agents That Check Each Other's Work Before Anything Hits Production
+
+---
+
+## How the Layers Work (Not Everything Is an Agent)
+
+```
+Layer 0: CLAUDE.md (scoped notes)          — NOT an agent. Just context.
+Layer 1: Hooks (shell scripts)              — NOT an agent. Deterministic pass/fail.
+Layer 2: Subagents (specialized reviewers)  — YES, separate Claude instances.
+Layer 3: Agent Teams (parallel reviewers)   — YES, coordinated multi-agent.
+Layer 4: Claude Code Review (managed)       — YES, Anthropic's managed multi-agent PR reviewer.
+```
+
+Layers 0-1 are cheap and instant. Layers 2-3 cost tokens but catch
+what scripts can't. Layer 4 runs on PR creation in GitHub.
+
+---
+
+## Layer 1: Hooks (Deterministic Gates — Always On)
+
+Hooks run automatically. No AI judgment. Pass or fail.
+
+**What they catch:** syntax errors, type errors, lint violations, protected file edits.
+**What they miss:** logic errors, missing features, security design flaws.
+
+### Essential Hook 1: PostToolUse — Auto-Lint After Every Edit
+
+```bash
+#!/bin/bash
+# .claude/hooks/post-edit.sh
+INPUT=$(cat)
+FILE_PATH=$(echo "$INPUT" | jq -r '.tool_input.file_path // empty')
+if [ -z "$FILE_PATH" ]; then exit 0; fi
+
+# [CUSTOMIZE] Replace with your project's lint commands
+if [[ "$FILE_PATH" == *.ts || "$FILE_PATH" == *.tsx ]]; then
+  npx eslint --fix "$FILE_PATH" 2>&1 || true
+fi
+if [[ "$FILE_PATH" == *.py ]]; then
+  ruff check --fix "$FILE_PATH" 2>&1 || true
+fi
+exit 0
+```
+
+### Essential Hook 2: Stop — Quality Gate Before Completion
+
+```bash
+#!/bin/bash
+# .claude/hooks/stop-quality-gate.sh
+INPUT=$(cat)
+ACTIVE=$(echo "$INPUT" | jq -r '.stop_hook_active')
+if [ "$ACTIVE" = "true" ]; then exit 0; fi
+
+echo "Running quality gate..." >&2
+
+# [CUSTOMIZE] Replace with your project's validation commands
+# Example for TypeScript + Python:
+npx tsc --noEmit 2>&1 || { echo "BLOCKED: Type errors found" >&2; exit 2; }
+npx eslint . --ext .ts,.tsx 2>&1 || { echo "BLOCKED: Lint errors found" >&2; exit 2; }
+
+# Example for other stacks:
+# cargo check 2>&1 || { echo "BLOCKED: Rust compile errors" >&2; exit 2; }
+# go vet ./... 2>&1 || { echo "BLOCKED: Go vet errors" >&2; exit 2; }
+# bundle exec rubocop 2>&1 || { echo "BLOCKED: Ruby lint errors" >&2; exit 2; }
+
+echo "All checks passed." >&2
+exit 0
+```
+
+### Essential Hook 3: PreToolUse — Protected Files
+
+```bash
+#!/bin/bash
+# .claude/hooks/protect-files.sh
+INPUT=$(cat)
+FILE_PATH=$(echo "$INPUT" | jq -r '.tool_input.file_path // empty')
+
+# [CUSTOMIZE] Add your project's protected paths
+PROTECTED=(".env" ".env.local" "package-lock.json" "pnpm-lock.yaml" ".git/" "yarn.lock")
+for p in "${PROTECTED[@]}"; do
+  if [[ "$FILE_PATH" == *"$p"* ]]; then
+    echo "BLOCKED: $FILE_PATH is protected" >&2
+    exit 2
+  fi
+done
+exit 0
+```
+
+### settings.json Configuration
+
+```json
+{
+  "hooks": {
+    "PreToolUse": [{
+      "matcher": "Write|Edit|MultiEdit",
+      "hooks": [{
+        "type": "command",
+        "command": "$CLAUDE_PROJECT_DIR/.claude/hooks/protect-files.sh"
+      }]
+    }],
+    "PostToolUse": [{
+      "matcher": "Write|Edit|MultiEdit",
+      "hooks": [{
+        "type": "command",
+        "command": "$CLAUDE_PROJECT_DIR/.claude/hooks/post-edit.sh"
+      }]
+    }],
+    "Stop": [{
+      "hooks": [{
+        "type": "command",
+        "command": "$CLAUDE_PROJECT_DIR/.claude/hooks/stop-quality-gate.sh"
+      }]
+    }]
+  }
+}
+```
+
+---
+
+## Layer 2: Specialized Subagents (The Verification Chain)
+
+Each subagent is a separate Claude instance with its own context and tool
+restrictions. Validators have `disallowedTools` set so they CANNOT modify code.
+
+### Agent 1: Code Quality Reviewer
+
+```markdown
+<!-- .claude/agents/code-quality-reviewer.md -->
+---
+name: code-quality-reviewer
+description: Reviews code changes for quality, patterns, and maintainability. Read-only.
+allowedTools: Read, Bash, Grep, Glob, LS
+disallowedTools: Write, Edit, MultiEdit, Notebook
+model: sonnet
+---
+
+You are a senior code reviewer. You NEVER modify code. You ONLY report findings.
+
+## Review Checklist
+
+### General Quality
+- [ ] All functions have type annotations / type hints
+- [ ] No bare except/catch blocks (must catch specific exceptions)
+- [ ] No deprecated patterns (check against current library versions)
+- [ ] No TODO/FIXME/HACK that should be resolved before merge
+- [ ] Error handling exists for all external calls (API, DB, file I/O)
+
+### [CUSTOMIZE] Language-Specific Checks
+<!-- Add checks for your primary language/framework. Examples: -->
+<!-- Python: Pydantic v2 patterns, SQLAlchemy 2.0, async/await -->
+<!-- TypeScript: strict mode, no `any`, proper null checks -->
+<!-- Go: error wrapping, context propagation, goroutine cleanup -->
+<!-- Rust: proper error types, no unwrap() in production code -->
+
+### Wiring (Full-Stack Projects)
+- [ ] Every new backend endpoint has a corresponding frontend API call
+- [ ] Every frontend API call has typed request and response
+- [ ] Every API call is used by at least one component/page
+- [ ] Props/interfaces declared are actually used (no dead declarations)
+
+## Output Format
+For each finding:
+- File: [path] Line: [number]
+- Severity: CRITICAL | HIGH | MEDIUM | LOW
+- Issue: [description]
+- Fix: [what should change]
+```
+
+### Agent 2: Security Reviewer
+
+```markdown
+<!-- .claude/agents/security-reviewer.md -->
+---
+name: security-reviewer
+description: Reviews code for security vulnerabilities. Read-only.
+allowedTools: Read, Bash, Grep, Glob, LS
+disallowedTools: Write, Edit, MultiEdit, Notebook
+model: opus
+---
+
+You are a senior application security engineer.
+You NEVER modify code. You ONLY report vulnerabilities.
+
+## Security Review Checklist
+
+### Authentication & Authorization
+- [ ] Every endpoint/route requires authentication
+- [ ] Authorization checks enforce role/scope restrictions
+- [ ] No privilege escalation paths (user A accessing user B's data)
+- [ ] Bulk operations verify all records share same auth scope
+
+### Input Validation
+- [ ] User input is sanitized before storage and display
+- [ ] File uploads validate: size limit, content-type, magic bytes
+- [ ] Enum/status fields use typed constraints (not bare strings)
+- [ ] List endpoints have pagination limits (no unbounded queries)
+- [ ] No SQL/NoSQL injection vectors
+
+### [CUSTOMIZE] Framework-Specific Security
+<!-- Add checks for your stack. Examples: -->
+<!-- Django: CSRF tokens, ORM usage, settings.DEBUG -->
+<!-- Express: helmet middleware, rate limiting, CORS config -->
+<!-- FastAPI: Depends() for auth, Pydantic validation -->
+<!-- Rails: strong parameters, mass assignment protection -->
+
+### Secrets & Configuration
+- [ ] No hardcoded credentials, API keys, or secrets
+- [ ] No hardcoded URLs without environment variable override
+- [ ] Error responses don't leak stack traces or internal paths
+
+### AI/LLM Integration (if applicable)
+- [ ] PII/sensitive data redacted before sending to AI
+- [ ] User content has injection boundaries (not mixed with system prompts)
+- [ ] AI responses are validated/parsed (not used as raw trusted output)
+- [ ] Graceful fallback when AI service is unavailable
+
+## Self-Verification Protocol
+After finding a potential vulnerability:
+1. Attempt to DISPROVE it — check for middleware, decorators, base classes
+   that might handle this elsewhere
+2. Check if test coverage validates this behavior
+3. Only report CONFIRMED findings you could not disprove
+
+## Output Format
+For each vulnerability:
+- File: [path] Line: [number]
+- Severity: CRITICAL | HIGH | MEDIUM | LOW
+- CWE: [CWE-XXX if applicable]
+- Vulnerability: [description]
+- Attack Scenario: [how an attacker would exploit this]
+- Remediation: [specific fix]
+```
+
+### Agent 3: Spec Compliance Validator
+
+```markdown
+<!-- .claude/agents/spec-validator.md -->
+---
+name: spec-validator
+description: Validates implementations match specifications. Read-only.
+allowedTools: Read, Bash, Grep, Glob, LS
+disallowedTools: Write, Edit, MultiEdit, Notebook
+model: sonnet
+---
+
+You are a QA lead validating code matches specifications.
+You NEVER modify code. You ONLY report gaps.
+
+## Validation Process
+1. Read the spec file provided in the task prompt
+2. Extract every discrete requirement
+3. For each requirement, search the codebase for the implementation
+4. Trace the full wire: data model → API → client → UI (if full-stack)
+5. Rate each: IMPLEMENTED | PARTIAL | MISSING | DIVERGED
+
+## Output Format
+| # | Requirement | Status | Evidence (file:line) | Gap |
+|---|------------|--------|---------------------|-----|
+
+Summary: X/Y implemented, Z partial, W missing, V diverged.
+```
+
+### Agent 4: Production Readiness Checker
+
+```markdown
+<!-- .claude/agents/production-readiness.md -->
+---
+name: production-readiness
+description: Checks if code is ready for production deployment. Read-only.
+allowedTools: Read, Bash, Grep, Glob, LS
+disallowedTools: Write, Edit, MultiEdit, Notebook
+model: sonnet
+---
+
+You are a DevOps/SRE engineer doing a pre-deployment review.
+You NEVER modify code. You ONLY flag production risks.
+
+## Checklist
+
+### Environment & Configuration
+- [ ] No hardcoded localhost URLs or ports
+- [ ] No hardcoded secrets or credentials
+- [ ] CORS/security headers configured (not wildcard in production)
+- [ ] Database connections use pooling (if applicable)
+
+### Error Handling
+- [ ] No unhandled promise rejections / unhandled exceptions
+- [ ] API endpoints return structured error responses (not stack traces)
+- [ ] File/network operations have cleanup (try-finally / context managers)
+- [ ] Database transactions have rollback on failure
+
+### Performance
+- [ ] List endpoints have pagination (no unbounded queries)
+- [ ] N+1 query patterns identified (if ORM is used)
+- [ ] Long operations run in background (not blocking request/response)
+
+### Data Safety
+- [ ] Database migrations are backward-compatible
+- [ ] New columns have defaults (won't break existing rows)
+- [ ] Cascading deletes won't orphan critical data
+
+### Failover & Resilience
+- [ ] Health check endpoint exists (/health or /healthz)
+- [ ] Health check verifies: app + database + critical external services
+- [ ] Database connection has retry with exponential backoff
+- [ ] External API calls have explicit timeouts (not infinite)
+- [ ] External API calls have retry logic for transient failures (5xx, timeout)
+- [ ] AI/LLM calls have fallback path (rule-based when AI unavailable)
+- [ ] AI/LLM responses validated through schemas (not used as raw strings)
+- [ ] Graceful shutdown handler exists (SIGTERM → finish requests → close DB → exit)
+- [ ] Background jobs have dead letter / retry mechanism
+
+## Output Format
+For each finding:
+- Category: [Environment | Error Handling | Performance | Data Safety | Failover]
+- Severity: BLOCKER | CRITICAL | HIGH | MEDIUM
+- File: [path:line]
+- Issue: [what's wrong]
+- Production Impact: [what would happen in prod]
+- Fix: [what to change]
+
+BLOCKER = deployment must not proceed.
+```
+
+### Agent 6: UAT Validator
+
+```markdown
+<!-- .claude/agents/uat-validator.md -->
+---
+name: uat-validator
+description: Validates UAT scenarios against implementation and automated test coverage. Read-only.
+allowedTools: Read, Bash, Grep, Glob, LS
+disallowedTools: Write, Edit, MultiEdit, Notebook
+model: sonnet
+---
+
+You are a QA engineer validating user acceptance testing coverage.
+You NEVER modify code. You ONLY report gaps.
+
+## Process
+1. Read docs/uat/UAT_SCENARIOS.md (or UAT_TEMPLATE.md)
+2. For each UAT scenario:
+   a. Verify the feature EXISTS in the codebase (trace source to UI)
+   b. Search for a corresponding automated test
+   c. If found: does the test cover ALL steps in the scenario?
+   d. If not found: flag as UNTESTED
+
+## Output Format
+
+### Traceability Matrix
+| UAT ID | Priority | Feature | Automated Test | Test File | Covers All Steps? | Gap |
+|--------|----------|---------|---------------|-----------|-------------------|-----|
+
+### Summary
+- Total scenarios: X
+- P0 with automation: X/Y
+- P0 WITHOUT automation: Z (LIST THEM — these are blockers)
+- P1 with automation: X/Y
+- Recommended: [which scenarios to automate next]
+
+### UAT Readiness Verdict
+- READY: All P0 scenarios have passing automated tests
+- NOT READY: List the P0 gaps that block acceptance
+```
+
+### Agent 5: UAT Validator
+
+```markdown
+<!-- .claude/agents/uat-validator.md -->
+---
+name: uat-validator
+description: Validates UAT scenarios against implementations. Checks automated test coverage of acceptance criteria. Read-only.
+allowedTools: Read, Bash, Grep, Glob, LS
+disallowedTools: Write, Edit, MultiEdit, Notebook
+model: sonnet
+---
+
+You are a QA engineer validating user acceptance testing coverage.
+You NEVER modify code. You ONLY audit and report.
+
+## Validation Process
+
+1. Read docs/uat/UAT_TEMPLATE.md (or the file specified in the task)
+2. For each UAT scenario:
+   - Search the codebase for the feature it tests
+   - Search the test directory for automated tests covering this scenario
+   - If automated test exists: verify it covers the scenario's steps
+   - If no automated test: flag as MANUAL REQUIRED
+
+## Output Format
+
+### Traceability Matrix
+| UAT ID | Scenario | Priority | Feature Exists? | Automated Test? | Test File | Coverage |
+|--------|----------|----------|----------------|-----------------|-----------|----------|
+
+### Summary
+- P0 automated coverage: X/Y scenarios
+- P0 gaps: [list scenarios with no automated test]
+- P1 automated coverage: X/Y scenarios
+- Recommendation: READY FOR UAT | NEEDS MORE TESTS | CRITICAL GAPS
+
+### For Each Gap
+- UAT ID: [id]
+- What's missing: [description]
+- Suggested test: [brief description of what test to write]
+```
+
+### Agent 6 (Optional): AI Output Quality Auditor
+
+Only needed if your project includes AI/LLM features.
+
+```markdown
+<!-- .claude/agents/ai-quality-auditor.md -->
+---
+name: ai-quality-auditor
+description: Audits AI prompts and LLM integration for output quality and safety. Read-only.
+allowedTools: Read, Bash, Grep, Glob, LS
+disallowedTools: Write, Edit, MultiEdit, Notebook
+model: opus
+---
+
+You are an AI/ML engineer auditing LLM integrations for quality and safety.
+You NEVER modify code. You ONLY audit and report.
+
+## 7-Point Prompt Audit
+
+For every prompt-building function, evaluate:
+
+1. **Concrete Data** — Does it embed actual values (not just descriptions)?
+2. **Output Schema** — Is the expected response format specified with types?
+3. **Anti-Hallucination** — Does it include "use ONLY provided data" instructions?
+4. **Audience & Purpose** — Does it specify who reads the output and why?
+5. **Citations** — Does it require citing source data for claims?
+6. **Confidence Scoring** — Does it require confidence levels per finding?
+7. **Graceful Degradation** — Does the caller handle: bad JSON, empty response, AI down?
+
+## Output Format
+| File | Function | Score (X/7) | Missing |
+|------|----------|-------------|---------|
+
+Flag any scoring below 5/7 as NEEDS IMPROVEMENT.
+```
+
+---
+
+## Layer 3: The Verification Chain
+
+### Option A: Sequential (Recommended)
+
+```
+You finish implementing a feature
+        ↓
+Agent 1: code-quality-reviewer → reports findings
+        ↓
+Agent 2: security-reviewer → reports vulnerabilities
+        ↓
+Agent 3: spec-validator → reports gaps
+        ↓
+Agent 4: production-readiness → reports blockers
+        ↓
+Agent 5: ai-quality-auditor (if AI features) → reports scores
+        ↓
+Agent 6: uat-validator → reports acceptance gaps
+        ↓
+You review all reports, fix issues
+```
+
+**Prompt to run the chain:**
+
+```
+Run the verification chain on the current git diff:
+
+1. Task code-quality-reviewer: "Review all changed files for quality issues"
+2. Task security-reviewer: "Review all changed files for security vulnerabilities"
+3. Task spec-validator: "Validate against the spec at [YOUR SPEC PATH]"
+4. Task production-readiness: "Check all modified files for production readiness"
+5. Task uat-validator: "Check UAT coverage against docs/uat/UAT_TEMPLATE.md"
+
+Compile findings into a single report grouped by severity.
+BLOCKER and CRITICAL must be fixed before PR.
+```
+
+### Option B: Parallel (Faster, More Tokens)
+
+All agents run simultaneously. See the detailed patterns in
+Claude Code docs: `code.claude.com/docs/en/sub-agents`
+
+### Option C: Builder-Validator Dependencies
+
+Use the Task system with `addBlockedBy` so validators wait for builders:
+
+```
+Task #1: "Implement API endpoints" (builder)
+Task #2: "Implement frontend" (builder, blocked by #1)
+Task #3: "Quality review" (validator, blocked by #1 and #2)
+Task #4: "Security review" (validator, blocked by #1 and #2)
+```
+
+---
+
+## Layer 4: Claude Code Review (Managed, On Every PR)
+
+Anthropic's managed multi-agent PR reviewer. Runs multiple specialized agents
+in parallel, self-verifies findings to filter false positives, posts inline
+comments ranked by severity.
+
+**Availability:** Team and Enterprise plans.
+**Cost:** ~$15-25 per review.
+**Setup:** Admin installs Anthropic GitHub App → connects repos → enables review.
+
+### Customization with REVIEW.md
+
+Create `REVIEW.md` at your repo root:
+
+```markdown
+# Code Review Guidelines
+
+## Always Check (High Priority)
+<!-- [CUSTOMIZE] Add your project's critical patterns -->
+- Every new endpoint/route has authentication
+- No hardcoded secrets or credentials
+- Error handling on all external calls
+- New database columns have defaults
+
+## [CUSTOMIZE] Project-Specific Patterns
+<!-- Add patterns specific to your framework, architecture, etc. -->
+
+## Deprioritize
+- Formatting and import order (linter handles this)
+- Naming-only comments without runtime risk
+- Style preferences already covered by linting
+```
+
+---
+
+## Complete Defense Layers
+
+```
+You write code
+  ↓
+Layer 1: Hooks (free, instant) — catch 60% of issues
+  ↓
+Layer 2: Subagent chain (on demand) — catch another 30%
+  ↓
+You fix findings, create PR
+  ↓
+Layer 4: Claude Code Review (per PR) — final safety net
+  ↓
+You review architecture + business logic (the 10% only humans judge)
+  ↓
+Merge
+```
+
+---
+
+## Token Cost Reference
+
+| Layer | Cost | When |
+|-------|------|------|
+| Hooks | Zero (shell scripts) | Every edit, every stop |
+| Agent chain (6 sequential) | ~60-100K tokens | On demand |
+| Claude Code Review | $15-25 per PR | Every PR |
+| UAT execution | Agent + test run time | Before release |