claude-mpm 5.0.9__py3-none-any.whl → 5.6.23__py3-none-any.whl

claude_mpm/skills/bundled/debugging/verification-before-completion/references/verification-patterns.md

@@ -0,0 +1,499 @@
+# Verification Patterns
+
+Complete patterns for verifying different types of claims before making them.
+
+## Test Verification
+
+**Correct Pattern:**
+```
+1. Run full test suite: pytest tests/
+2. Read complete output
+3. Count: 34 tests, 34 passed, 0 failed
+4. Claim: "All 34 tests pass"
+```
+
+**Incorrect Patterns:**
+- ❌ "Should pass now" (no evidence)
+- ❌ "Looks correct" (subjective)
+- ❌ "Tests were passing earlier" (stale)
+- ❌ "I fixed the issue" (assumes, doesn't verify)
+
+**Evidence Required:**
+- Full test command executed
+- Complete output visible
+- Exact pass/fail counts
+- Exit code confirmed (0 = success)
+
+**Edge Cases:**
+- Flaky tests: Run multiple times, document flakiness
+- Timeout tests: Verify timeout values appropriate
+- Skipped tests: Count and document why skipped
+- Warnings in output: Document and address
+
+**Troubleshooting:**
+- **Tests hang**: Check for infinite loops, deadlocks
+- **Random failures**: Run 10x, identify flaky tests
+- **Environment issues**: Verify dependencies, config
+- **Coverage gaps**: Check coverage report for holes
+
+## Regression Test Verification (TDD Red-Green Cycle)
+
+**Correct Pattern:**
+```
+1. Write regression test
+2. Run test → MUST FAIL (RED)
+3. Implement fix
+4. Run test → MUST PASS (GREEN)
+5. Revert fix temporarily
+6. Run test → MUST FAIL AGAIN (confirms test works)
+7. Restore fix
+8. Run test → MUST PASS (final confirmation)
+```
+
+**Why Red-Green-Red-Green?**
+- First RED: Confirms test catches the bug
+- GREEN: Confirms fix works
+- Second RED: Confirms test isn't passing by accident
+- Final GREEN: Confirms fix is stable
+
+**Incorrect Patterns:**
+- ❌ "I've written a regression test" (didn't verify it fails)
+- ❌ Test passes on first run (never confirmed it catches bug)
+- ❌ Single pass without reverting (could be false positive)
+
+**Edge Cases:**
+- Test passes immediately: Test doesn't catch bug, rewrite
+- Test fails differently: Fix changed behavior, investigate
+- Can't revert cleanly: Use version control, stash changes
+
+## Build Verification
+
+**Correct Pattern:**
+```
+1. Run full build: make build
+2. Read complete output
+3. Check exit code: echo $? → 0
+4. Verify artifacts created: ls dist/
+5. Claim: "Build succeeds, artifacts in dist/"
+```
+
+**Incorrect Patterns:**
+- ❌ "Linter passed, so build should work" (linter ≠ compiler)
+- ❌ "No errors in logs" (didn't actually build)
+- ❌ "Build was working earlier" (stale verification)
+
+**Common Gotchas:**
+- Linter passing ≠ compilation passing
+- TypeScript errors ≠ build errors
+- Local build ≠ CI build
+- Dev build ≠ production build
+
+**Build Types to Verify:**
+- Development build: Fast iteration
+- Production build: Minification, optimization
+- Test build: Coverage instrumentation
+- Distribution build: Platform-specific artifacts
+
+**Troubleshooting:**
+- **Build succeeds but artifacts missing**: Check output directory
+- **Incremental build issues**: Clean build, verify again
+- **Platform-specific failures**: Test on target platform
+- **Resource exhaustion**: Check memory, disk space
+
+## Linter Verification
+
+**Correct Pattern:**
+```
+1. Run linter: ruff check .
+2. Read full output
+3. Count violations: 0 errors, 0 warnings
+4. Check exit code: 0
+5. Claim: "Linter clean (0 violations)"
+```
+
+**Incorrect Patterns:**
+- ❌ "Fixed the obvious issues" (partial check)
+- ❌ "Linter passed on one file" (not comprehensive)
+- ❌ "Should be clean now" (no verification)
+
+**Multiple Linters Pattern:**
+```
+1. Run each linter separately
+2. Verify each independently
+3. Document results for each
+4. Only claim clean if ALL pass
+```
+
+**Edge Cases:**
+- Warnings vs errors: Document acceptable warnings
+- Auto-fix available: Run auto-fix, verify results
+- Custom rules: Verify custom rules active
+- Ignored files: Document why files ignored
+
+## Bug Fix Verification
+
+**Correct Pattern:**
+```
+1. Document original symptom
+2. Create reproduction steps
+3. Run reproduction → FAILS (confirms bug)
+4. Implement fix
+5. Run reproduction → PASSES
+6. Run full test suite → PASSES (no regressions)
+7. Claim: "Bug fixed, verified with reproduction and tests"
+```
+
+**Incorrect Patterns:**
+- ❌ "Code changed, bug should be fixed" (assumes)
+- ❌ "Logic looks correct" (theory, not evidence)
+- ❌ "Can't reproduce anymore" (didn't verify with test)
+
+**Bug Types:**
+- **Logic bugs**: Unit test verifies correct behavior
+- **Race conditions**: Stress test, run 100x
+- **Edge cases**: Test boundary conditions
+- **Integration bugs**: End-to-end test verifies
+- **Performance bugs**: Benchmark before/after
+
+**Troubleshooting:**
+- **Can't reproduce**: Document steps, environment
+- **Intermittent bug**: Increase test iterations
+- **Different in production**: Test in production-like environment
+- **Fix causes regression**: Test full suite, not just bug test
+
+## Requirements Verification
+
+**Correct Pattern:**
+```
+1. Re-read original requirements
+2. Create line-by-line checklist
+3. Verify each requirement individually
+4. Document evidence for each
+5. Report: "5/5 requirements met" OR "4/5 met, missing X"
+```
+
+**Incorrect Patterns:**
+- ❌ "Tests pass, so requirements met" (tests ≠ requirements)
+- ❌ "I implemented what was asked" (subjective)
+- ❌ "Phase complete" (vague, no checklist)
+
+**Requirement Types:**
+- **Functional**: Feature works as specified
+- **Non-functional**: Performance, scalability, security
+- **User experience**: UI/UX matches design
+- **Integration**: Works with other systems
+- **Documentation**: Docs complete and accurate
+
+**Checklist Format:**
+```
+[ ] Requirement 1: [Evidence: test_feature_x passes]
+[ ] Requirement 2: [Evidence: benchmark shows 50ms response]
+[ ] Requirement 3: [Evidence: manual testing confirms behavior]
+[X] Requirement 4: MISSING - not yet implemented
+[ ] Requirement 5: [Evidence: integration test passes]
+```
+
+## Agent Delegation Verification
+
+**Correct Pattern:**
+```
+1. Agent reports: "Task complete"
+2. Check version control diff: git diff
+3. Read all changes made
+4. Verify changes match task requirements
+5. Run verification commands (tests, build, etc.)
+6. Report actual state: "Agent made changes to X, Y verified"
+```
+
+**Incorrect Patterns:**
+- ❌ Trusting agent success report
+- ❌ "Agent said success, moving on"
+- ❌ Not checking actual changes made
+
+**Agent Verification Checklist:**
+```
+1. Review git diff: What changed?
+2. Run tests: Do tests still pass?
+3. Check task requirements: All met?
+4. Look for unexpected changes: Any surprises?
+5. Verify no secrets added: .env, keys, tokens?
+6. Check for commented code: Any TODOs added?
+7. Verify imports/dependencies: Any new ones?
+```
+
+**Common Agent Failures:**
+- Agent adds TODO instead of implementing
+- Agent introduces syntax errors
+- Agent misunderstands requirements
+- Agent makes unrelated changes
+- Agent adds secrets to version control
+- Agent breaks existing functionality
+
+**Troubleshooting:**
+- **Agent claims success but tests fail**: Review diff, find issue
+- **Agent made unexpected changes**: Revert, clarify task
+- **Agent added placeholder code**: Complete implementation
+- **Agent broke unrelated code**: Partial revert, fix issue
+
+## Deployment Verification
+
+**Correct Pattern:**
+```
+1. Deploy to environment
+2. Check deployment logs: SUCCESS
+3. Verify endpoint accessible: curl https://...
+4. Check application logs: No errors
+5. Run smoke tests on deployed version
+6. Claim: "Deployed successfully, endpoint responding"
+```
+
+**Incorrect Patterns:**
+- ❌ "Deployment command succeeded" (didn't check endpoint)
+- ❌ "Should be live now" (no verification)
+- ❌ "Deployed to staging" (didn't verify it works)
+
+**Environment-Specific Verification:**
+
+**Staging:**
+```
+1. Deploy to staging
+2. Run full test suite against staging
+3. Check all endpoints
+4. Verify database migrations
+5. Test with production-like data
+```
+
+**Production:**
+```
+1. Deploy to production
+2. Verify zero-downtime deployment
+3. Check health endpoints
+4. Monitor error rates
+5. Verify key user flows
+6. Keep rollback ready
+```
+
+**Deployment Types:**
+- **Blue-green**: Verify both environments
+- **Canary**: Monitor canary metrics
+- **Rolling**: Verify each batch
+- **Feature flags**: Verify flag state
+
+**Troubleshooting:**
+- **Deployment succeeds but service down**: Check logs, restart
+- **Configuration mismatch**: Verify environment variables
+- **Database migration failed**: Rollback, investigate
+- **Partial deployment**: Complete or rollback fully
+
+## Performance Verification
+
+**Correct Pattern:**
+```
+1. Run performance benchmark
+2. Record baseline: 150ms average
+3. Implement optimization
+4. Run benchmark again
+5. Record new measurement: 45ms average
+6. Calculate improvement: 70% faster
+7. Run multiple times to confirm consistency
+8. Claim: "Performance improved 70% (150ms → 45ms, 10 runs)"
+```
+
+**Incorrect Patterns:**
+- ❌ "Should be faster now" (no measurement)
+- ❌ "Looks quicker" (subjective)
+- ❌ Single measurement (could be outlier)
+
+**Benchmark Requirements:**
+- **Sample size**: Minimum 10 runs
+- **Statistical significance**: Calculate standard deviation
+- **Cold vs warm**: Test both scenarios
+- **Load levels**: Test under different loads
+- **Environment**: Same environment for before/after
+
+**Metrics to Measure:**
+- **Response time**: p50, p95, p99 latency
+- **Throughput**: Requests per second
+- **Resource usage**: CPU, memory, disk I/O
+- **Database queries**: Query count, duration
+- **Network**: Bandwidth, connection count
+
+**Edge Cases:**
+- Performance regression: Identify cause, revert if needed
+- Inconsistent results: Check for external factors
+- Different under load: Load test before claiming
+- Memory leaks: Profile memory over time
+
+## Security Verification
+
+**Correct Pattern:**
+```
+1. Run security scanner: bandit -r .
+2. Read full report
+3. Review each finding
+4. Document: "3 high, 2 medium, 5 low"
+5. Address critical issues
+6. Re-run scanner
+7. Claim: "Security scan: 0 high, 0 medium, 5 low (accepted)"
+```
+
+**Incorrect Patterns:**
+- ❌ "Looks secure" (no scan)
+- ❌ "No obvious vulnerabilities" (didn't scan)
+- ❌ "Should be safe" (assumption)
+
+**Security Layers to Verify:**
+
+**Static Analysis:**
+```
+1. Code scanning: bandit, semgrep
+2. Dependency scanning: safety check
+3. Secret scanning: git secrets
+4. License compliance: license checker
+```
+
+**Dynamic Analysis:**
+```
+1. Penetration testing: OWASP ZAP
+2. Fuzzing: Input validation
+3. Load testing: DoS resistance
+4. Authentication testing: Auth flows
+```
+
+**Security Checklist:**
+- [ ] Input validation implemented
+- [ ] SQL injection prevention
+- [ ] XSS prevention
+- [ ] CSRF protection
+- [ ] Authentication secure
+- [ ] Authorization checks
+- [ ] Secrets not in code
+- [ ] HTTPS enforced
+- [ ] Headers configured
+- [ ] Dependencies updated
+
+**Troubleshooting:**
+- **False positives**: Document and suppress with justification
+- **Critical findings**: Fix immediately, don't suppress
+- **Dependency vulnerabilities**: Update or replace
+- **Configuration issues**: Fix in deployment configs
+
+## Documentation Verification
+
+**Correct Pattern:**
+```
+1. Write documentation
+2. Have someone else read it
+3. Ask them to follow steps
+4. Observe if they succeed without questions
+5. Fix confusing parts
+6. Repeat until successful
+7. Claim: "Documentation verified with fresh user"
+```
+
+**Incorrect Patterns:**
+- ❌ "Documentation complete" (not tested)
+- ❌ "Clear to me" (author bias)
+- ❌ "Should be understandable" (no verification)
+
+**Documentation Types:**
+
+**API Documentation:**
+```
+1. Document all endpoints
+2. Include request/response examples
+3. Document error codes
+4. Test with curl/Postman
+5. Verify examples work
+```
+
+**User Documentation:**
+```
+1. Write step-by-step guide
+2. Include screenshots
+3. Test with fresh user
+4. Record where they get stuck
+5. Improve those sections
+```
+
+**Developer Documentation:**
+```
+1. Document setup steps
+2. Fresh clone, follow steps
+3. Verify can build and run
+4. Document all prerequisites
+5. Test on clean machine
+```
+
+**Troubleshooting:**
+- **User gets stuck**: Add more detail, screenshots
+- **Prerequisites missing**: Document all dependencies
+- **Examples don't work**: Verify and fix examples
+- **Outdated documentation**: Regular review and updates
+
+## CI/CD Pipeline Verification
+
+**Correct Pattern:**
+```
+1. Push to branch
+2. Watch CI pipeline run
+3. Verify all stages pass: build, test, lint, security
+4. Check pipeline logs for warnings
+5. Verify artifacts created
+6. Claim: "CI pipeline passes, all stages green"
+```
+
+**Pipeline Stages to Verify:**
+- **Build**: Code compiles, artifacts created
+- **Test**: All tests pass, coverage threshold met
+- **Lint**: Code style compliance
+- **Security**: No vulnerabilities found
+- **Deploy**: Successful deployment to target
+
+**Edge Cases:**
+- Flaky pipeline: Investigate root cause, fix
+- Timeout issues: Optimize slow stages
+- Cache issues: Clear cache, rebuild
+- Secrets missing: Verify environment variables
+
+## Database Migration Verification
+
+**Correct Pattern:**
+```
+1. Backup database
+2. Run migration on backup
+3. Verify data integrity
+4. Test rollback
+5. Run on staging
+6. Verify application works
+7. Document any data changes
+8. Claim: "Migration tested, rollback verified"
+```
+
+**Critical Checks:**
+- **Data loss prevention**: Verify no data dropped
+- **Rollback tested**: Must be able to undo
+- **Application compatibility**: App works during migration
+- **Performance impact**: Migration doesn't lock tables long
+
+**Edge Cases:**
+- Large tables: Use online migration tools
+- Zero-downtime: Test deployment order
+- Data transformation: Verify transformation logic
+- Foreign key constraints: Check constraint violations
+
+## The Universal Pattern
+
+All verification follows this structure:
+
+```
+1. IDENTIFY: What proves this claim?
+2. RUN: Execute the full verification command
+3. READ: Complete output, not just summary
+4. ANALYZE: Does evidence support claim?
+5. DECIDE:
+   - If YES: Claim with evidence
+   - If NO: Report actual state with evidence
+```
+
+**Never skip steps. Never assume. Always verify.**