fraim-framework 2.0.26 → 2.0.30

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (104) hide show
  1. package/.github/workflows/deploy-fraim.yml +1 -1
  2. package/dist/registry/scripts/build-scripts-generator.js +205 -0
  3. package/dist/registry/scripts/cleanup-branch.js +258 -0
  4. package/dist/registry/scripts/evaluate-code-quality.js +66 -0
  5. package/dist/registry/scripts/exec-with-timeout.js +142 -0
  6. package/dist/registry/scripts/fraim-config.js +61 -0
  7. package/dist/registry/scripts/generate-engagement-emails.js +630 -0
  8. package/dist/registry/scripts/generic-issues-api.js +100 -0
  9. package/dist/registry/scripts/newsletter-helpers.js +731 -0
  10. package/dist/registry/scripts/openapi-generator.js +664 -0
  11. package/dist/registry/scripts/performance/profile-server.js +390 -0
  12. package/dist/registry/scripts/run-thank-you-workflow.js +92 -0
  13. package/dist/registry/scripts/send-newsletter-simple.js +85 -0
  14. package/dist/registry/scripts/send-thank-you-emails.js +54 -0
  15. package/dist/registry/scripts/validate-openapi-limits.js +311 -0
  16. package/dist/registry/scripts/validate-test-coverage.js +262 -0
  17. package/dist/registry/scripts/verify-test-coverage.js +66 -0
  18. package/dist/src/cli/commands/init.js +14 -12
  19. package/dist/src/cli/commands/sync.js +19 -2
  20. package/dist/src/cli/fraim.js +24 -22
  21. package/dist/src/cli/setup/first-run.js +13 -6
  22. package/dist/src/fraim/config-loader.js +0 -8
  23. package/dist/src/fraim/db-service.js +26 -15
  24. package/dist/src/fraim/issues.js +67 -0
  25. package/dist/src/fraim/setup-wizard.js +1 -69
  26. package/dist/src/fraim/types.js +0 -11
  27. package/dist/src/fraim-mcp-server.js +272 -18
  28. package/dist/src/utils/git-utils.js +1 -1
  29. package/dist/src/utils/version-utils.js +32 -0
  30. package/dist/tests/debug-tools.js +79 -0
  31. package/dist/tests/esm-compat.js +11 -0
  32. package/dist/tests/test-chalk-esm-issue.js +159 -0
  33. package/dist/tests/test-chalk-real-world.js +265 -0
  34. package/dist/tests/test-chalk-regression.js +327 -0
  35. package/dist/tests/test-chalk-resolution-issue.js +304 -0
  36. package/dist/tests/test-cli.js +0 -2
  37. package/dist/tests/test-fraim-install-chalk-issue.js +254 -0
  38. package/dist/tests/test-fraim-issues.js +59 -0
  39. package/dist/tests/test-genericization.js +1 -3
  40. package/dist/tests/test-mcp-connection.js +166 -0
  41. package/dist/tests/test-mcp-issue-integration.js +144 -0
  42. package/dist/tests/test-mcp-lifecycle-methods.js +312 -0
  43. package/dist/tests/test-node-compatibility.js +71 -0
  44. package/dist/tests/test-npm-install.js +66 -0
  45. package/dist/tests/test-npm-resolution-diagnostic.js +140 -0
  46. package/dist/tests/test-session-rehydration.js +145 -0
  47. package/dist/tests/test-standalone.js +2 -8
  48. package/dist/tests/test-sync-version-update.js +93 -0
  49. package/dist/tests/test-telemetry.js +190 -0
  50. package/package.json +10 -8
  51. package/registry/agent-guardrails.md +62 -54
  52. package/registry/rules/agent-success-criteria.md +52 -0
  53. package/registry/rules/agent-testing-guidelines.md +502 -502
  54. package/registry/rules/communication.md +121 -121
  55. package/registry/rules/continuous-learning.md +54 -54
  56. package/registry/rules/ephemeral-execution.md +10 -5
  57. package/registry/rules/hitl-ppe-record-analysis.md +302 -302
  58. package/registry/rules/local-development.md +251 -251
  59. package/registry/rules/software-development-lifecycle.md +104 -104
  60. package/registry/rules/successful-debugging-patterns.md +482 -478
  61. package/registry/rules/telemetry.md +67 -0
  62. package/registry/scripts/build-scripts-generator.ts +216 -215
  63. package/registry/scripts/cleanup-branch.ts +303 -284
  64. package/registry/scripts/code-quality-check.sh +559 -559
  65. package/registry/scripts/detect-tautological-tests.sh +38 -38
  66. package/registry/scripts/evaluate-code-quality.ts +1 -1
  67. package/registry/scripts/generate-engagement-emails.ts +744 -744
  68. package/registry/scripts/generic-issues-api.ts +110 -150
  69. package/registry/scripts/newsletter-helpers.ts +874 -874
  70. package/registry/scripts/openapi-generator.ts +695 -693
  71. package/registry/scripts/performance/profile-server.ts +5 -3
  72. package/registry/scripts/prep-issue.sh +468 -455
  73. package/registry/scripts/validate-openapi-limits.ts +366 -365
  74. package/registry/scripts/validate-test-coverage.ts +280 -280
  75. package/registry/scripts/verify-pr-comments.sh +70 -70
  76. package/registry/scripts/verify-test-coverage.ts +1 -1
  77. package/registry/templates/bootstrap/ARCHITECTURE-TEMPLATE.md +53 -53
  78. package/registry/templates/evidence/Implementation-BugEvidence.md +85 -85
  79. package/registry/templates/evidence/Implementation-FeatureEvidence.md +120 -120
  80. package/registry/templates/marketing/HBR-ARTICLE-TEMPLATE.md +66 -0
  81. package/registry/workflows/bootstrap/create-architecture.md +2 -2
  82. package/registry/workflows/bootstrap/evaluate-code-quality.md +3 -3
  83. package/registry/workflows/bootstrap/verify-test-coverage.md +2 -2
  84. package/registry/workflows/customer-development/insight-analysis.md +156 -156
  85. package/registry/workflows/customer-development/interview-preparation.md +421 -421
  86. package/registry/workflows/customer-development/strategic-brainstorming.md +146 -146
  87. package/registry/workflows/customer-development/thank-customers.md +193 -191
  88. package/registry/workflows/customer-development/weekly-newsletter.md +362 -352
  89. package/registry/workflows/improve-fraim/contribute.md +32 -0
  90. package/registry/workflows/improve-fraim/file-issue.md +32 -0
  91. package/registry/workflows/marketing/hbr-article.md +73 -0
  92. package/registry/workflows/performance/analyze-performance.md +63 -59
  93. package/registry/workflows/product-building/design.md +3 -2
  94. package/registry/workflows/product-building/implement.md +4 -3
  95. package/registry/workflows/product-building/prep-issue.md +28 -17
  96. package/registry/workflows/product-building/resolve.md +3 -2
  97. package/registry/workflows/product-building/retrospect.md +3 -2
  98. package/registry/workflows/product-building/spec.md +5 -4
  99. package/registry/workflows/product-building/test.md +3 -2
  100. package/registry/workflows/quality-assurance/iterative-improvement-cycle.md +562 -562
  101. package/registry/workflows/replicate/website-discovery-analysis.md +3 -3
  102. package/registry/workflows/reviewer/review-implementation-vs-design-spec.md +632 -632
  103. package/registry/workflows/reviewer/review-implementation-vs-feature-spec.md +669 -669
  104. package/tsconfig.json +2 -1
@@ -1,632 +1,632 @@
1
- # Review Implementation vs Design Spec
2
-
3
- ## INTENT
4
- To systematically verify that the implementation matches the approved technical design (RFC), ensuring all technical requirements, test cases, and validation scenarios are complete.
5
-
6
- ## PRINCIPLES
7
- - **Design-Driven Validation**: Every requirement in the RFC must be verified
8
- - **Test Matrix Compliance**: All test cases from design spec must exist and pass
9
- - **Architecture Alignment**: Implementation must follow design decisions
10
- - **Evidence-Based**: All claims must be backed by actual evidence
11
- - **Gap Identification**: Systematically identify what's missing, not just what exists
12
-
13
- ## REVIEW WORKFLOW
14
-
15
- ### Step 1: Issue Identification and Evidence Loading
16
- - Get {issue_number} from context
17
- - Verify issue has `phase:impl` and `status:needs-review` labels
18
- - Locate RFC: `docs/rfcs/{issue}-*.md`
19
- - If RFC doesn't exist: ❌ BLOCKER - Cannot review without design spec. Request design phase first.
20
- - **Determine Issue Type**: Check if this is a bug fix or feature
21
- - Check RFC title/description or issue title for "bug", "fix", "error", etc.
22
- - Bug fixes require regression tests (see Step 4.4 Regression Test Verification)
23
- - **Load Implementation Evidence**: Read `docs/evidence/{issue}-implementation-evidence.md`
24
- - If evidence doesn't exist: ❌ BLOCKER - Implementation agent must create evidence document first
25
- - Review evidence for completeness and quality
26
- - **Check for Existing Feedback**: Check if `docs/evidence/{issue}-design-reviewer-feedback.md` exists
27
- - If exists, this is an iteration (check iteration count)
28
- - Track iteration number (max 3 iterations)
29
-
30
- ### Step 2: RFC Analysis
31
- **MUST read and extract:**
32
- - Technical requirements (API changes, schema changes, etc.)
33
- - Test Matrix section (unit, integration, E2E tests required)
34
- - Validation Plan section (all validation scenarios)
35
- - Architecture decisions (service boundaries, patterns)
36
- - Risk mitigations (how they were addressed)
37
- - Observability requirements (logs, metrics, alerts)
38
-
39
- **Create a checklist mapping:**
40
- - List each requirement from RFC
41
- - List each test case from Test Matrix
42
- - List each validation scenario from Validation Plan
43
- - This becomes your review checklist
44
-
45
- ### Step 3: Implementation Code Review
46
- **For each technical requirement in RFC:**
47
- - [ ] Verify requirement is implemented in code
48
- - Search codebase for implementation
49
- - Check file paths match design
50
- - Verify code exists and is not just stubbed
51
- - [ ] Check code follows architecture decisions
52
- - Service boundaries match design
53
- - Patterns match design (e.g., spike-first, deterministic separation)
54
- - No over-engineering beyond design scope
55
- - [ ] Verify API/schema changes match RFC exactly
56
- - Check OpenAPI spec matches (if applicable)
57
- - Check database schema matches (if applicable)
58
- - Verify field names, types, constraints match
59
- - [ ] Check error handling matches design
60
- - Error responses match design
61
- - Error codes match design
62
- - Error messages match design
63
- - [ ] Verify observability (logs, metrics) matches design
64
- - Log statements exist where specified
65
- - Metrics tracked where specified
66
- - Alerts configured where specified
67
-
68
- **Document findings:**
69
- - Create table: Requirement | Status | Evidence Location | Notes
70
-
71
- ### Step 4: Test Matrix Validation
72
- **For each test type in RFC Test Matrix:**
73
-
74
- #### Unit Tests
75
- - [ ] **Count Required**: How many unit tests does RFC specify?
76
- - [ ] **Count Found**: How many unit test files exist? (`test-{issue}-*.ts` or similar)
77
- - [ ] **Verify Existence**: Check each listed test case exists
78
- - [ ] **Run Tests**: Execute `npm run test test-{issue}-*.ts` (or appropriate command)
79
- - [ ] **Verify Exit Code**: Must be 0 (success)
80
- - [ ] **Check Output**: All tests must pass (not just "some tests pass")
81
- - [ ] **Document Results**: Include actual test output in review evidence
82
-
83
- #### Integration Tests
84
- - [ ] **Count Required**: How many integration tests does RFC specify?
85
- - [ ] **Count Found**: How many integration test files exist?
86
- - [ ] **Verify Existence**: Check each listed test case exists
87
- - [ ] **Run Tests**: Execute integration tests with proper mocks
88
- - [ ] **Verify Service Interactions**: Check mocks validate service calls match design
89
- - [ ] **Verify Exit Code**: Must be 0 (success)
90
- - [ ] **Document Results**: Include actual test output in review evidence
91
-
92
- #### E2E Tests
93
- - [ ] **Count Required**: How many E2E tests does RFC specify?
94
- - [ ] **Count Found**: How many E2E test files exist?
95
- - [ ] **Verify Existence**: Check each listed test case exists
96
- - [ ] **Run Tests**: Execute E2E tests (may require server running)
97
- - [ ] **Verify End-to-End Flow**: Complete workflow must work
98
- - [ ] **Verify Exit Code**: Must be 0 (success)
99
- - [ ] **Document Results**: Include actual test output in review evidence
100
-
101
- **If test matrix says "X test cases" but only Y exist:**
102
- - ❌ BLOCKER: Missing test cases
103
- - Must identify which tests are missing (list specific test names)
104
- - Cannot approve until all tests from matrix exist and pass
105
-
106
- **Document findings:**
107
- - Create table: Test Type | Required | Found | Passing | Evidence | Missing Tests
108
-
109
- #### Test Quality Validation (Anti-Pattern Detection)
110
- **🚨 CRITICAL: Verify tests validate runtime behavior, not code structure**
111
-
112
- **For each test file found (test-{issue}-*.ts):**
113
- - [ ] **Check for Static Analysis Anti-Pattern**: Verify tests don't use `fs.readFileSync()` for code structure checks
114
- - Check: `grep -E "fs\.readFileSync|readFileSync" test-{issue}-*.ts`
115
- - ❌ BLOCKER if found - tests are doing static analysis, not runtime testing
116
- - Expected: Tests should execute code, not read source files
117
- - Reference: `retrospectives/issue-723-weak-test-validation-anti-pattern-postmortem.md`
118
- - [ ] **Verify Runtime Behavior Testing**: Check tests actually execute code paths
119
- - Review test code to ensure it calls actual functions/services
120
- - Verify tests check observable outcomes (API responses, database state, logs)
121
- - ❌ BLOCKER if tests only check code structure without execution
122
- - [ ] **Check for Mock Anti-Pattern**: Verify tests don't mock the core functionality being tested
123
- - Check: Are mocks used for dependencies (✅) or for the thing being tested (❌)?
124
- - ❌ BLOCKER if core functionality is mocked
125
- - Expected: Mock dependencies, test actual implementation
126
- - [ ] **Verify State Validation**: Check tests validate state changes
127
- - Tests should check database state, API responses, or service interactions
128
- - ❌ BLOCKER if tests don't validate any state changes
129
- - Expected: Tests verify before/after state or observable outcomes
130
-
131
- **Document findings:**
132
- - Create table: Test File | Static Analysis Check | Runtime Behavior | Mock Usage | State Validation | Status
133
-
134
- #### Regression Test Verification (For Bug Fixes)
135
- **If this is a bug fix (check RFC or issue description):**
136
- - [ ] **Identify Bug**: Understand what the bug was from RFC/issue
137
- - [ ] **Check for Regression Test**: Verify test file includes regression test for the bug
138
- - Check test file for test case that reproduces the bug
139
- - Test should be named something like "regression test for bug X" or similar
140
- - [ ] **Verify Test Pattern**: Regression test must follow correct pattern
141
- - ❌ BLOCKER if test passes/fails regardless of bug (wrong pattern)
142
- - ✅ CORRECT: Test fails with bug → Test passes with fix
143
- - ❌ WRONG: Test passes with bug AND with fix (doesn't reproduce bug)
144
- - ❌ WRONG: Test fails with bug AND with fix (environmental issue)
145
- - [ ] **Verify Test Behavior** (if possible):
146
- - Ideally: Revert fix temporarily, run test → should fail
147
- - Reapply fix, run test → should pass
148
- - If not possible, verify test logic would fail with buggy code
149
-
150
- **Document findings:**
151
- - Bug Fix: ✅ Yes / ❌ No
152
- - Regression Test: ✅ Found / ❌ Missing
153
- - Test Pattern: ✅ Correct (fails with bug, passes with fix) / ❌ Wrong pattern
154
- - Notes: [Any issues with regression test]
155
-
156
- ### Step 4.5: Common Error Pattern Checks
157
- **🚨 CRITICAL: Check for common test and code quality issues**
158
-
159
- #### Test Structure Validation
160
- **For each test file found (test-{issue}-*.ts):**
161
- - [ ] **BaseTestCase Extension**: Verify test cases extend `BaseTestCase` interface
162
- - Check: `grep -E "extends BaseTestCase|interface.*extends BaseTestCase" test-{issue}-*.ts`
163
- - ❌ BLOCKER if test cases don't extend BaseTestCase
164
- - Expected pattern: `interface MyTestCase extends BaseTestCase { ... }`
165
- - All test case objects must use interface that extends BaseTestCase
166
- - [ ] **Main Function**: Verify test file has `main()` function
167
- - Check: `grep -E "async function main\(\)|function main\(\)" test-{issue}-*.ts`
168
- - ❌ BLOCKER if main() function missing
169
- - Expected pattern: `async function main() { ... runTests(...) }`
170
- - Must have `main().catch(console.error)` at end of file
171
- - [ ] **runTests Usage**: Verify test file uses `runTests()` function
172
- - Check: `grep "runTests" test-{issue}-*.ts`
173
- - ❌ BLOCKER if runTests() not used
174
- - Expected: `runTests(TEST_CASES, runMyTest, 'Test Suite Name')`
175
- - Must import runTests from './test-utils' or './test-utils.ts'
176
-
177
- **Document findings:**
178
- - Create table: Test File | BaseTestCase | main() | runTests() | Status
179
-
180
- #### Code Quality Check
181
- - [ ] **Run Code Quality Check**: Execute `.ai-agents/scripts/code-quality-check.sh pre-pr`
182
- - Command: `bash .ai-agents/scripts/code-quality-check.sh pre-pr`
183
- - Verify exit code is 0 (success)
184
- - Check for critical failures (❌) vs warnings (⚠️)
185
- - ❌ BLOCKER if critical checks fail (e.g., `as any` usage, TypeScript errors)
186
- - ⚠️ WARNING if warnings exist (document in feedback)
187
- - [ ] **Review Quality Check Output**:
188
- - Check for `as any` type bypassing in src/
189
- - Check TypeScript compilation passes
190
- - Check linter passes (warnings acceptable, errors are blockers)
191
- - Check all test files referenced in evidence have been executed
192
-
193
- **Document findings:**
194
- - Code Quality Check: ✅ Passed / ❌ Failed / ⚠️ Warnings
195
- - Critical Issues: [list any critical failures]
196
- - Warnings: [list any warnings]
197
-
198
- #### Evidence File Location Validation
199
- - [ ] **Check Evidence Files**: Verify evidence files are ONLY in `docs/evidence/` folder
200
- - Check: `find . -name "*evidence*.md" -not -path "./docs/evidence/*" -not -path "./.git/*" -not -path "./node_modules/*"`
201
- - ❌ BLOCKER if evidence files found outside `docs/evidence/`
202
- - Expected: All evidence files should be in `docs/evidence/{issue}-*.md`
203
- - [ ] **Check for Evidence in Git**: Verify evidence folder is gitignored
204
- - Check: `git check-ignore docs/evidence/` should return `docs/evidence/`
205
- - ⚠️ WARNING if evidence folder not in .gitignore (should be gitignored)
206
- - [ ] **Check PR for Evidence Files**: Verify no evidence files committed to repo
207
- - Check git status for any files in docs/evidence/
208
- - ❌ BLOCKER if evidence files are staged/committed
209
-
210
- **Document findings:**
211
- - Evidence Location Check: ✅ All in docs/evidence/ / ❌ Found outside
212
- - Git Ignore Check: ✅ Ignored / ⚠️ Not ignored
213
- - Committed Evidence: ✅ None / ❌ Found in commits
214
-
215
- **If any blockers found:**
216
- - ❌ BLOCKER: Must fix before approval
217
- - List specific files and issues
218
- - Cannot approve until all blockers resolved
219
-
220
- ### Step 5: Validation Plan Execution
221
- **For each validation scenario in RFC Validation Plan:**
222
- - [ ] **Scenario Identified**: Map each scenario from validation plan
223
- - [ ] **Evidence Found**: Check PR evidence for scenario validation
224
- - Look in PR comments for evidence
225
- - Check for screenshots, curl outputs, test results
226
- - Verify evidence shows actual execution (not just "tested")
227
- - [ ] **Method Verified**: Ensure validation method matches (UI, API, DB, etc.)
228
- - If RFC says "API validation", check for curl/Postman results
229
- - If RFC says "Browser validation", check for screenshots/recordings
230
- - If RFC says "Database validation", check for DB query results
231
- - [ ] **Result Verified**: Ensure scenario actually passes (not just "tested")
232
- - Check evidence shows success, not just attempt
233
- - Verify no "pending" or "TODO" markers
234
-
235
- **If validation plan has 10 scenarios but only 7 have evidence:**
236
- - ❌ BLOCKER: Missing validation evidence
237
- - Must identify which scenarios are missing (list specific scenario names)
238
- - Cannot approve until all scenarios validated with evidence
239
-
240
- **Document findings:**
241
- - Create table: Scenario | Method | Evidence | Status | Missing Evidence
242
-
243
- ### Step 6: Architecture Compliance Check
244
- **Verify implementation matches design:**
245
- - [ ] Service boundaries match design
246
- - Check service files match design boundaries
247
- - Verify no cross-boundary violations
248
- - [ ] Patterns match design (e.g., spike-first, deterministic separation)
249
- - Check code follows patterns specified in RFC
250
- - Verify no anti-patterns introduced
251
- - [ ] Dependencies match design
252
- - Check package.json matches design dependencies
253
- - Verify no unexpected dependencies added
254
- - [ ] No over-engineering beyond design scope
255
- - Check code doesn't add features not in RFC
256
- - Verify minimal implementation (not over-engineered)
257
-
258
- **Document findings:**
259
- - List any architecture deviations
260
- - Note if deviations are acceptable or blockers
261
-
262
- ### Step 7: Risk Mitigation Verification
263
- **For each risk in RFC Risks & Mitigations:**
264
- - [ ] Verify mitigation is implemented
265
- - Check code implements mitigation strategy
266
- - Verify mitigation code exists and is not just commented
267
- - [ ] Check mitigation actually addresses the risk
268
- - Verify mitigation logic matches risk description
269
- - Check mitigation is complete (not partial)
270
- - [ ] Verify observability for risk detection
271
- - Check logs/alerts exist for risk scenarios
272
- - Verify monitoring is in place
273
-
274
- **Document findings:**
275
- - Create table: Risk | Mitigation Status | Implementation Location | Notes
276
-
277
- ### Step 8: Evidence Quality Review
278
- **Verify PR evidence quality:**
279
- - [ ] All test outputs included (not just "tests pass")
280
- - Check for actual test command output
281
- - Verify exit codes shown
282
- - Check for test case names and results
283
- - [ ] All validation scenarios documented with results
284
- - Check each scenario has evidence
285
- - Verify evidence shows actual execution
286
- - [ ] No "pending" items that could be validated automatically
287
- - Flag any "pending" items that could be tested
288
- - Check for "TODO" or "needs manual validation" markers
289
- - [ ] Build verification included
290
- - Check for `npm run build` output
291
- - Verify build succeeds
292
- - [ ] Quality gate results included
293
- - Check for quality gate script output
294
- - Verify all gates pass (or warnings documented)
295
-
296
- **Document findings:**
297
- - List any evidence quality issues
298
- - Note if evidence is sufficient or needs improvement
299
-
300
- ### Step 9: Review Decision and Feedback Creation
301
-
302
- **Check Iteration Count:**
303
- - If `docs/evidence/{issue}-design-reviewer-feedback.md` exists, check iteration number
304
- - Maximum 3 iterations allowed
305
- - If iteration 3 and still issues: ❌ REJECT (max iterations reached)
306
-
307
- **Options:**
308
- - ✅ **APPROVE**: All requirements met, tests pass, evidence complete
309
- - ❌ **REJECT**: Missing requirements, tests fail, or evidence incomplete (after max iterations)
310
- - ⚠️ **REQUEST CHANGES**: Minor gaps that need addressing (iteration < 3)
311
-
312
- **If REQUEST CHANGES (iteration < 3):**
313
- - **Create Feedback Document**: Create `docs/evidence/{issue}-design-reviewer-feedback.md`
314
- - Use template below for feedback structure
315
- - List specific blockers with evidence locations
316
- - Provide actionable feedback for each issue
317
- - Include iteration number
318
- - Label issue `status:wip` (remove `status:needs-review`)
319
- - Add PR comment referencing feedback document
320
- - Implementation agent must address feedback and update evidence
321
- - Implementation agent re-submits by marking `status:needs-review` again
322
-
323
- **If REJECT (max iterations reached):**
324
- - Create final feedback document with all remaining issues
325
- - Label issue `status:wip` (remove `status:needs-review`)
326
- - Add PR comment with final rejection notice
327
- - Implementation agent must address all issues before re-submission
328
-
329
- **If APPROVE:**
330
- - **Create Summary for Feature Spec Review**: Create `docs/evidence/{issue}-design-review-summary.md`
331
- - Summarize what was reviewed and approved
332
- - Include key findings and evidence locations
333
- - Provide context for feature spec review agent
334
- - Label issue `status:design-review-passed` (remove `status:needs-review`)
335
- - Add PR comment confirming design spec compliance
336
- - Issue ready for Feature Spec Review (next workflow)
337
-
338
- ## FEEDBACK DOCUMENT TEMPLATE
339
-
340
- Create `docs/evidence/{issue}-design-reviewer-feedback.md`:
341
-
342
- ```markdown
343
- # Design Review Feedback - Issue #{issue}
344
-
345
- ## Iteration Information
346
- - Iteration Number: {1, 2, or 3}
347
- - Review Date: {date}
348
- - Reviewer: Design Spec Review Agent
349
-
350
- ## Review Summary
351
- - RFC Location: `docs/rfcs/{issue}-*.md`
352
- - Evidence Reviewed: `docs/evidence/{issue}-implementation-evidence.md`
353
- - Overall Status: ⚠️ REQUEST CHANGES
354
-
355
- ## Issues Found
356
-
357
- ### Critical Blockers (Must Fix)
358
- 1. **Issue Title**
359
- - **Location**: `src/file.ts:line`
360
- - **Problem**: [Specific problem description]
361
- - **Expected**: [What RFC requires]
362
- - **Found**: [What implementation has]
363
- - **Action Required**: [Specific fix needed]
364
-
365
- 2. **Issue Title**
366
- - [Same structure]
367
-
368
- ### Missing Requirements
369
- 1. **Requirement Name**
370
- - **RFC Section**: [Section reference]
371
- - **What's Missing**: [Specific requirement]
372
- - **Action Required**: [What needs to be implemented]
373
-
374
- ### Test Matrix Issues
375
- 1. **Missing Test**: Test case name
376
- - **RFC Test Matrix**: [Reference]
377
- - **Action Required**: Create test file `test-{issue}-*.ts` with test case
378
-
379
- ### Test Structure Issues (Common Error Patterns)
380
- 1. **Test Not Extending BaseTestCase**: `test-{issue}-*.ts`
381
- - **Problem**: Test case interface doesn't extend BaseTestCase
382
- - **Expected**: `interface MyTestCase extends BaseTestCase { ... }`
383
- - **Action Required**: Update test case interface to extend BaseTestCase
384
-
385
- 2. **Missing main() Function**: `test-{issue}-*.ts`
386
- - **Problem**: Test file doesn't have main() function
387
- - **Expected**: `async function main() { ... runTests(...) }` with `main().catch(console.error)`
388
- - **Action Required**: Add main() function that calls runTests()
389
-
390
- 3. **Not Using runTests()**: `test-{issue}-*.ts`
391
- - **Problem**: Test file doesn't use runTests() function
392
- - **Expected**: Import runTests from './test-utils' and call it in main()
393
- - **Action Required**: Refactor to use runTests() function
394
-
395
- ### Test Quality Issues (Anti-Patterns)
396
- 1. **Static Analysis Anti-Pattern**: `test-{issue}-*.ts`
397
- - **Problem**: Tests use `fs.readFileSync()` to check code structure instead of runtime behavior
398
- - **Expected**: Tests should execute code and validate outcomes, not read source files
399
- - **Action Required**: Rewrite tests to validate runtime behavior
400
- - **Reference**: `retrospectives/issue-723-weak-test-validation-anti-pattern-postmortem.md`
401
-
402
- 2. **No Runtime Behavior Validation**: `test-{issue}-*.ts`
403
- - **Problem**: Tests don't execute actual code paths or validate observable outcomes
404
- - **Expected**: Tests should call functions/services and check results
405
- - **Action Required**: Add runtime behavior validation to tests
406
-
407
- 3. **Mocking Core Functionality**: `test-{issue}-*.ts`
408
- - **Problem**: Tests mock the thing being tested instead of dependencies
409
- - **Expected**: Mock dependencies, test actual implementation
410
- - **Action Required**: Refactor mocks to only mock dependencies
411
-
412
- 4. **No State Validation**: `test-{issue}-*.ts`
413
- - **Problem**: Tests don't validate state changes (database, API responses, etc.)
414
- - **Expected**: Tests should verify before/after state or observable outcomes
415
- - **Action Required**: Add state validation to tests
416
-
417
- ### Regression Test Issues (For Bug Fixes)
418
- 1. **Missing Regression Test**: `test-{issue}-*.ts`
419
- - **Problem**: Bug fix doesn't have regression test
420
- - **Expected**: Regression test that fails with bug, passes with fix
421
- - **Action Required**: Add regression test following correct pattern
422
-
423
- 2. **Wrong Regression Test Pattern**: `test-{issue}-*.ts`
424
- - **Problem**: Test passes/fails regardless of bug (doesn't reproduce bug)
425
- - **Expected**: Test fails with bug → Test passes with fix
426
- - **Action Required**: Fix regression test to follow correct pattern
427
- - **Reference**: `retrospectives/task-api-calendar-event-id-bug-postmortem.md`
428
-
429
- ### Code Quality Issues
430
- 1. **Code Quality Check Failed**: [Specific issue]
431
- - **Problem**: [What failed - e.g., "as any" usage, TypeScript errors]
432
- - **Action Required**: Fix code quality issues and re-run check
433
- - **Command**: `bash .ai-agents/scripts/code-quality-check.sh pre-pr`
434
-
435
- ### Evidence File Location Issues
436
- 1. **Evidence File Outside docs/evidence/**: [File path]
437
- - **Problem**: Evidence file found outside `docs/evidence/` folder
438
- - **Expected**: All evidence files should be in `docs/evidence/{issue}-*.md`
439
- - **Action Required**: Move evidence file to `docs/evidence/` folder
440
-
441
- ### Validation Plan Issues
442
- 1. **Missing Validation**: Scenario name
443
- - **RFC Validation Plan**: [Reference]
444
- - **Action Required**: Execute validation and add evidence
445
-
446
- ### Evidence Quality Issues
447
- 1. **Missing Evidence**: [What's missing]
448
- - **Action Required**: [What evidence needs to be added]
449
-
450
- ## Action Items for Implementation Agent
451
- - [ ] Fix critical blocker 1
452
- - [ ] Fix critical blocker 2
453
- - [ ] Implement missing requirement X
454
- - [ ] Add missing test case Y
455
- - [ ] Execute missing validation Z
456
- - [ ] Update evidence document with fixes
457
-
458
- ## Next Steps
459
- 1. Implementation agent addresses all feedback items
460
- 2. Implementation agent updates `docs/evidence/{issue}-implementation-evidence.md`
461
- 3. Implementation agent marks issue `status:needs-review` again
462
- 4. Design review agent will re-review (iteration {next_number})
463
- ```
464
-
465
- ## SUMMARY DOCUMENT TEMPLATE (For Feature Spec Review)
466
-
467
- Create `docs/evidence/{issue}-design-review-summary.md` when approving:
468
-
469
- ```markdown
470
- # Design Review Summary - Issue #{issue}
471
-
472
- ## Review Outcome
473
- - Status: ✅ APPROVED
474
- - Iterations: {number} (1-3)
475
- - RFC Location: `docs/rfcs/{issue}-*.md`
476
-
477
- ## What Was Reviewed
478
- - Technical Requirements: X/Y implemented ✅
479
- - Test Matrix: All test cases exist and pass ✅
480
- - Validation Plan: All scenarios validated ✅
481
- - Architecture: Matches design ✅
482
- - Risk Mitigations: All implemented ✅
483
-
484
- ## Key Findings
485
- - All RFC requirements implemented correctly
486
- - All tests from test matrix exist and pass
487
- - All validation scenarios executed with evidence
488
- - Architecture follows design decisions
489
- - No over-engineering detected
490
-
491
- ## Evidence Locations
492
- - Implementation Evidence: `docs/evidence/{issue}-implementation-evidence.md`
493
- - Test Results: [locations]
494
- - Validation Evidence: [locations]
495
-
496
- ## Notes for Feature Spec Review
497
- - Technical implementation is complete and correct
498
- - All technical requirements met
499
- - Ready for functional/user experience validation
500
- - No technical blockers for feature spec review
501
- ```
502
-
503
- ## REVIEW EVIDENCE TEMPLATE (PR Comment)
504
-
505
- Add this as a PR comment:
506
-
507
- ```markdown
508
- # Design Spec Review - Issue #{issue}
509
-
510
- ## RFC Compliance Summary
511
- - RFC Location: `docs/rfcs/{issue}-*.md`
512
- - Technical Requirements: X/Y implemented ✅/❌
513
- - Test Matrix Compliance: X/Y test types complete ✅/❌
514
- - Validation Plan Coverage: X/Y scenarios validated ✅/❌
515
- - Architecture Compliance: ✅/❌
516
- - Risk Mitigations: X/Y implemented ✅/❌
517
-
518
- ## Detailed Findings
519
-
520
- ### Technical Requirements
521
- | Requirement | Status | Evidence Location | Notes |
522
- |------------|--------|-------------------|-------|
523
- | API endpoint X | ✅ | `src/api/...` | Matches RFC |
524
- | Schema change Y | ❌ | Missing | RFC requires Z field |
525
-
526
- ### Test Matrix Validation
527
- | Test Type | Required | Found | Passing | Evidence | Missing Tests |
528
- |-----------|----------|-------|---------|----------|---------------|
529
- | Unit Tests | 5 | 5 | ✅ | `test-{issue}.ts` lines 10-50 | None |
530
- | Integration | 3 | 2 | ❌ | Partial | Missing test for scenario Z |
531
- | E2E | 1 | 0 | ❌ | Not found | E2E test not implemented |
532
-
533
- ### Test Quality Validation
534
- | Test File | Static Analysis | Runtime Behavior | Mock Usage | State Validation | Status |
535
- |-----------|----------------|-------------------|------------|------------------|--------|
536
- | test-{issue}.ts | ✅ No fs.readFileSync | ✅ Executes code | ✅ Mocks dependencies | ✅ Validates state | ✅ Pass |
537
- | test-{issue}-2.ts | ❌ Uses fs.readFileSync | ❌ No execution | ❌ Mocks core | ❌ No validation | ❌ Fail |
538
-
539
- ### Regression Test Verification (If Bug Fix)
540
- - Bug Fix: ✅ Yes / ❌ No
541
- - Regression Test: ✅ Found / ❌ Missing
542
- - Test Pattern: ✅ Correct (fails with bug, passes with fix) / ❌ Wrong pattern
543
- - Notes: [Any issues with regression test]
544
-
545
- ### Validation Plan Coverage
546
- | Scenario | Method | Evidence | Status | Missing Evidence |
547
- |----------|--------|----------|--------|------------------|
548
- | Create operation | API | PR comment | ✅ | None |
549
- | Update operation | API | Missing | ❌ | No curl output found |
550
-
551
- ### Architecture Compliance
552
- - Service Boundaries: ✅ Match design
553
- - Patterns: ✅ Follow design patterns
554
- - Dependencies: ⚠️ Extra dependency X added (not in RFC)
555
- - Over-engineering: ✅ Minimal implementation
556
-
557
- ### Risk Mitigations
558
- | Risk | Mitigation Status | Implementation Location | Notes |
559
- |------|-------------------|-------------------------|-------|
560
- | Risk X | ✅ Implemented | `src/...` | Matches design |
561
- | Risk Y | ❌ Missing | N/A | Not implemented |
562
-
563
- ### Common Error Pattern Checks
564
- - Test Structure: ✅ All tests extend BaseTestCase / ❌ Issues found
565
- - Main Functions: ✅ All tests have main() / ❌ Missing in some tests
566
- - runTests Usage: ✅ All tests use runTests() / ❌ Not used in some tests
567
- - Code Quality Check: ✅ Passed / ❌ Failed / ⚠️ Warnings
568
- - Evidence Location: ✅ All in docs/evidence/ / ❌ Found outside
569
-
570
- ### Evidence Quality
571
- - Test Outputs: ✅ Complete (all test results included)
572
- - Validation Evidence: ⚠️ Partial (missing scenario Y evidence)
573
- - Build Verification: ✅ Included
574
- - Quality Gate: ✅ Passed
575
-
576
- ## Decision
577
- ✅ APPROVE / ❌ REJECT / ⚠️ REQUEST CHANGES
578
-
579
- ## Feedback Document
580
- - Location: `docs/evidence/{issue}-design-reviewer-feedback.md`
581
- - Iteration: {1, 2, or 3}
582
-
583
- ## Blockers (if any)
584
- 1. Missing test case for scenario X (RFC Test Matrix requires it)
585
- 2. Validation scenario Y not executed (RFC Validation Plan requires it)
586
- 3. Technical requirement Z not implemented (RFC Technical Details requires it)
587
-
588
- ## Next Steps (if REQUEST CHANGES)
589
- - [ ] Implementation agent addresses feedback in `docs/evidence/{issue}-design-reviewer-feedback.md`
590
- - [ ] Implementation agent updates `docs/evidence/{issue}-implementation-evidence.md`
591
- - [ ] Implementation agent marks `status:needs-review` for re-review
592
- - [ ] Design review agent will re-review (max 3 iterations)
593
- ```
594
-
595
- ## EXAMPLES
596
-
597
- ### Good: Complete Review
598
- ```
599
- Issue #533: Design Spec Review
600
- ✅ RFC Compliance: 15/15 requirements implemented
601
- ✅ Test Matrix: All 8 test cases exist and pass
602
- ✅ Validation Plan: All 5 scenarios validated
603
- ✅ Architecture: Matches design
604
- ✅ Evidence: Complete with test outputs
605
- Decision: ✅ APPROVE
606
- ```
607
-
608
- ### Bad: Incomplete Review
609
- ```
610
- Issue #533: Design Spec Review
611
- ⚠️ RFC Compliance: 12/15 requirements (3 missing)
612
- ❌ Test Matrix: 5/8 test cases exist (3 missing)
613
- ⚠️ Validation Plan: 3/5 scenarios validated (2 missing)
614
- Decision: ❌ REJECT
615
- Blockers: Missing tests, missing validations
616
- ```
617
-
618
- ## INTEGRATION
619
-
620
- ### Status Label Flow
621
- ```
622
- phase:impl + status:needs-review
623
- ↓ (Design Spec Review)
624
- status:design-review-passed (or status:wip if rejected)
625
- ↓ (Feature Spec Review - next workflow)
626
- status:feature-review-passed
627
- ```
628
-
629
- ### After This Review
630
- - If APPROVE: Issue moves to Feature Spec Review
631
- - If REJECT: Issue returns to implementation (status:wip)
632
- - Implementation agent addresses blockers and re-submits
1
+ # Review Implementation vs Design Spec
2
+
3
+ ## INTENT
4
+ To systematically verify that the implementation matches the approved technical design (RFC), ensuring all technical requirements, test cases, and validation scenarios are complete.
5
+
6
+ ## PRINCIPLES
7
+ - **Design-Driven Validation**: Every requirement in the RFC must be verified
8
+ - **Test Matrix Compliance**: All test cases from design spec must exist and pass
9
+ - **Architecture Alignment**: Implementation must follow design decisions
10
+ - **Evidence-Based**: All claims must be backed by actual evidence
11
+ - **Gap Identification**: Systematically identify what's missing, not just what exists
12
+
13
+ ## REVIEW WORKFLOW
14
+
15
+ ### Step 1: Issue Identification and Evidence Loading
16
+ - Get {issue_number} from context
17
+ - Verify issue has `phase:impl` and `status:needs-review` labels
18
+ - Locate RFC: `docs/rfcs/{issue}-*.md`
19
+ - If RFC doesn't exist: ❌ BLOCKER - Cannot review without design spec. Request design phase first.
20
+ - **Determine Issue Type**: Check if this is a bug fix or feature
21
+ - Check RFC title/description or issue title for "bug", "fix", "error", etc.
22
+ - Bug fixes require regression tests (see Step 4.4 Regression Test Verification)
23
+ - **Load Implementation Evidence**: Read `docs/evidence/{issue}-implementation-evidence.md`
24
+ - If evidence doesn't exist: ❌ BLOCKER - Implementation agent must create evidence document first
25
+ - Review evidence for completeness and quality
26
+ - **Check for Existing Feedback**: Check if `docs/evidence/{issue}-design-reviewer-feedback.md` exists
27
+ - If exists, this is an iteration (check iteration count)
28
+ - Track iteration number (max 3 iterations)
29
+
30
+ ### Step 2: RFC Analysis
31
+ **MUST read and extract:**
32
+ - Technical requirements (API changes, schema changes, etc.)
33
+ - Test Matrix section (unit, integration, E2E tests required)
34
+ - Validation Plan section (all validation scenarios)
35
+ - Architecture decisions (service boundaries, patterns)
36
+ - Risk mitigations (how they were addressed)
37
+ - Observability requirements (logs, metrics, alerts)
38
+
39
+ **Create a checklist mapping:**
40
+ - List each requirement from RFC
41
+ - List each test case from Test Matrix
42
+ - List each validation scenario from Validation Plan
43
+ - This becomes your review checklist
44
+
45
+ ### Step 3: Implementation Code Review
46
+ **For each technical requirement in RFC:**
47
+ - [ ] Verify requirement is implemented in code
48
+ - Search codebase for implementation
49
+ - Check file paths match design
50
+ - Verify code exists and is not just stubbed
51
+ - [ ] Check code follows architecture decisions
52
+ - Service boundaries match design
53
+ - Patterns match design (e.g., spike-first, deterministic separation)
54
+ - No over-engineering beyond design scope
55
+ - [ ] Verify API/schema changes match RFC exactly
56
+ - Check OpenAPI spec matches (if applicable)
57
+ - Check database schema matches (if applicable)
58
+ - Verify field names, types, constraints match
59
+ - [ ] Check error handling matches design
60
+ - Error responses match design
61
+ - Error codes match design
62
+ - Error messages match design
63
+ - [ ] Verify observability (logs, metrics) matches design
64
+ - Log statements exist where specified
65
+ - Metrics tracked where specified
66
+ - Alerts configured where specified
67
+
68
+ **Document findings:**
69
+ - Create table: Requirement | Status | Evidence Location | Notes
70
+
71
+ ### Step 4: Test Matrix Validation
72
+ **For each test type in RFC Test Matrix:**
73
+
74
+ #### Unit Tests
75
+ - [ ] **Count Required**: How many unit tests does RFC specify?
76
+ - [ ] **Count Found**: How many unit test files exist? (`test-{issue}-*.ts` or similar)
77
+ - [ ] **Verify Existence**: Check each listed test case exists
78
+ - [ ] **Run Tests**: Execute `npm run test test-{issue}-*.ts` (or appropriate command)
79
+ - [ ] **Verify Exit Code**: Must be 0 (success)
80
+ - [ ] **Check Output**: All tests must pass (not just "some tests pass")
81
+ - [ ] **Document Results**: Include actual test output in review evidence
82
+
83
+ #### Integration Tests
84
+ - [ ] **Count Required**: How many integration tests does RFC specify?
85
+ - [ ] **Count Found**: How many integration test files exist?
86
+ - [ ] **Verify Existence**: Check each listed test case exists
87
+ - [ ] **Run Tests**: Execute integration tests with proper mocks
88
+ - [ ] **Verify Service Interactions**: Check mocks validate service calls match design
89
+ - [ ] **Verify Exit Code**: Must be 0 (success)
90
+ - [ ] **Document Results**: Include actual test output in review evidence
91
+
92
+ #### E2E Tests
93
+ - [ ] **Count Required**: How many E2E tests does RFC specify?
94
+ - [ ] **Count Found**: How many E2E test files exist?
95
+ - [ ] **Verify Existence**: Check each listed test case exists
96
+ - [ ] **Run Tests**: Execute E2E tests (may require server running)
97
+ - [ ] **Verify End-to-End Flow**: Complete workflow must work
98
+ - [ ] **Verify Exit Code**: Must be 0 (success)
99
+ - [ ] **Document Results**: Include actual test output in review evidence
100
+
101
+ **If test matrix says "X test cases" but only Y exist:**
102
+ - ❌ BLOCKER: Missing test cases
103
+ - Must identify which tests are missing (list specific test names)
104
+ - Cannot approve until all tests from matrix exist and pass
105
+
106
+ **Document findings:**
107
+ - Create table: Test Type | Required | Found | Passing | Evidence | Missing Tests
108
+
109
+ #### Test Quality Validation (Anti-Pattern Detection)
110
+ **🚨 CRITICAL: Verify tests validate runtime behavior, not code structure**
111
+
112
+ **For each test file found (test-{issue}-*.ts):**
113
+ - [ ] **Check for Static Analysis Anti-Pattern**: Verify tests don't use `fs.readFileSync()` for code structure checks
114
+ - Check: `grep -E "fs\.readFileSync|readFileSync" test-{issue}-*.ts`
115
+ - ❌ BLOCKER if found - tests are doing static analysis, not runtime testing
116
+ - Expected: Tests should execute code, not read source files
117
+ - Reference: `retrospectives/issue-723-weak-test-validation-anti-pattern-postmortem.md`
118
+ - [ ] **Verify Runtime Behavior Testing**: Check tests actually execute code paths
119
+ - Review test code to ensure it calls actual functions/services
120
+ - Verify tests check observable outcomes (API responses, database state, logs)
121
+ - ❌ BLOCKER if tests only check code structure without execution
122
+ - [ ] **Check for Mock Anti-Pattern**: Verify tests don't mock the core functionality being tested
123
+ - Check: Are mocks used for dependencies (✅) or for the thing being tested (❌)?
124
+ - ❌ BLOCKER if core functionality is mocked
125
+ - Expected: Mock dependencies, test actual implementation
126
+ - [ ] **Verify State Validation**: Check tests validate state changes
127
+ - Tests should check database state, API responses, or service interactions
128
+ - ❌ BLOCKER if tests don't validate any state changes
129
+ - Expected: Tests verify before/after state or observable outcomes
130
+
131
+ **Document findings:**
132
+ - Create table: Test File | Static Analysis Check | Runtime Behavior | Mock Usage | State Validation | Status
133
+
134
+ #### Regression Test Verification (For Bug Fixes)
135
+ **If this is a bug fix (check RFC or issue description):**
136
+ - [ ] **Identify Bug**: Understand what the bug was from RFC/issue
137
+ - [ ] **Check for Regression Test**: Verify test file includes regression test for the bug
138
+ - Check test file for test case that reproduces the bug
139
+ - Test should be named something like "regression test for bug X" or similar
140
+ - [ ] **Verify Test Pattern**: Regression test must follow correct pattern
141
+ - ❌ BLOCKER if test passes/fails regardless of bug (wrong pattern)
142
+ - ✅ CORRECT: Test fails with bug → Test passes with fix
143
+ - ❌ WRONG: Test passes with bug AND with fix (doesn't reproduce bug)
144
+ - ❌ WRONG: Test fails with bug AND with fix (environmental issue)
145
+ - [ ] **Verify Test Behavior** (if possible):
146
+ - Ideally: Revert fix temporarily, run test → should fail
147
+ - Reapply fix, run test → should pass
148
+ - If not possible, verify test logic would fail with buggy code
149
+
150
+ **Document findings:**
151
+ - Bug Fix: ✅ Yes / ❌ No
152
+ - Regression Test: ✅ Found / ❌ Missing
153
+ - Test Pattern: ✅ Correct (fails with bug, passes with fix) / ❌ Wrong pattern
154
+ - Notes: [Any issues with regression test]
155
+
156
+ ### Step 4.5: Common Error Pattern Checks
157
+ **🚨 CRITICAL: Check for common test and code quality issues**
158
+
159
+ #### Test Structure Validation
160
+ **For each test file found (test-{issue}-*.ts):**
161
+ - [ ] **BaseTestCase Extension**: Verify test cases extend `BaseTestCase` interface
162
+ - Check: `grep -E "extends BaseTestCase|interface.*extends BaseTestCase" test-{issue}-*.ts`
163
+ - ❌ BLOCKER if test cases don't extend BaseTestCase
164
+ - Expected pattern: `interface MyTestCase extends BaseTestCase { ... }`
165
+ - All test case objects must use interface that extends BaseTestCase
166
+ - [ ] **Main Function**: Verify test file has `main()` function
167
+ - Check: `grep -E "async function main\(\)|function main\(\)" test-{issue}-*.ts`
168
+ - ❌ BLOCKER if main() function missing
169
+ - Expected pattern: `async function main() { ... runTests(...) }`
170
+ - Must have `main().catch(console.error)` at end of file
171
+ - [ ] **runTests Usage**: Verify test file uses `runTests()` function
172
+ - Check: `grep "runTests" test-{issue}-*.ts`
173
+ - ❌ BLOCKER if runTests() not used
174
+ - Expected: `runTests(TEST_CASES, runMyTest, 'Test Suite Name')`
175
+ - Must import runTests from './test-utils' or './test-utils.ts'
176
+
177
+ **Document findings:**
178
+ - Create table: Test File | BaseTestCase | main() | runTests() | Status
179
+
180
+ #### Code Quality Check
181
+ - [ ] **Run Code Quality Check**: Fetch and execute the code quality script ephemerally.
182
+ - Command: `get_fraim_file({ path: "scripts/code-quality-check.sh" })` -> save to temp -> run with argument `pre-pr`
183
+ - Verify exit code is 0 (success)
184
+ - Check for critical failures (❌) vs warnings (⚠️)
185
+ - ❌ BLOCKER if critical checks fail (e.g., `as any` usage, TypeScript errors)
186
+ - ⚠️ WARNING if warnings exist (document in feedback)
187
+ - [ ] **Review Quality Check Output**:
188
+ - Check for `as any` type bypassing in src/
189
+ - Check TypeScript compilation passes
190
+ - Check linter passes (warnings acceptable, errors are blockers)
191
+ - Check all test files referenced in evidence have been executed
192
+
193
+ **Document findings:**
194
+ - Code Quality Check: ✅ Passed / ❌ Failed / ⚠️ Warnings
195
+ - Critical Issues: [list any critical failures]
196
+ - Warnings: [list any warnings]
197
+
198
+ #### Evidence File Location Validation
199
+ - [ ] **Check Evidence Files**: Verify evidence files are ONLY in `docs/evidence/` folder
200
+ - Check: `find . -name "*evidence*.md" -not -path "./docs/evidence/*" -not -path "./.git/*" -not -path "./node_modules/*"`
201
+ - ❌ BLOCKER if evidence files found outside `docs/evidence/`
202
+ - Expected: All evidence files should be in `docs/evidence/{issue}-*.md`
203
+ - [ ] **Check for Evidence in Git**: Verify evidence folder is gitignored
204
+ - Check: `git check-ignore docs/evidence/` should return `docs/evidence/`
205
+ - ⚠️ WARNING if evidence folder not in .gitignore (should be gitignored)
206
+ - [ ] **Check PR for Evidence Files**: Verify no evidence files committed to repo
207
+ - Check git status for any files in docs/evidence/
208
+ - ❌ BLOCKER if evidence files are staged/committed
209
+
210
+ **Document findings:**
211
+ - Evidence Location Check: ✅ All in docs/evidence/ / ❌ Found outside
212
+ - Git Ignore Check: ✅ Ignored / ⚠️ Not ignored
213
+ - Committed Evidence: ✅ None / ❌ Found in commits
214
+
215
+ **If any blockers found:**
216
+ - ❌ BLOCKER: Must fix before approval
217
+ - List specific files and issues
218
+ - Cannot approve until all blockers resolved
219
+
220
+ ### Step 5: Validation Plan Execution
221
+ **For each validation scenario in RFC Validation Plan:**
222
+ - [ ] **Scenario Identified**: Map each scenario from validation plan
223
+ - [ ] **Evidence Found**: Check PR evidence for scenario validation
224
+ - Look in PR comments for evidence
225
+ - Check for screenshots, curl outputs, test results
226
+ - Verify evidence shows actual execution (not just "tested")
227
+ - [ ] **Method Verified**: Ensure validation method matches (UI, API, DB, etc.)
228
+ - If RFC says "API validation", check for curl/Postman results
229
+ - If RFC says "Browser validation", check for screenshots/recordings
230
+ - If RFC says "Database validation", check for DB query results
231
+ - [ ] **Result Verified**: Ensure scenario actually passes (not just "tested")
232
+ - Check evidence shows success, not just attempt
233
+ - Verify no "pending" or "TODO" markers
234
+
235
+ **If validation plan has 10 scenarios but only 7 have evidence:**
236
+ - ❌ BLOCKER: Missing validation evidence
237
+ - Must identify which scenarios are missing (list specific scenario names)
238
+ - Cannot approve until all scenarios validated with evidence
239
+
240
+ **Document findings:**
241
+ - Create table: Scenario | Method | Evidence | Status | Missing Evidence
242
+
243
+ ### Step 6: Architecture Compliance Check
244
+ **Verify implementation matches design:**
245
+ - [ ] Service boundaries match design
246
+ - Check service files match design boundaries
247
+ - Verify no cross-boundary violations
248
+ - [ ] Patterns match design (e.g., spike-first, deterministic separation)
249
+ - Check code follows patterns specified in RFC
250
+ - Verify no anti-patterns introduced
251
+ - [ ] Dependencies match design
252
+ - Check package.json matches design dependencies
253
+ - Verify no unexpected dependencies added
254
+ - [ ] No over-engineering beyond design scope
255
+ - Check code doesn't add features not in RFC
256
+ - Verify minimal implementation (not over-engineered)
257
+
258
+ **Document findings:**
259
+ - List any architecture deviations
260
+ - Note if deviations are acceptable or blockers
261
+
262
+ ### Step 7: Risk Mitigation Verification
263
+ **For each risk in RFC Risks & Mitigations:**
264
+ - [ ] Verify mitigation is implemented
265
+ - Check code implements mitigation strategy
266
+ - Verify mitigation code exists and is not just commented
267
+ - [ ] Check mitigation actually addresses the risk
268
+ - Verify mitigation logic matches risk description
269
+ - Check mitigation is complete (not partial)
270
+ - [ ] Verify observability for risk detection
271
+ - Check logs/alerts exist for risk scenarios
272
+ - Verify monitoring is in place
273
+
274
+ **Document findings:**
275
+ - Create table: Risk | Mitigation Status | Implementation Location | Notes
276
+
277
+ ### Step 8: Evidence Quality Review
278
+ **Verify PR evidence quality:**
279
+ - [ ] All test outputs included (not just "tests pass")
280
+ - Check for actual test command output
281
+ - Verify exit codes shown
282
+ - Check for test case names and results
283
+ - [ ] All validation scenarios documented with results
284
+ - Check each scenario has evidence
285
+ - Verify evidence shows actual execution
286
+ - [ ] No "pending" items that could be validated automatically
287
+ - Flag any "pending" items that could be tested
288
+ - Check for "TODO" or "needs manual validation" markers
289
+ - [ ] Build verification included
290
+ - Check for `npm run build` output
291
+ - Verify build succeeds
292
+ - [ ] Quality gate results included
293
+ - Check for quality gate script output
294
+ - Verify all gates pass (or warnings documented)
295
+
296
+ **Document findings:**
297
+ - List any evidence quality issues
298
+ - Note if evidence is sufficient or needs improvement
299
+
300
+ ### Step 9: Review Decision and Feedback Creation
301
+
302
+ **Check Iteration Count:**
303
+ - If `docs/evidence/{issue}-design-reviewer-feedback.md` exists, check iteration number
304
+ - Maximum 3 iterations allowed
305
+ - If iteration 3 and still issues: ❌ REJECT (max iterations reached)
306
+
307
+ **Options:**
308
+ - ✅ **APPROVE**: All requirements met, tests pass, evidence complete
309
+ - ❌ **REJECT**: Missing requirements, tests fail, or evidence incomplete (after max iterations)
310
+ - ⚠️ **REQUEST CHANGES**: Minor gaps that need addressing (iteration < 3)
311
+
312
+ **If REQUEST CHANGES (iteration < 3):**
313
+ - **Create Feedback Document**: Create `docs/evidence/{issue}-design-reviewer-feedback.md`
314
+ - Use template below for feedback structure
315
+ - List specific blockers with evidence locations
316
+ - Provide actionable feedback for each issue
317
+ - Include iteration number
318
+ - Label issue `status:wip` (remove `status:needs-review`)
319
+ - Add PR comment referencing feedback document
320
+ - Implementation agent must address feedback and update evidence
321
+ - Implementation agent re-submits by marking `status:needs-review` again
322
+
323
+ **If REJECT (max iterations reached):**
324
+ - Create final feedback document with all remaining issues
325
+ - Label issue `status:wip` (remove `status:needs-review`)
326
+ - Add PR comment with final rejection notice
327
+ - Implementation agent must address all issues before re-submission
328
+
329
+ **If APPROVE:**
330
+ - **Create Summary for Feature Spec Review**: Create `docs/evidence/{issue}-design-review-summary.md`
331
+ - Summarize what was reviewed and approved
332
+ - Include key findings and evidence locations
333
+ - Provide context for feature spec review agent
334
+ - Label issue `status:design-review-passed` (remove `status:needs-review`)
335
+ - Add PR comment confirming design spec compliance
336
+ - Issue ready for Feature Spec Review (next workflow)
337
+
338
+ ## FEEDBACK DOCUMENT TEMPLATE
339
+
340
+ Create `docs/evidence/{issue}-design-reviewer-feedback.md`:
341
+
342
+ ```markdown
343
+ # Design Review Feedback - Issue #{issue}
344
+
345
+ ## Iteration Information
346
+ - Iteration Number: {1, 2, or 3}
347
+ - Review Date: {date}
348
+ - Reviewer: Design Spec Review Agent
349
+
350
+ ## Review Summary
351
+ - RFC Location: `docs/rfcs/{issue}-*.md`
352
+ - Evidence Reviewed: `docs/evidence/{issue}-implementation-evidence.md`
353
+ - Overall Status: ⚠️ REQUEST CHANGES
354
+
355
+ ## Issues Found
356
+
357
+ ### Critical Blockers (Must Fix)
358
+ 1. **Issue Title**
359
+ - **Location**: `src/file.ts:line`
360
+ - **Problem**: [Specific problem description]
361
+ - **Expected**: [What RFC requires]
362
+ - **Found**: [What implementation has]
363
+ - **Action Required**: [Specific fix needed]
364
+
365
+ 2. **Issue Title**
366
+ - [Same structure]
367
+
368
+ ### Missing Requirements
369
+ 1. **Requirement Name**
370
+ - **RFC Section**: [Section reference]
371
+ - **What's Missing**: [Specific requirement]
372
+ - **Action Required**: [What needs to be implemented]
373
+
374
+ ### Test Matrix Issues
375
+ 1. **Missing Test**: Test case name
376
+ - **RFC Test Matrix**: [Reference]
377
+ - **Action Required**: Create test file `test-{issue}-*.ts` with test case
378
+
379
+ ### Test Structure Issues (Common Error Patterns)
380
+ 1. **Test Not Extending BaseTestCase**: `test-{issue}-*.ts`
381
+ - **Problem**: Test case interface doesn't extend BaseTestCase
382
+ - **Expected**: `interface MyTestCase extends BaseTestCase { ... }`
383
+ - **Action Required**: Update test case interface to extend BaseTestCase
384
+
385
+ 2. **Missing main() Function**: `test-{issue}-*.ts`
386
+ - **Problem**: Test file doesn't have main() function
387
+ - **Expected**: `async function main() { ... runTests(...) }` with `main().catch(console.error)`
388
+ - **Action Required**: Add main() function that calls runTests()
389
+
390
+ 3. **Not Using runTests()**: `test-{issue}-*.ts`
391
+ - **Problem**: Test file doesn't use runTests() function
392
+ - **Expected**: Import runTests from './test-utils' and call it in main()
393
+ - **Action Required**: Refactor to use runTests() function
394
+
395
+ ### Test Quality Issues (Anti-Patterns)
396
+ 1. **Static Analysis Anti-Pattern**: `test-{issue}-*.ts`
397
+ - **Problem**: Tests use `fs.readFileSync()` to check code structure instead of runtime behavior
398
+ - **Expected**: Tests should execute code and validate outcomes, not read source files
399
+ - **Action Required**: Rewrite tests to validate runtime behavior
400
+ - **Reference**: `retrospectives/issue-723-weak-test-validation-anti-pattern-postmortem.md`
401
+
402
+ 2. **No Runtime Behavior Validation**: `test-{issue}-*.ts`
403
+ - **Problem**: Tests don't execute actual code paths or validate observable outcomes
404
+ - **Expected**: Tests should call functions/services and check results
405
+ - **Action Required**: Add runtime behavior validation to tests
406
+
407
+ 3. **Mocking Core Functionality**: `test-{issue}-*.ts`
408
+ - **Problem**: Tests mock the thing being tested instead of dependencies
409
+ - **Expected**: Mock dependencies, test actual implementation
410
+ - **Action Required**: Refactor mocks to only mock dependencies
411
+
412
+ 4. **No State Validation**: `test-{issue}-*.ts`
413
+ - **Problem**: Tests don't validate state changes (database, API responses, etc.)
414
+ - **Expected**: Tests should verify before/after state or observable outcomes
415
+ - **Action Required**: Add state validation to tests
416
+
417
+ ### Regression Test Issues (For Bug Fixes)
418
+ 1. **Missing Regression Test**: `test-{issue}-*.ts`
419
+ - **Problem**: Bug fix doesn't have regression test
420
+ - **Expected**: Regression test that fails with bug, passes with fix
421
+ - **Action Required**: Add regression test following correct pattern
422
+
423
+ 2. **Wrong Regression Test Pattern**: `test-{issue}-*.ts`
424
+ - **Problem**: Test passes/fails regardless of bug (doesn't reproduce bug)
425
+ - **Expected**: Test fails with bug → Test passes with fix
426
+ - **Action Required**: Fix regression test to follow correct pattern
427
+ - **Reference**: `retrospectives/task-api-calendar-event-id-bug-postmortem.md`
428
+
429
+ ### Code Quality Issues
430
+ 1. **Code Quality Check Failed**: [Specific issue]
431
+ - **Problem**: [What failed - e.g., "as any" usage, TypeScript errors]
432
+ - **Action Required**: Fix code quality issues and re-run check
433
+ - **Command**: Run quality check (fetch `scripts/code-quality-check.sh` via `get_fraim_file`)
434
+
435
+ ### Evidence File Location Issues
436
+ 1. **Evidence File Outside docs/evidence/**: [File path]
437
+ - **Problem**: Evidence file found outside `docs/evidence/` folder
438
+ - **Expected**: All evidence files should be in `docs/evidence/{issue}-*.md`
439
+ - **Action Required**: Move evidence file to `docs/evidence/` folder
440
+
441
+ ### Validation Plan Issues
442
+ 1. **Missing Validation**: Scenario name
443
+ - **RFC Validation Plan**: [Reference]
444
+ - **Action Required**: Execute validation and add evidence
445
+
446
+ ### Evidence Quality Issues
447
+ 1. **Missing Evidence**: [What's missing]
448
+ - **Action Required**: [What evidence needs to be added]
449
+
450
+ ## Action Items for Implementation Agent
451
+ - [ ] Fix critical blocker 1
452
+ - [ ] Fix critical blocker 2
453
+ - [ ] Implement missing requirement X
454
+ - [ ] Add missing test case Y
455
+ - [ ] Execute missing validation Z
456
+ - [ ] Update evidence document with fixes
457
+
458
+ ## Next Steps
459
+ 1. Implementation agent addresses all feedback items
460
+ 2. Implementation agent updates `docs/evidence/{issue}-implementation-evidence.md`
461
+ 3. Implementation agent marks issue `status:needs-review` again
462
+ 4. Design review agent will re-review (iteration {next_number})
463
+ ```
464
+
465
+ ## SUMMARY DOCUMENT TEMPLATE (For Feature Spec Review)
466
+
467
+ Create `docs/evidence/{issue}-design-review-summary.md` when approving:
468
+
469
+ ```markdown
470
+ # Design Review Summary - Issue #{issue}
471
+
472
+ ## Review Outcome
473
+ - Status: ✅ APPROVED
474
+ - Iterations: {number} (1-3)
475
+ - RFC Location: `docs/rfcs/{issue}-*.md`
476
+
477
+ ## What Was Reviewed
478
+ - Technical Requirements: X/Y implemented ✅
479
+ - Test Matrix: All test cases exist and pass ✅
480
+ - Validation Plan: All scenarios validated ✅
481
+ - Architecture: Matches design ✅
482
+ - Risk Mitigations: All implemented ✅
483
+
484
+ ## Key Findings
485
+ - All RFC requirements implemented correctly
486
+ - All tests from test matrix exist and pass
487
+ - All validation scenarios executed with evidence
488
+ - Architecture follows design decisions
489
+ - No over-engineering detected
490
+
491
+ ## Evidence Locations
492
+ - Implementation Evidence: `docs/evidence/{issue}-implementation-evidence.md`
493
+ - Test Results: [locations]
494
+ - Validation Evidence: [locations]
495
+
496
+ ## Notes for Feature Spec Review
497
+ - Technical implementation is complete and correct
498
+ - All technical requirements met
499
+ - Ready for functional/user experience validation
500
+ - No technical blockers for feature spec review
501
+ ```
502
+
503
+ ## REVIEW EVIDENCE TEMPLATE (PR Comment)
504
+
505
+ Add this as a PR comment:
506
+
507
+ ```markdown
508
+ # Design Spec Review - Issue #{issue}
509
+
510
+ ## RFC Compliance Summary
511
+ - RFC Location: `docs/rfcs/{issue}-*.md`
512
+ - Technical Requirements: X/Y implemented ✅/❌
513
+ - Test Matrix Compliance: X/Y test types complete ✅/❌
514
+ - Validation Plan Coverage: X/Y scenarios validated ✅/❌
515
+ - Architecture Compliance: ✅/❌
516
+ - Risk Mitigations: X/Y implemented ✅/❌
517
+
518
+ ## Detailed Findings
519
+
520
+ ### Technical Requirements
521
+ | Requirement | Status | Evidence Location | Notes |
522
+ |------------|--------|-------------------|-------|
523
+ | API endpoint X | ✅ | `src/api/...` | Matches RFC |
524
+ | Schema change Y | ❌ | Missing | RFC requires Z field |
525
+
526
+ ### Test Matrix Validation
527
+ | Test Type | Required | Found | Passing | Evidence | Missing Tests |
528
+ |-----------|----------|-------|---------|----------|---------------|
529
+ | Unit Tests | 5 | 5 | ✅ | `test-{issue}.ts` lines 10-50 | None |
530
+ | Integration | 3 | 2 | ❌ | Partial | Missing test for scenario Z |
531
+ | E2E | 1 | 0 | ❌ | Not found | E2E test not implemented |
532
+
533
+ ### Test Quality Validation
534
+ | Test File | Static Analysis | Runtime Behavior | Mock Usage | State Validation | Status |
535
+ |-----------|----------------|-------------------|------------|------------------|--------|
536
+ | test-{issue}.ts | ✅ No fs.readFileSync | ✅ Executes code | ✅ Mocks dependencies | ✅ Validates state | ✅ Pass |
537
+ | test-{issue}-2.ts | ❌ Uses fs.readFileSync | ❌ No execution | ❌ Mocks core | ❌ No validation | ❌ Fail |
538
+
539
+ ### Regression Test Verification (If Bug Fix)
540
+ - Bug Fix: ✅ Yes / ❌ No
541
+ - Regression Test: ✅ Found / ❌ Missing
542
+ - Test Pattern: ✅ Correct (fails with bug, passes with fix) / ❌ Wrong pattern
543
+ - Notes: [Any issues with regression test]
544
+
545
+ ### Validation Plan Coverage
546
+ | Scenario | Method | Evidence | Status | Missing Evidence |
547
+ |----------|--------|----------|--------|------------------|
548
+ | Create operation | API | PR comment | ✅ | None |
549
+ | Update operation | API | Missing | ❌ | No curl output found |
550
+
551
+ ### Architecture Compliance
552
+ - Service Boundaries: ✅ Match design
553
+ - Patterns: ✅ Follow design patterns
554
+ - Dependencies: ⚠️ Extra dependency X added (not in RFC)
555
+ - Over-engineering: ✅ Minimal implementation
556
+
557
+ ### Risk Mitigations
558
+ | Risk | Mitigation Status | Implementation Location | Notes |
559
+ |------|-------------------|-------------------------|-------|
560
+ | Risk X | ✅ Implemented | `src/...` | Matches design |
561
+ | Risk Y | ❌ Missing | N/A | Not implemented |
562
+
563
+ ### Common Error Pattern Checks
564
+ - Test Structure: ✅ All tests extend BaseTestCase / ❌ Issues found
565
+ - Main Functions: ✅ All tests have main() / ❌ Missing in some tests
566
+ - runTests Usage: ✅ All tests use runTests() / ❌ Not used in some tests
567
+ - Code Quality Check: ✅ Passed / ❌ Failed / ⚠️ Warnings
568
+ - Evidence Location: ✅ All in docs/evidence/ / ❌ Found outside
569
+
570
+ ### Evidence Quality
571
+ - Test Outputs: ✅ Complete (all test results included)
572
+ - Validation Evidence: ⚠️ Partial (missing scenario Y evidence)
573
+ - Build Verification: ✅ Included
574
+ - Quality Gate: ✅ Passed
575
+
576
+ ## Decision
577
+ ✅ APPROVE / ❌ REJECT / ⚠️ REQUEST CHANGES
578
+
579
+ ## Feedback Document
580
+ - Location: `docs/evidence/{issue}-design-reviewer-feedback.md`
581
+ - Iteration: {1, 2, or 3}
582
+
583
+ ## Blockers (if any)
584
+ 1. Missing test case for scenario X (RFC Test Matrix requires it)
585
+ 2. Validation scenario Y not executed (RFC Validation Plan requires it)
586
+ 3. Technical requirement Z not implemented (RFC Technical Details requires it)
587
+
588
+ ## Next Steps (if REQUEST CHANGES)
589
+ - [ ] Implementation agent addresses feedback in `docs/evidence/{issue}-design-reviewer-feedback.md`
590
+ - [ ] Implementation agent updates `docs/evidence/{issue}-implementation-evidence.md`
591
+ - [ ] Implementation agent marks `status:needs-review` for re-review
592
+ - [ ] Design review agent will re-review (max 3 iterations)
593
+ ```
594
+
595
+ ## EXAMPLES
596
+
597
+ ### Good: Complete Review
598
+ ```
599
+ Issue #533: Design Spec Review
600
+ ✅ RFC Compliance: 15/15 requirements implemented
601
+ ✅ Test Matrix: All 8 test cases exist and pass
602
+ ✅ Validation Plan: All 5 scenarios validated
603
+ ✅ Architecture: Matches design
604
+ ✅ Evidence: Complete with test outputs
605
+ Decision: ✅ APPROVE
606
+ ```
607
+
608
+ ### Bad: Incomplete Review
609
+ ```
610
+ Issue #533: Design Spec Review
611
+ ⚠️ RFC Compliance: 12/15 requirements (3 missing)
612
+ ❌ Test Matrix: 5/8 test cases exist (3 missing)
613
+ ⚠️ Validation Plan: 3/5 scenarios validated (2 missing)
614
+ Decision: ❌ REJECT
615
+ Blockers: Missing tests, missing validations
616
+ ```
617
+
618
+ ## INTEGRATION
619
+
620
+ ### Status Label Flow
621
+ ```
622
+ phase:impl + status:needs-review
623
+ ↓ (Design Spec Review)
624
+ status:design-review-passed (or status:wip if rejected)
625
+ ↓ (Feature Spec Review - next workflow)
626
+ status:feature-review-passed
627
+ ```
628
+
629
+ ### After This Review
630
+ - If APPROVE: Issue moves to Feature Spec Review
631
+ - If REJECT: Issue returns to implementation (status:wip)
632
+ - Implementation agent addresses blockers and re-submits