prizmkit 1.1.86 → 1.1.87
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bundled/VERSION.json +3 -3
- package/bundled/dev-pipeline/scripts/generate-bootstrap-prompt.py +462 -4
- package/bundled/dev-pipeline/templates/bootstrap-prompt.md +1 -1
- package/bundled/dev-pipeline/templates/bootstrap-tier1.md +14 -0
- package/bundled/dev-pipeline/templates/bootstrap-tier2.md +18 -3
- package/bundled/dev-pipeline/templates/bootstrap-tier3.md +18 -3
- package/bundled/dev-pipeline/templates/sections/phase-commit-full.md +2 -2
- package/bundled/dev-pipeline/templates/sections/phase-commit.md +2 -2
- package/bundled/dev-pipeline/templates/sections/phase-prizmkit-test.md +260 -0
- package/bundled/dev-pipeline/templates/sections/phase-review-agent.md +5 -1
- package/bundled/dev-pipeline/templates/sections/phase-review-full.md +5 -1
- package/bundled/dev-pipeline/tests/test_generate_bootstrap_prompt.py +429 -0
- package/bundled/dev-pipeline-windows/scripts/generate-bootstrap-prompt.py +462 -4
- package/bundled/dev-pipeline-windows/templates/bootstrap-prompt.md +1 -1
- package/bundled/dev-pipeline-windows/templates/bootstrap-tier1.md +14 -0
- package/bundled/dev-pipeline-windows/templates/bootstrap-tier2.md +18 -3
- package/bundled/dev-pipeline-windows/templates/bootstrap-tier3.md +18 -3
- package/bundled/dev-pipeline-windows/templates/sections/phase-commit-full.md +2 -2
- package/bundled/dev-pipeline-windows/templates/sections/phase-commit.md +2 -2
- package/bundled/dev-pipeline-windows/templates/sections/phase-prizmkit-test.md +186 -0
- package/bundled/dev-pipeline-windows/templates/sections/phase-review-agent.md +5 -1
- package/bundled/dev-pipeline-windows/templates/sections/phase-review-full.md +5 -1
- package/bundled/skills/_metadata.json +1 -1
- package/bundled/skills/prizmkit-code-review/SKILL.md +7 -2
- package/bundled/skills/prizmkit-code-review/references/reviewer-agent-prompt.md +6 -2
- package/bundled/skills/prizmkit-test/SKILL.md +106 -14
- package/bundled/skills/prizmkit-test/references/boundary-coverage-protocol.md +220 -0
- package/bundled/skills/prizmkit-test/references/examples.md +82 -9
- package/bundled/skills/prizmkit-test/references/test-generation-steps.md +68 -12
- package/bundled/skills/prizmkit-test/references/test-report-template.md +73 -9
- package/bundled/skills/prizmkit-test/scripts/validate_boundary_report.py +347 -0
- package/bundled/skills-windows/prizmkit-code-review/SKILL.md +7 -2
- package/bundled/skills-windows/prizmkit-code-review/references/reviewer-agent-prompt.md +6 -2
- package/bundled/skills-windows/prizmkit-test/SKILL.md +106 -14
- package/bundled/skills-windows/prizmkit-test/references/boundary-coverage-protocol.md +220 -0
- package/bundled/skills-windows/prizmkit-test/references/examples.md +82 -9
- package/bundled/skills-windows/prizmkit-test/references/service-boundary-test-catalog.md +4 -4
- package/bundled/skills-windows/prizmkit-test/references/test-generation-steps.md +68 -12
- package/bundled/skills-windows/prizmkit-test/references/test-report-template.md +73 -9
- package/bundled/skills-windows/prizmkit-test/scripts/validate_boundary_report.py +347 -0
- package/package.json +1 -1
|
@@ -0,0 +1,220 @@
|
|
|
1
|
+
# Boundary Coverage Protocol
|
|
2
|
+
|
|
3
|
+
Use this protocol during `/prizmkit-test` Phase 4 and update it during Phase 5. The goal is to prevent a common false positive: treating "there is a test for this interface" as equivalent to "this interface is safely covered." A happy path proves that the interface can work once; boundary tests prove that the interface handles realistic invalid inputs, authorization failures, ownership rules, state transitions, and dependency failures.
|
|
4
|
+
|
|
5
|
+
## Required Boundary Matrix
|
|
6
|
+
|
|
7
|
+
Build this matrix before writing tests and update it after generated tests pass or produce `needs-review` findings. In `scope=this-change`, build the matrix only for in-scope feature interfaces/endpoints; unrelated interfaces are recorded under `Out of Scope Gaps` and do not trigger generated tests.
|
|
8
|
+
|
|
9
|
+
| Module | Interface | Service Type | Happy Path | Request Validation | Auth / Permission / Ownership | Domain Invariants | Collection / Pagination | Date / Time | State Transitions | Dependency Failure | Response Contract | N/A Reasons | Final Status |
|
|
10
|
+
|--------|-----------|--------------|------------|--------------------|-------------------------------|-------------------|-------------------------|-------------|-------------------|-------------------|-------------------|-------------|--------------|
|
|
11
|
+
| auth | POST /auth/login | Auth / Session | covered | generated | N/A — public route | generated | N/A — no collection input | N/A — no date input | covered | skipped — password hasher not injectable | covered | ... | boundary-covered |
|
|
12
|
+
|
|
13
|
+
## Status Values
|
|
14
|
+
|
|
15
|
+
Use these values consistently:
|
|
16
|
+
|
|
17
|
+
- `covered` — an existing test already asserts this behavior.
|
|
18
|
+
- `generated` — this run generated a test and the test passed.
|
|
19
|
+
- `needs-review` — a generated test is valid, but production behavior mismatched the expected contract.
|
|
20
|
+
- `unresolved` — the test could not be made runnable after the allowed fix attempts.
|
|
21
|
+
- `skipped` — not applicable or not practical in this run, with a concrete reason in `N/A Reasons`.
|
|
22
|
+
- `boundary-missing` — applicable boundary exists but no test covers it yet.
|
|
23
|
+
|
|
24
|
+
## Completion Gate
|
|
25
|
+
|
|
26
|
+
An in-scope interface is complete only when:
|
|
27
|
+
|
|
28
|
+
1. It has happy-path coverage when a happy path exists.
|
|
29
|
+
2. Every applicable boundary category is `covered`, `generated`, or `needs-review`.
|
|
30
|
+
3. Every non-applicable category has a concrete reason.
|
|
31
|
+
4. No applicable category remains `boundary-missing`.
|
|
32
|
+
5. The final status is one of:
|
|
33
|
+
- `boundary-covered`
|
|
34
|
+
- `generated-needs-review`
|
|
35
|
+
- `skipped-with-reason`
|
|
36
|
+
|
|
37
|
+
If an interface has only happy-path coverage, set `Final Status` to `boundary-missing` and continue generating applicable boundary tests. Do not call the run complete while any in-scope interface is `boundary-missing` or `happy-path-only`.
|
|
38
|
+
|
|
39
|
+
## How to Decide Whether Existing Tests Count
|
|
40
|
+
|
|
41
|
+
An existing test counts for a boundary category only when it:
|
|
42
|
+
|
|
43
|
+
- targets the same public interface or endpoint,
|
|
44
|
+
- creates the boundary input, state, dependency failure, or authorization context,
|
|
45
|
+
- asserts the expected error code, exception, state change, response shape, or side effect,
|
|
46
|
+
- verifies behavior rather than only checking that the call returned a response.
|
|
47
|
+
|
|
48
|
+
Examples:
|
|
49
|
+
|
|
50
|
+
- `POST /users/create` returns HTTP 200 with non-empty data → happy path only.
|
|
51
|
+
- `POST /users/create` with missing email asserts `code=10001` → request validation boundary.
|
|
52
|
+
- `POST /items/delete` with another user's item asserts forbidden/not-found code → ownership boundary.
|
|
53
|
+
- Service create method with mocked repository failure asserts no partial side effect → dependency failure boundary.
|
|
54
|
+
|
|
55
|
+
## Boundary Categories
|
|
56
|
+
|
|
57
|
+
### Request Validation
|
|
58
|
+
|
|
59
|
+
Use when the interface accepts client input or public parameters.
|
|
60
|
+
|
|
61
|
+
Generate applicable cases for:
|
|
62
|
+
- missing required fields,
|
|
63
|
+
- empty string / empty array / empty object,
|
|
64
|
+
- malformed JSON or wrong content type when the framework exposes that behavior,
|
|
65
|
+
- wrong type,
|
|
66
|
+
- unsupported enum,
|
|
67
|
+
- invalid numeric values: zero, negative, min/max, just above max,
|
|
68
|
+
- invalid string format: URL, email, phone, date, timestamp, UUID/ULID, object key.
|
|
69
|
+
|
|
70
|
+
### Auth / Permission / Ownership
|
|
71
|
+
|
|
72
|
+
Use when the interface depends on identity, roles, tenant, ownership, entitlement, or token state.
|
|
73
|
+
|
|
74
|
+
Generate applicable cases for:
|
|
75
|
+
- missing auth,
|
|
76
|
+
- malformed auth header,
|
|
77
|
+
- invalid/expired/revoked token,
|
|
78
|
+
- authenticated user accessing another user's resource,
|
|
79
|
+
- missing role/permission/entitlement,
|
|
80
|
+
- default-deny when ownership cannot be proven.
|
|
81
|
+
|
|
82
|
+
### Domain Invariants
|
|
83
|
+
|
|
84
|
+
Use when business rules constrain combinations of fields or state.
|
|
85
|
+
|
|
86
|
+
Generate applicable cases for:
|
|
87
|
+
- ratios or totals that must sum to a value,
|
|
88
|
+
- start date before end date,
|
|
89
|
+
- calorie/weight/amount ranges,
|
|
90
|
+
- unsupported lifecycle status,
|
|
91
|
+
- duplicate unique identifiers,
|
|
92
|
+
- device binding requirements,
|
|
93
|
+
- token rotation/reuse semantics,
|
|
94
|
+
- image quality / provider classification rules,
|
|
95
|
+
- product-specific thresholds such as 90%-110% calorie hit boundaries.
|
|
96
|
+
|
|
97
|
+
### Collection / Pagination
|
|
98
|
+
|
|
99
|
+
Use when the interface accepts or returns lists, pages, filters, or cursors.
|
|
100
|
+
|
|
101
|
+
Generate applicable cases for:
|
|
102
|
+
- empty result,
|
|
103
|
+
- one item,
|
|
104
|
+
- many items,
|
|
105
|
+
- duplicate IDs,
|
|
106
|
+
- omitted page / page size defaults,
|
|
107
|
+
- page 0, negative page, max page size, max + 1,
|
|
108
|
+
- invalid filters,
|
|
109
|
+
- nullable filters such as true/false/null,
|
|
110
|
+
- stable ordering or total count when specified.
|
|
111
|
+
|
|
112
|
+
### Date / Time
|
|
113
|
+
|
|
114
|
+
Use when the interface parses, stores, filters, expires, schedules, or compares time.
|
|
115
|
+
|
|
116
|
+
Generate applicable cases for:
|
|
117
|
+
- invalid date / timestamp format,
|
|
118
|
+
- timezone assumptions,
|
|
119
|
+
- inclusive vs exclusive date range boundaries,
|
|
120
|
+
- start after end,
|
|
121
|
+
- exact expiry instant and just before/after expiry,
|
|
122
|
+
- today/yesterday/stale streak or schedule semantics,
|
|
123
|
+
- leap day/month end when relevant.
|
|
124
|
+
|
|
125
|
+
### State Transitions
|
|
126
|
+
|
|
127
|
+
Use when the interface creates, updates, deletes, confirms, unbinds, retries, logs out, refreshes, or changes status.
|
|
128
|
+
|
|
129
|
+
Generate applicable cases for:
|
|
130
|
+
- create then read,
|
|
131
|
+
- update sets expected derived fields,
|
|
132
|
+
- delete excludes from later reads,
|
|
133
|
+
- double delete / unbind idempotency when specified,
|
|
134
|
+
- refresh rotates token and rejects old token,
|
|
135
|
+
- batch operation with empty list vs explicit IDs,
|
|
136
|
+
- retry count below/at/above max,
|
|
137
|
+
- no partial writes when one operation fails.
|
|
138
|
+
|
|
139
|
+
### Dependency Failure
|
|
140
|
+
|
|
141
|
+
Use when collaborators are mockable or the test layer can safely simulate failures.
|
|
142
|
+
|
|
143
|
+
Generate applicable cases for:
|
|
144
|
+
- repository/database error,
|
|
145
|
+
- storage/signing provider failure,
|
|
146
|
+
- SMS/push/email provider failure,
|
|
147
|
+
- LLM/API client error,
|
|
148
|
+
- timeout/network failure,
|
|
149
|
+
- malformed provider response,
|
|
150
|
+
- ID/random/clock failure when injectable.
|
|
151
|
+
|
|
152
|
+
If a dependency cannot be mocked in the current test layer, mark it `skipped` with a reason and consider a lower-level unit test where it can be mocked.
|
|
153
|
+
|
|
154
|
+
### Response Contract
|
|
155
|
+
|
|
156
|
+
Use for API endpoints and public adapters.
|
|
157
|
+
|
|
158
|
+
Generate applicable cases for:
|
|
159
|
+
- unified envelope fields (`code`, `message`, `data`) or framework-specific response format,
|
|
160
|
+
- fixed business error codes,
|
|
161
|
+
- response field names and snake_case/camelCase contract,
|
|
162
|
+
- omitted vs empty data shape,
|
|
163
|
+
- generated IDs and derived fields,
|
|
164
|
+
- side-effect visibility in response.
|
|
165
|
+
|
|
166
|
+
## Endpoint and Interface Mapping
|
|
167
|
+
|
|
168
|
+
Use names, routes, methods, schemas, binding tags, repository calls, and service logic to choose categories. When names and behavior disagree, trust behavior.
|
|
169
|
+
|
|
170
|
+
| Pattern | Likely Categories |
|
|
171
|
+
|---------|-------------------|
|
|
172
|
+
| register / signup / create user | request validation, auth/session, domain invariants, dependency failure, response contract |
|
|
173
|
+
| login | request validation, auth/session, rate limit, token response, dependency failure |
|
|
174
|
+
| refresh / logout | token state, rotation/revocation, idempotency, dependency failure |
|
|
175
|
+
| get / detail | auth/ownership, not found, response contract, dependency failure |
|
|
176
|
+
| list / search / trends | auth, filters, pagination/empty result, date range, response contract, dependency failure |
|
|
177
|
+
| create | request validation, domain invariants, state transition, dependency failure, response contract |
|
|
178
|
+
| update / set goal / confirm | request validation, ownership, partial update semantics, domain invariants, state transition |
|
|
179
|
+
| delete / unbind | request validation, ownership, not found, soft delete, double operation behavior |
|
|
180
|
+
| upload / sign url | request validation, file/storage rules, provider failure, response contract |
|
|
181
|
+
| AI / provider / recognize | request validation, date/time, provider error mapping, quality classifications, async job state |
|
|
182
|
+
| stats / analytics | date range, empty data, boundary thresholds, response shape, dependency failure |
|
|
183
|
+
| achievement / streak | empty state, date continuity, threshold boundaries, state persistence |
|
|
184
|
+
|
|
185
|
+
## N/A Reason Rules
|
|
186
|
+
|
|
187
|
+
A skipped category must explain why it does not apply or why it cannot be exercised at this level.
|
|
188
|
+
|
|
189
|
+
Good reasons:
|
|
190
|
+
- `N/A — public route; no authenticated principal expected`
|
|
191
|
+
- `N/A — endpoint has no collection input or paginated response`
|
|
192
|
+
- `N/A — dependency is not injectable at handler level; covered by service unit test`
|
|
193
|
+
- `N/A — function is pure and has no stateful side effects`
|
|
194
|
+
- `N/A — E2E skipped because project has no UI layer`
|
|
195
|
+
|
|
196
|
+
Bad reasons:
|
|
197
|
+
- `N/A`
|
|
198
|
+
- `not tested`
|
|
199
|
+
- `probably fine`
|
|
200
|
+
- `covered by happy path`
|
|
201
|
+
|
|
202
|
+
## Report Requirements
|
|
203
|
+
|
|
204
|
+
The final report must include:
|
|
205
|
+
|
|
206
|
+
1. The Boundary Matrix.
|
|
207
|
+
2. A Completion Gate summary:
|
|
208
|
+
- interfaces total,
|
|
209
|
+
- boundary-covered count,
|
|
210
|
+
- generated-needs-review count,
|
|
211
|
+
- skipped-with-reason count,
|
|
212
|
+
- boundary-missing count,
|
|
213
|
+
- happy-path-only count.
|
|
214
|
+
3. A clear final status:
|
|
215
|
+
- `All tests passing and boundary coverage complete`, or
|
|
216
|
+
- `Tests passing, but boundary coverage incomplete`, or
|
|
217
|
+
- `Tests failing`, or
|
|
218
|
+
- `Needs human review`.
|
|
219
|
+
|
|
220
|
+
If any interface remains `boundary-missing` or `happy-path-only`, the final report must not say "ready for commit" without qualifying that boundary coverage is incomplete.
|
|
@@ -26,17 +26,26 @@ Scope: Full project (all modules, all specs)
|
|
|
26
26
|
| payment | api/checkout | integration | missing |
|
|
27
27
|
| — | Checkout flow | e2e | missing |
|
|
28
28
|
|
|
29
|
-
**Phase 4
|
|
29
|
+
**Phase 4 — Boundary Matrix** (excerpt):
|
|
30
|
+
| Module | Interface | Service Type | Happy Path | Request Validation | Auth / Permission / Ownership | Domain Invariants | Dependency Failure | Final Status |
|
|
31
|
+
|--------|-----------|--------------|------------|--------------------|-------------------------------|-------------------|-------------------|--------------|
|
|
32
|
+
| auth | login() | Auth / Session | covered | boundary-missing | boundary-missing | skipped | boundary-missing | boundary-missing |
|
|
33
|
+
| payment | processPayment() | Payment / Billing | covered | boundary-missing | boundary-missing | boundary-missing | boundary-missing | boundary-missing |
|
|
30
34
|
|
|
31
|
-
**Phase 5
|
|
35
|
+
**Phase 5**: Generated 5 unit boundary tests (3 passing, 2 assertion failures marked `needs-review`), 1 integration boundary test (passing), 1 E2E test (passing).
|
|
36
|
+
|
|
37
|
+
**Phase 6 — Report Summary**:
|
|
32
38
|
```
|
|
33
39
|
Generated: 5 unit (2 needs-review), 1 integration, 1 E2E
|
|
40
|
+
Boundary gate: failed — 2 interfaces need human review
|
|
34
41
|
Report: .prizmkit/test/2026_05_23_14_30_00_testresult/test-report.md
|
|
35
42
|
Needs human review: processPayment (returns 400 for negative amounts, expected 422), refund (missing authorization header causes 500 instead of 401)
|
|
36
43
|
```
|
|
37
44
|
|
|
38
45
|
## Example 2: Single-Feature Targeted Test
|
|
39
46
|
|
|
47
|
+
This is an interactive single-feature selection. It is different from headless `scope=this-change`, which is driven by `.prizmkit/specs/<FEATURE_SLUG>/` and changed files.
|
|
48
|
+
|
|
40
49
|
**User**: `/prizmkit-test`
|
|
41
50
|
|
|
42
51
|
**Phase 0 — Architecture Detection**:
|
|
@@ -49,22 +58,86 @@ Test framework: Vitest
|
|
|
49
58
|
```
|
|
50
59
|
Scope: Single feature — 042-payment-gateway
|
|
51
60
|
Test files in scope: 2
|
|
61
|
+
```
|
|
52
62
|
|
|
53
63
|
**Phase 2 — Existing Tests**: 2 test files, 8 tests, all passing.
|
|
54
64
|
|
|
55
65
|
**Phase 3 — Gap Analysis** (excerpt):
|
|
56
66
|
| Module | Interface | Type | Status |
|
|
57
67
|
|--------|-----------|------|--------|
|
|
58
|
-
| payment | processPayment() | unit | missing |
|
|
59
|
-
| payment | refund() | unit | missing |
|
|
68
|
+
| payment | processPayment() | unit | boundary-missing |
|
|
69
|
+
| payment | refund() | unit | boundary-missing |
|
|
70
|
+
|
|
71
|
+
**Phase 4 — Boundary Matrix**:
|
|
72
|
+
| Module | Interface | Service Type | Happy Path | Request Validation | Auth / Permission / Ownership | Domain Invariants | State Transitions | Dependency Failure | Final Status |
|
|
73
|
+
|--------|-----------|--------------|------------|--------------------|-------------------------------|-------------------|-------------------|-------------------|--------------|
|
|
74
|
+
| payment | processPayment() | Payment / Billing | covered | boundary-missing | boundary-missing | boundary-missing | generated | boundary-missing | boundary-missing |
|
|
75
|
+
| payment | refund() | Payment / Refund | covered | boundary-missing | boundary-missing | boundary-missing | boundary-missing | skipped — provider sandbox unavailable | boundary-missing |
|
|
60
76
|
|
|
61
|
-
**Phase
|
|
77
|
+
**Phase 5**: Generated 8 unit tests covering amount boundaries, currency mismatch, missing entitlement, duplicate idempotency key, partial refund, full refund, over-refund, and repository failure. All passing.
|
|
62
78
|
|
|
63
|
-
**Phase
|
|
79
|
+
**Phase 6 — Report Summary**:
|
|
64
80
|
```
|
|
65
|
-
Generated:
|
|
81
|
+
Generated: 8 unit tests (all passing)
|
|
82
|
+
Boundary gate: passed
|
|
66
83
|
Report: .prizmkit/test/2026_05_23_15_00_00_testresult/test-report.md
|
|
67
|
-
All tests passing. Ready for commit.
|
|
84
|
+
All tests passing and boundary coverage complete. Ready for commit.
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
## Example 3: Existing Happy Path Is Not Enough
|
|
88
|
+
|
|
89
|
+
**User**: `/prizmkit-test user module`
|
|
90
|
+
|
|
91
|
+
**Existing tests found**:
|
|
92
|
+
```
|
|
93
|
+
TestHandler_UpdateProfile_Success
|
|
94
|
+
TestHandler_GetProfile_Success
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
**Boundary Matrix result**:
|
|
98
|
+
| Module | Interface | Service Type | Happy Path | Request Validation | Auth / Permission / Ownership | Domain Invariants | Dependency Failure | Final Status |
|
|
99
|
+
|--------|-----------|--------------|------------|--------------------|-------------------------------|-------------------|-------------------|--------------|
|
|
100
|
+
| user | POST /user/update-profile | Validation / CRUD | covered | boundary-missing | boundary-missing | boundary-missing | boundary-missing | boundary-missing |
|
|
101
|
+
| user | POST /user/get-profile | CRUD / Auth | covered | N/A — empty request | boundary-missing | N/A — no domain input | boundary-missing | boundary-missing |
|
|
102
|
+
|
|
103
|
+
**Correct action**:
|
|
104
|
+
Generate tests for:
|
|
105
|
+
- missing auth,
|
|
106
|
+
- invalid gender,
|
|
107
|
+
- invalid activity level,
|
|
108
|
+
- implausible height/weight when specified by service contract,
|
|
109
|
+
- user not found,
|
|
110
|
+
- repository failure,
|
|
111
|
+
- response envelope shape.
|
|
112
|
+
|
|
113
|
+
**Incorrect action**:
|
|
114
|
+
Do not report `user` as covered just because both endpoints have success tests.
|
|
115
|
+
|
|
116
|
+
|
|
117
|
+
## Example 3: Feature Pipeline this-change Run
|
|
118
|
+
|
|
119
|
+
**Caller**: `/prizmkit-test scope=this-change artifact_dir=.prizmkit/specs/{{FEATURE_SLUG}}/`
|
|
120
|
+
|
|
121
|
+
**Expected behavior**:
|
|
122
|
+
- Reads `.prizmkit/specs/{{FEATURE_SLUG}}/spec.md`, `plan.md`, and optional `context-snapshot.md`.
|
|
123
|
+
- Builds in-scope files from changed files and feature artifacts.
|
|
124
|
+
- Generates or updates only tests for this feature's changed files, public interfaces, and acceptance criteria.
|
|
125
|
+
- Records unrelated historical gaps under `Out of Scope Gaps`.
|
|
126
|
+
- Does not generate tests for unrelated modules.
|
|
127
|
+
- Does not infer this-change mode from `.prizmkit/bugfix/{{BUG_ID}}/` or `.prizmkit/refactor/{{REFACTOR_ID}}/`.
|
|
128
|
+
|
|
129
|
+
**Report excerpt**:
|
|
130
|
+
```markdown
|
|
131
|
+
## Scope
|
|
132
|
+
- Mode: this-change
|
|
133
|
+
- Artifact Dir: .prizmkit/specs/{{FEATURE_SLUG}}/
|
|
134
|
+
- Generation Policy: in-scope-only
|
|
135
|
+
|
|
136
|
+
## Out of Scope Gaps
|
|
137
|
+
- payment: missing legacy tests, but this feature did not change payment code.
|
|
138
|
+
|
|
139
|
+
## Verdict
|
|
140
|
+
PASS
|
|
68
141
|
```
|
|
69
142
|
|
|
70
|
-
**HANDOFF**: Independent skill — no handoff. User may proceed to `/prizmkit-committer` if
|
|
143
|
+
**HANDOFF**: Independent skill — no handoff. User may proceed to `/prizmkit-committer` only if tests pass and the boundary completion gate passes, or fix issues manually if tests are marked `needs-review`.
|
|
@@ -1,6 +1,13 @@
|
|
|
1
|
-
# Test Generation Steps (Phase
|
|
1
|
+
# Test Generation Steps (Phase 5)
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
## Scope Guard
|
|
4
|
+
|
|
5
|
+
Before generating any test, determine whether the run is `scope=this-change`.
|
|
6
|
+
|
|
7
|
+
For `scope=this-change`, use `generation_policy=in-scope-only` by default. Generate tests only for the in-scope source/test set built from the feature artifact directory and changed files. If a missing test is unrelated to the feature, record it in `Out of Scope Gaps` and do not generate it. Do not infer this-change scope from `.prizmkit/bugfix/` or `.prizmkit/refactor/`.
|
|
8
|
+
|
|
9
|
+
|
|
10
|
+
Generate tests from the Phase 4 Boundary Coverage Matrix in priority order: unit → integration → E2E. After each batch, run immediately.
|
|
4
11
|
|
|
5
12
|
When generated tests fail, distinguish two cases:
|
|
6
13
|
- **Test bug** (syntax error, wrong import, wrong mock, wrong framework API usage) → fix the test and re-run
|
|
@@ -8,9 +15,49 @@ When generated tests fail, distinguish two cases:
|
|
|
8
15
|
|
|
9
16
|
If a test fails repeatedly after 2 fix attempts, skip it and mark as "unresolved" in the report.
|
|
10
17
|
|
|
11
|
-
##
|
|
18
|
+
## 5a. Boundary Matrix First
|
|
19
|
+
|
|
20
|
+
Before writing any new test, create or update the per-interface Boundary Test Matrix from `references/boundary-coverage-protocol.md`. This step matters because a test suite with broad happy-path coverage can still miss the conditions that cause production failures: invalid input, unauthorized access, ownership mistakes, state transitions, dependency errors, and product-specific invariants.
|
|
21
|
+
|
|
22
|
+
For each in-scope interface or endpoint:
|
|
23
|
+
- Identify existing happy-path coverage.
|
|
24
|
+
- Identify existing boundary coverage by reading test assertions, not just test names.
|
|
25
|
+
- Classify applicable categories: request validation, auth/permission/ownership, domain invariants, collection/pagination, date/time, state transitions, dependency failure, and response contract.
|
|
26
|
+
- Mark non-applicable categories as `N/A` with a concrete reason.
|
|
27
|
+
- Mark applicable uncovered categories as `boundary-missing`.
|
|
28
|
+
|
|
29
|
+
Do not generate only happy-path tests. If happy path is missing, generate it, then continue with applicable boundary tests in the same batch.
|
|
30
|
+
|
|
31
|
+
### Existing Test Evaluation
|
|
32
|
+
|
|
33
|
+
An existing test counts as boundary coverage only when it:
|
|
34
|
+
- targets the same interface/endpoint,
|
|
35
|
+
- sends the relevant boundary input or creates the relevant state/dependency failure,
|
|
36
|
+
- asserts the expected business code, exception, response shape, state change, or side effect,
|
|
37
|
+
- verifies behavior beyond "HTTP 200" or "non-empty response".
|
|
12
38
|
|
|
13
|
-
|
|
39
|
+
A test that only checks the route returns successfully is happy-path coverage, not boundary coverage.
|
|
40
|
+
|
|
41
|
+
### API Endpoint Boundary Discovery
|
|
42
|
+
|
|
43
|
+
For API endpoints, derive boundary cases from:
|
|
44
|
+
- OpenAPI request schemas and response schemas,
|
|
45
|
+
- handler binding tags and request parsing behavior,
|
|
46
|
+
- service validation logic,
|
|
47
|
+
- repository/service error handling,
|
|
48
|
+
- Prizm L2 TRAPS and DECISIONS,
|
|
49
|
+
- existing tests and source behavior,
|
|
50
|
+
- product rules in project brief/specs.
|
|
51
|
+
|
|
52
|
+
Map each endpoint to business-specific cases. Examples:
|
|
53
|
+
- `POST /auth/refresh`: missing token, malformed token, reused/rotated token, revoked token, token-version race if testable.
|
|
54
|
+
- `POST /meals/update`: missing ID, missing foods, invalid meal type, invalid timestamp, not found, wrong owner, recomputed totals, confirmed state transition.
|
|
55
|
+
- `POST /stats/trends`: invalid type, invalid period, invalid date range, empty data, each supported trend type, inclusive end date behavior.
|
|
56
|
+
- `POST /upload/sign-url`: unsupported content type, file_count 0/default/max/max+1, scene handling, storage provider failure.
|
|
57
|
+
|
|
58
|
+
## 5b. Unit Tests (always applicable)
|
|
59
|
+
|
|
60
|
+
For each in-scope uncovered interface:
|
|
14
61
|
- Read the source file to understand function signature, branch logic, dependencies, state changes, side effects, and failure paths.
|
|
15
62
|
- Classify the function/class method by service responsibility using names, module path, parameters, dependencies, and related Prizm docs/spec acceptance criteria.
|
|
16
63
|
- Read `references/service-boundary-test-catalog.md` when the unit is service-like or has business-specific inputs — it maps service responsibilities to boundary cases such as auth/session, payment/wallet, repository, pagination, file storage, notification, queue, cache/idempotency, API client, and date/expiration behavior.
|
|
@@ -24,15 +71,16 @@ For each uncovered interface:
|
|
|
24
71
|
- Do NOT test framework internals or third-party library behavior
|
|
25
72
|
- Run tests immediately after generating
|
|
26
73
|
|
|
27
|
-
##
|
|
74
|
+
## 5c. Integration Tests
|
|
28
75
|
|
|
29
|
-
For each module with dependencies:
|
|
30
|
-
- Generate tests for the module's primary interface exercising its real dependencies
|
|
31
|
-
- If API endpoints exist
|
|
32
|
-
- If database operations exist
|
|
33
|
-
-
|
|
76
|
+
For each in-scope module with dependencies:
|
|
77
|
+
- Generate tests for the module's primary interface exercising its real dependencies.
|
|
78
|
+
- If API endpoints exist, generate request/response tests for every applicable boundary category in the Boundary Matrix, not only valid input. Cover auth, missing params, invalid formats/enums, ownership/not-found, state transitions, and response envelope or schema.
|
|
79
|
+
- If database operations exist, generate CRUD tests using the project's existing test database config when available. Cover transactions, duplicate keys, soft-delete visibility, rollback, and query boundary behavior when applicable.
|
|
80
|
+
- If a dependency failure is not practical at integration level, mark it `N/A` with the reason and cover it in a unit test where mocks are available.
|
|
81
|
+
- Run tests immediately after generating.
|
|
34
82
|
|
|
35
|
-
##
|
|
83
|
+
## 5d. E2E Tests (conditional)
|
|
36
84
|
|
|
37
85
|
Preconditions (ALL must be met):
|
|
38
86
|
- Playwright is available (`@playwright/test` or `playwright` in package.json dependencies, or `npx playwright --version` succeeds as fallback)
|
|
@@ -44,10 +92,18 @@ Preconditions (ALL must be met):
|
|
|
44
92
|
If ALL met:
|
|
45
93
|
- **Before starting dev server**: detect whether it's already running: (1) check `package.json` scripts for port flags (`--port`, `-p`, `PORT=`), (2) fall back to framework defaults (3000 for React/Next/Express, 5173 for Vite, 8000 for Django, 5000 for Flask), (3) if still unknown, ask the user. Use the detected port in `lsof -i :<port>` to check. If running, tell user: "Dev server appears to be already running on port {N}. Use this running instance for E2E tests?" If user confirms, use existing instance. If user declines, skip E2E.
|
|
46
94
|
- Start the project dev server (only if not already running). Wait for it to be ready.
|
|
47
|
-
- For each uncovered acceptance criterion, generate a Playwright test script
|
|
95
|
+
- For each in-scope uncovered acceptance criterion, generate a Playwright test script.
|
|
96
|
+
- Include boundary user journeys when acceptance criteria mention validation, permissions, empty states, error messages, or lifecycle transitions.
|
|
48
97
|
- Run generated E2E tests, capture screenshots of failure.
|
|
49
98
|
- **Only stop dev server if you started it.** Never stop a server that was already running.
|
|
50
99
|
|
|
51
100
|
If NOT met:
|
|
52
101
|
- Note what was skipped and why in the report
|
|
53
102
|
- If only the server wasn't startable: still generate E2E script files but mark as "not run — manual execution needed"
|
|
103
|
+
|
|
104
|
+
## 5e. Boundary Completion Check
|
|
105
|
+
|
|
106
|
+
After generating and running tests:
|
|
107
|
+
- Update the Boundary Matrix statuses based on passing tests and `needs-review` findings.
|
|
108
|
+
- If any interface remains `boundary-missing` or happy-path-only, run another targeted generation pass for those categories before final reporting.
|
|
109
|
+
- Stop only when all in-scope interfaces are `boundary-covered`, `generated-needs-review`, or `skipped-with-reason`, or when remaining gaps are explicitly listed as unresolved.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Test Report Output Template
|
|
2
2
|
|
|
3
|
-
Used in Phase
|
|
3
|
+
Used in Phase 6 of `/prizmkit-test`. Create `.prizmkit/test/{YYYY_MM_DD_HH_MM_SS}_testresult/` directory and write `test-report.md`.
|
|
4
4
|
|
|
5
5
|
## test-report.md Format
|
|
6
6
|
|
|
@@ -13,7 +13,19 @@ Used in Phase 5 of `/prizmkit-test`. Create `.prizmkit/test/{YYYY_MM_DD_HH_MM_SS
|
|
|
13
13
|
- Test framework: {name}
|
|
14
14
|
- Mode: {interactive / headless}
|
|
15
15
|
|
|
16
|
-
##
|
|
16
|
+
## Scope
|
|
17
|
+
- Mode: {full-project / module:<name> / feature:<slug> / this-change}
|
|
18
|
+
- Artifact Dir: {none / .prizmkit/specs/<FEATURE_SLUG>/}
|
|
19
|
+
- Diff Base: {ref or none}
|
|
20
|
+
- Changed Files Source: {changed_files / diff_base / plan.md / context-snapshot.md / spec.md / not applicable}
|
|
21
|
+
- Generation Policy: {in-scope-only / full-project}
|
|
22
|
+
- In-Scope Source Files:
|
|
23
|
+
- {...}
|
|
24
|
+
- In-Scope Test Files:
|
|
25
|
+
- {...}
|
|
26
|
+
|
|
27
|
+
|
|
28
|
+
## Existing Tests Run
|
|
17
29
|
- Total: {N} | Passed: {N} | Failed: {N}
|
|
18
30
|
{failed test details or "All passing"}
|
|
19
31
|
{or "No existing tests found — generated first batch"}
|
|
@@ -21,21 +33,57 @@ Used in Phase 5 of `/prizmkit-test`. Create `.prizmkit/test/{YYYY_MM_DD_HH_MM_SS
|
|
|
21
33
|
## Coverage Gaps Found
|
|
22
34
|
| Module | Interface | Type | Status |
|
|
23
35
|
|--------|-----------|------|--------|
|
|
24
|
-
| ... | ... | unit/integration/e2e | covered / generated / needs-review / unresolved / skipped |
|
|
36
|
+
| ... | ... | unit/integration/e2e | covered / generated / needs-review / unresolved / skipped / boundary-missing |
|
|
37
|
+
|
|
38
|
+
## Boundary Matrix
|
|
39
|
+
| Module | Interface | Service Type | Happy Path | Request Validation | Auth / Permission / Ownership | Domain Invariants | Collection / Pagination | Date / Time | State Transitions | Dependency Failure | Response Contract | N/A Reasons | Final Status |
|
|
40
|
+
|--------|-----------|--------------|------------|--------------------|-------------------------------|-------------------|-------------------------|-------------|-------------------|-------------------|-------------------|-------------|--------------|
|
|
41
|
+
| ... | ... | auth/repository/upload/etc. | covered/generated/skipped | covered/generated/needs-review/skipped/boundary-missing | ... | ... | ... | ... | ... | ... | ... | category — reason | boundary-covered / generated-needs-review / skipped-with-reason / boundary-missing |
|
|
42
|
+
|
|
43
|
+
## Boundary Completion Gate
|
|
44
|
+
- Interfaces total: {N}
|
|
45
|
+
- Boundary-covered: {N}
|
|
46
|
+
- Generated-needs-review: {N}
|
|
47
|
+
- Skipped-with-reason: {N}
|
|
48
|
+
- Boundary-missing: {N}
|
|
49
|
+
- Happy-path-only: {N}
|
|
50
|
+
- Completion gate passed: {yes / no}
|
|
25
51
|
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|--------|-----------|--------------|-----------------------------|----------------|
|
|
29
|
-
| ... | ... | auth/payment/repository/etc. | happy path, service-specific, branch paths, side effects, dependency failures | concurrency — no shared mutable state |
|
|
52
|
+
If `Boundary-missing > 0` or `Happy-path-only > 0`, write:
|
|
53
|
+
`Tests may pass, but boundary coverage is incomplete.`
|
|
30
54
|
|
|
31
|
-
##
|
|
55
|
+
## Boundary Coverage Notes
|
|
56
|
+
| Module | Interface | Boundary Categories Covered | Missing Applicable Boundaries | N/A Categories with Reasons |
|
|
57
|
+
|--------|-----------|-----------------------------|-------------------------------|-----------------------------|
|
|
58
|
+
| ... | ... | request validation, ownership, dependency failure | invalid timestamp, repo failure | pagination — endpoint has no list semantics |
|
|
59
|
+
|
|
60
|
+
## Generated / Updated Tests
|
|
32
61
|
- Unit: {N} generated, {N} needs-review, {N} unresolved
|
|
33
62
|
- Integration: {N} generated, {N} needs-review, {N} unresolved
|
|
34
63
|
- E2E: {N} generated, {N} skipped ({reason})
|
|
35
64
|
|
|
65
|
+
## Boundary Validation
|
|
66
|
+
- Validator command: `{command or "not run — reason"}`
|
|
67
|
+
- Validator result: {passed / failed / skipped}
|
|
68
|
+
- Validator output: {summary or path to raw output}
|
|
69
|
+
|
|
70
|
+
## In-Scope Failures
|
|
71
|
+
- {failures caused by this-change scope, or "None"}
|
|
72
|
+
|
|
73
|
+
## Baseline Failures
|
|
74
|
+
- {pre-existing failures, or "None"}
|
|
75
|
+
|
|
76
|
+
## Out of Scope Gaps
|
|
77
|
+
- {unrelated historical gaps; informational only for scope=this-change}
|
|
78
|
+
|
|
79
|
+
## Verdict
|
|
80
|
+
PASS | NEEDS_FIXES | BLOCKED
|
|
81
|
+
|
|
36
82
|
## Final Status
|
|
37
83
|
- All tests passing: {yes / no}
|
|
38
|
-
-
|
|
84
|
+
- Boundary completion gate passed: {yes / no}
|
|
85
|
+
- Ready for commit: {yes / no — include reason}
|
|
86
|
+
- Needs human review: {list of tests/interfaces marked needs-review}
|
|
39
87
|
{remaining failures if any}
|
|
40
88
|
```
|
|
41
89
|
|
|
@@ -45,3 +93,19 @@ Also save alongside `test-report.md`:
|
|
|
45
93
|
- `existing-test-output.txt` — raw output from Phase 2
|
|
46
94
|
- `generated-tests/` — copies of all generated test files
|
|
47
95
|
- `e2e-output/` — E2E logs and failure screenshots (if applicable)
|
|
96
|
+
- `boundary-validation-output.txt` — output from `scripts/validate_boundary_report.py` when run
|
|
97
|
+
|
|
98
|
+
## Final Status Rules
|
|
99
|
+
|
|
100
|
+
Use these rules so the report does not overstate quality:
|
|
101
|
+
|
|
102
|
+
| Tests Passing | Boundary Gate | Needs Review | Ready for Commit |
|
|
103
|
+
|---------------|---------------|--------------|------------------|
|
|
104
|
+
| yes | yes | no | yes |
|
|
105
|
+
| yes | yes | yes | no — human review required |
|
|
106
|
+
| yes | no | any | no — boundary coverage incomplete |
|
|
107
|
+
| no | any | any | no — tests failing |
|
|
108
|
+
|
|
109
|
+
Do not write "All tests passing. Ready for commit." if any interface is `boundary-missing`, `happy-path-only`, `unresolved`, or `needs-review`.
|
|
110
|
+
|
|
111
|
+
Out-of-scope gaps are informational only. They must not trigger generated tests and must not fail a `scope=this-change` run.
|