prizmkit 1.1.139 → 1.1.141

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (74) hide show
  1. package/bundled/VERSION.json +3 -3
  2. package/bundled/dev-pipeline/prizmkit_runtime/checkpoint_state.py +41 -121
  3. package/bundled/dev-pipeline/prizmkit_runtime/runner_models.py +0 -6
  4. package/bundled/dev-pipeline/prizmkit_runtime/runners.py +1 -2
  5. package/bundled/dev-pipeline/prizmkit_runtime/test_result.py +150 -0
  6. package/bundled/dev-pipeline/scripts/generate-bootstrap-prompt.py +21 -804
  7. package/bundled/dev-pipeline/scripts/generate-bugfix-prompt.py +4 -6
  8. package/bundled/dev-pipeline/scripts/generate-recovery-prompt.py +1 -1
  9. package/bundled/dev-pipeline/scripts/generate-refactor-prompt.py +4 -6
  10. package/bundled/dev-pipeline/scripts/prompt_framework.py +56 -55
  11. package/bundled/dev-pipeline/scripts/update-bug-status.py +1 -2
  12. package/bundled/dev-pipeline/scripts/update-checkpoint.py +14 -96
  13. package/bundled/dev-pipeline/scripts/update-feature-status.py +1 -2
  14. package/bundled/dev-pipeline/scripts/update-refactor-status.py +1 -2
  15. package/bundled/dev-pipeline/templates/bootstrap-prompt.md +3 -3
  16. package/bundled/dev-pipeline/templates/bootstrap-tier3.md +7 -18
  17. package/bundled/dev-pipeline/templates/bugfix-bootstrap-prompt.md +6 -6
  18. package/bundled/dev-pipeline/templates/refactor-bootstrap-prompt.md +5 -5
  19. package/bundled/dev-pipeline/templates/sections/bugfix-phase-commit-report.md +1 -1
  20. package/bundled/dev-pipeline/templates/sections/bugfix-phase-diagnose-plan.md +1 -1
  21. package/bundled/dev-pipeline/templates/sections/bugfix-phase-manual-verification.md +1 -1
  22. package/bundled/dev-pipeline/templates/sections/bugfix-phase-test.md +13 -10
  23. package/bundled/dev-pipeline/templates/sections/bugfix-reminders.md +1 -1
  24. package/bundled/dev-pipeline/templates/sections/checkpoint-system.md +14 -9
  25. package/bundled/dev-pipeline/templates/sections/headless-commit-authorization.md +1 -1
  26. package/bundled/dev-pipeline/templates/sections/phase-commit-full.md +1 -1
  27. package/bundled/dev-pipeline/templates/sections/phase-commit.md +1 -1
  28. package/bundled/dev-pipeline/templates/sections/phase-context-snapshot-agent-suffix.md +2 -2
  29. package/bundled/dev-pipeline/templates/sections/phase-context-snapshot-base.md +1 -1
  30. package/bundled/dev-pipeline/templates/sections/phase-context-snapshot-lite-suffix.md +1 -1
  31. package/bundled/dev-pipeline/templates/sections/phase-implement-lite.md +1 -1
  32. package/bundled/dev-pipeline/templates/sections/phase-prizmkit-test.md +14 -26
  33. package/bundled/dev-pipeline/templates/sections/phase-specify-plan-full.md +3 -3
  34. package/bundled/dev-pipeline/templates/sections/refactor-phase-test.md +13 -10
  35. package/bundled/dev-pipeline/templates/sections/refactor-reminders.md +1 -1
  36. package/bundled/dev-pipeline/tests/test_checkpoint_state.py +287 -384
  37. package/bundled/dev-pipeline/tests/test_generate_bootstrap_prompt.py +119 -684
  38. package/bundled/dev-pipeline/tests/test_generate_bugfix_prompt.py +17 -40
  39. package/bundled/dev-pipeline/tests/test_generate_refactor_prompt.py +20 -23
  40. package/bundled/dev-pipeline/tests/test_python_runner_parity.py +29 -32
  41. package/bundled/skills/_metadata.json +6 -6
  42. package/bundled/skills/prizmkit/SKILL.md +8 -13
  43. package/bundled/skills/prizmkit/references/workflow-state-protocol.md +61 -80
  44. package/bundled/skills/prizmkit-code-review/SKILL.md +3 -3
  45. package/bundled/skills/prizmkit-code-review/references/workflow-state-protocol.md +61 -80
  46. package/bundled/skills/prizmkit-committer/SKILL.md +3 -3
  47. package/bundled/skills/prizmkit-committer/references/workflow-state-protocol.md +61 -80
  48. package/bundled/skills/prizmkit-implement/references/workflow-state-protocol.md +61 -80
  49. package/bundled/skills/prizmkit-plan/references/workflow-state-protocol.md +61 -80
  50. package/bundled/skills/prizmkit-retrospective/SKILL.md +2 -2
  51. package/bundled/skills/prizmkit-retrospective/references/workflow-state-protocol.md +61 -80
  52. package/bundled/skills/prizmkit-test/SKILL.md +217 -187
  53. package/bundled/skills/prizmkit-test/references/external-contract-mock-guidance.md +119 -0
  54. package/bundled/skills/prizmkit-test/references/independent-test-review.md +269 -0
  55. package/bundled/skills/prizmkit-test/references/test-coverage-model.md +193 -0
  56. package/bundled/skills/prizmkit-test/references/test-report-template.md +164 -79
  57. package/bundled/skills/prizmkit-workflow/SKILL.md +15 -23
  58. package/bundled/skills/prizmkit-workflow/references/workflow-state-protocol.md +61 -80
  59. package/package.json +1 -1
  60. package/bundled/skills/prizmkit-test/assets/authoritative-records.schema.json +0 -476
  61. package/bundled/skills/prizmkit-test/assets/behavior-risk-matrix.schema.json +0 -116
  62. package/bundled/skills/prizmkit-test/assets/evidence-manifest.schema.json +0 -111
  63. package/bundled/skills/prizmkit-test/assets/evidence-package-template.json +0 -66
  64. package/bundled/skills/prizmkit-test/references/boundary-coverage-protocol.md +0 -104
  65. package/bundled/skills/prizmkit-test/references/contract-mock-protocol.md +0 -65
  66. package/bundled/skills/prizmkit-test/references/evidence-protocol.md +0 -208
  67. package/bundled/skills/prizmkit-test/references/evidence-request-protocol.md +0 -80
  68. package/bundled/skills/prizmkit-test/references/examples.md +0 -100
  69. package/bundled/skills/prizmkit-test/references/service-boundary-test-catalog.md +0 -228
  70. package/bundled/skills/prizmkit-test/references/test-generation-steps.md +0 -118
  71. package/bundled/skills/prizmkit-test/references/trusted-evidence-execution.md +0 -110
  72. package/bundled/skills/prizmkit-test/references/workflow-state-protocol.md +0 -181
  73. package/bundled/skills/prizmkit-test/scripts/build_test_evidence.py +0 -1851
  74. package/bundled/skills/prizmkit-test/scripts/validate_test_evidence.py +0 -1793
@@ -1,100 +0,0 @@
1
- # Test Evidence Examples
2
-
3
- ## 1. Normally Tested Module Change — TEST_PASS
4
-
5
- Input:
6
-
7
- ```text
8
- /prizmkit-test scope=this-change artifact_dir=<change-artifact>
9
- ```
10
-
11
- A library parser changes one accepted input form. Scope discovery selects the complete parser module and two direct consumers. Existing module tests already cover most behaviors.
12
-
13
- The run:
14
-
15
- 1. Classifies `behavior`.
16
- 2. Adds the changed parser behavior plus existing observable parser behavior to the matrix.
17
- 3. Selects focused, affected-module regression, and Regression Ring layers; module-component and contract-integration are N/A with concrete pure-library rationale.
18
- 4. Adds missing malformed-input and consumer-shape tests.
19
- 5. Proves changed tests fail on baseline and pass currently.
20
- 6. Packages all command/environment/output history.
21
- 7. Deterministically validates.
22
-
23
- Handoff:
24
-
25
- ```text
26
- TEST_PASS
27
- Evidence: .prizmkit/test/evidence/<evidence-id>/
28
- Validator: passed
29
- Mocked code-level evidence does not verify a real deployed environment.
30
- ```
31
-
32
- ## 2. Incompletely Tested Legacy Module — Whole-Module Gap Filling
33
-
34
- A queue worker retry change touches a legacy worker with one happy-path test. Changed lines identify the module, but Primary Scope includes parsing, claim, processing, retry, dead-letter, acknowledgment, idempotency, and dependency-failure behavior.
35
-
36
- `TEST_BUILD` fills observable gaps across the module, not unrelated private utility lines. It adds deterministic fake-clock and contract-backed queue tests for retry just below/at/above maximum, duplicate delivery, malformed payload, lock failure, no partial side effect, ack/nack calls, and dead-letter transition. Regression Ring runs the scheduler and producer contract checks. Coverage percentage is recorded only as a diagnostic signal.
37
-
38
- The run cannot pass until every applicable behavior-risk cell and Regression Ring execution is mapped and covered by reliable current-workspace execution.
39
-
40
- ## 3. Contract-Coupled Modules — Contract-Driven Mocks
41
-
42
- An API client and consumer share OpenAPI. The run snapshots the OpenAPI document into `contracts/`, records its source/hash, and generates mock responses for success, schema error, 429, timeout, malformed body, and pagination token boundaries.
43
-
44
- The mock is rejected if it invents an incompatible field or cannot trace behavior to the contract. Contract and consumer executions pass, and the report states `environment_claim=mocked-code-level-only`; it makes no claim about a deployed SaaS environment.
45
-
46
- ## 4. Reliable Business Defect — TEST_FAIL
47
-
48
- Resolved specification and OpenAPI require a negative amount to return a validation error without a write. A generated contract test reliably observes success plus a database-adapter call.
49
-
50
- The test is valid, current and affected-module executions reproduce the failure, and raw output is preserved. `/prizmkit-test` does not change production business behavior. It packages:
51
-
52
- ```text
53
- TEST_FAIL
54
- Reproduced failure: amount-boundary-negative
55
- Evidence: .prizmkit/test/evidence/<evidence-id>/
56
- ```
57
-
58
- ## 5. Unknown Coupling or Conflicting Truth — TEST_BLOCKED
59
-
60
- A dynamically loaded plugin may consume the changed result shape, but its registry is generated remotely and unavailable. The edge can change the verdict, so it remains verdict-capable and unresolved.
61
-
62
- Alternatively, specification and shared schema disagree on nullability with no authoritative resolution. The skill does not freeze current behavior into a characterization test.
63
-
64
- Both produce:
65
-
66
- ```text
67
- TEST_BLOCKED
68
- Blocker: verdict-capable unresolved edge or conflicting truth
69
- Evidence: .prizmkit/test/evidence/<evidence-id>/
70
- ```
71
-
72
- ## 6. Current-State New Behavior
73
-
74
- A newly added CLI command has no pre-change interface, so the skill adds a focused native command test and runs it against the current project implementation. The test passes, its behavior-risk and test-plan mappings are complete, and the runner receipt and raw output are preserved. The evidence claims current code-level conformance; it does not claim that the test failed on a historical implementation.
75
-
76
- ## 7. Interrupted or Budgeted Run
77
-
78
- On recovery, the skill validates stage outputs and target hashes. A changed lockfile invalidates `INFRA_READY` and every downstream stage but preserves earlier execution history immutably. The run resumes there.
79
-
80
- If the budget ends after half a large module's required matrix is tested, progress persists and the result is `TEST_BLOCKED`; no scope is silently truncated.
81
-
82
- ## 8. Tampering
83
-
84
- After packaging, changing `raw/execution-003.stdout`, a contract snapshot, generated test, source patch, or derived report produces a content-hash mismatch. Changing identity inputs without recomputing the canonical evidence ID also fails.
85
-
86
- ```bash
87
- python3 ${SKILL_DIR}/scripts/validate_test_evidence.py \
88
- --evidence-dir .prizmkit/test/evidence/<evidence-id> \
89
- --project-root <target-project-root>
90
- ```
91
-
92
- Result: non-zero with `Test evidence validation: FAIL`. A tampered package cannot retain `TEST_PASS`.
93
-
94
- ## 9. Lightweight Change
95
-
96
- A comments-only diff is deterministically classified and parser/format/generated-output checks pass. Behavior-only stages are marked not applicable with reasons. `TEST_PASS` means the lightweight claim is proven; the report explicitly does not claim full runtime test evidence.
97
-
98
- ## Compatibility Notice
99
-
100
- The testing-domain verdict remains `TEST_PASS | TEST_FAIL | TEST_BLOCKED`. Consumers must use validated authoritative evidence and may use `test-report.md` only as a derived compatibility view. The formal lifecycle does not reinterpret a testing result as a review or commit verdict.
@@ -1,228 +0,0 @@
1
- # Service Behavior-Risk Catalog
2
-
3
- Use this catalog during `CONTRACT_MODEL` and `TEST_BUILD` to discover domain-specific observable risks. It supplies questions, not truth: every expected value must come from the recorded truth precedence and every external dependency double must follow `references/contract-mock-protocol.md`.
4
-
5
- ## How to Use This Catalog
6
-
7
- 1. Classify the affected module by its public behavior, dependencies, state changes, and resolved contracts.
8
- 2. Select every service type that applies; one behavior can combine several types.
9
- 3. Add applicable cases to `behavior-risk-matrix.json` and map them to test/execution IDs.
10
- 4. Derive mock requests/responses from shared machine-readable or locally snapshotted contracts. Never independently invent provider/database/queue behavior.
11
- 5. Record a concrete rationale for every non-applicable risk.
12
- 6. Treat coverage metrics as diagnostic only; complete observable behavior and risk rather than private lines.
13
-
14
- Code-level tests are mock-first and must state that they do not verify a real deployed environment. Production credentials, services, and data are prohibited.
15
-
16
- ## Service Type Detection
17
-
18
- | Signal | Likely Service Type |
19
- |--------|---------------------|
20
- | login, logout, signup, password, session, token, jwt, oauth, refresh | Auth / Session / Token |
21
- | role, permission, policy, access, entitlement, plan, quota, ownership | Authorization / Permission / Entitlement |
22
- | payment, wallet, billing, subscription, invoice, refund, balance, currency | Payment / Wallet / Billing / Refund |
23
- | repository, model, dao, create, read, update, delete, insert, save, remove | CRUD / Repository / Database |
24
- | search, list, page, cursor, filter, sort, query, limit, offset | Search / List / Pagination |
25
- | validate, parse, normalize, transform, schema, sanitize, format | Validation / Parser / Normalizer |
26
- | upload, download, storage, file, blob, s3, image, mime, path | File Upload / Storage |
27
- | notify, email, sms, webhook, template, recipient, provider | Notification / Email / Webhook |
28
- | job, queue, worker, scheduler, cron, retry, backoff, task | Job / Queue / Scheduler |
29
- | cache, ttl, rate limit, throttle, idempotency, dedupe, lock | Cache / Rate Limit / Idempotency |
30
- | client, sdk, request, fetch, provider, api key, endpoint, response | API Client / Third-Party Integration |
31
- | date, time, expiry, expiration, ttl, schedule, window, timezone | Date / Time / Expiration |
32
-
33
- When names and behavior disagree, trust behavior and dependencies over names. For example, `createCheckoutSession()` is payment-related even if it lives in a controller module.
34
-
35
- ## Universal Boundary Matrix
36
-
37
- Apply these categories to every public function/class method before adding service-specific cases:
38
-
39
- - **Valid input**: typical valid input, minimal valid input, and complex valid input when supported.
40
- - **Empty input**: `null` / `undefined` / `None`, empty string, empty array/object, whitespace-only string.
41
- - **Numeric boundaries**: zero, one, minimum allowed value, maximum allowed value, just below/above limits.
42
- - **Collection boundaries**: zero items, one item, many items, duplicate items, first/last element behavior.
43
- - **Type and format errors**: wrong type, malformed string, unsupported enum, missing required parameter.
44
- - **Branch paths**: each `if/else`, `switch/case/default`, ternary side, and loop path with zero/one/many iterations.
45
- - **State and side effects**: before/after state, writes, emitted events, dependency method calls, call count/order/arguments.
46
- - **Dependency failures**: mocked database errors, network timeouts, storage failures, provider errors, clock/randomness behavior.
47
-
48
- ## Service-Specific Boundary Cases
49
-
50
- ### Auth / Session / Token Services
51
-
52
- Generate applicable cases for:
53
-
54
- - Missing identifier, missing password, empty credentials, malformed email/username.
55
- - Identifier normalization: uppercase/lowercase email, leading/trailing whitespace, Unicode usernames when supported.
56
- - User not found, wrong password, unverified account, disabled account, locked account.
57
- - Failed login count just below/at/above lock threshold.
58
- - Successful login creates exactly one session/token with expected expiry and persisted state.
59
- - Token missing, malformed, invalid signature, wrong algorithm, expired, not-yet-valid, revoked.
60
- - Refresh token rotation: old refresh token reuse, missing refresh token, expired refresh token, revoked refresh token.
61
- - Logout idempotency: logout with valid token, already logged-out token, missing token.
62
- - OAuth/callback flows: missing code, missing state, state mismatch, provider error, provider returns incomplete profile.
63
- - Dependency failures: user store error, password hasher error, token signer/verifier error, clock skew.
64
-
65
- ### Authorization / Permission / Entitlement Services
66
-
67
- Generate applicable cases for:
68
-
69
- - Missing principal, anonymous principal, principal without roles/permissions.
70
- - Default-deny behavior when no policy matches.
71
- - Role hierarchy: basic role, elevated role, admin override, deny rule precedence.
72
- - Resource ownership: owner, non-owner, tenant mismatch, organization mismatch.
73
- - Entitlement states: active, expired, canceled, trial, disabled feature, missing plan.
74
- - Quota boundaries: usage below limit, exactly at limit, one over limit, reset window boundary.
75
- - Duplicate or conflicting permissions.
76
- - Policy conditions with missing attributes, malformed attributes, unknown action/resource.
77
- - Dependency failures: policy store error, entitlement lookup error, quota service timeout.
78
-
79
- ### Payment / Wallet / Billing / Refund Services
80
-
81
- Generate applicable cases for:
82
-
83
- - Amount zero, negative amount, minimum billable unit, maximum allowed amount, just above maximum.
84
- - Currency precision: too many decimal places, integer minor units, rounding at half-unit boundaries.
85
- - Balance boundaries: balance exactly equals debit amount, balance one minor unit below amount, zero balance.
86
- - Currency mismatch between wallet, invoice, and payment request.
87
- - Idempotency: missing key, duplicate key with same payload, duplicate key with different payload.
88
- - Transaction states: pending, succeeded, failed, canceled, already paid, already refunded.
89
- - Refund boundaries: zero refund, partial refund, full refund, cumulative refunds equal original amount, cumulative refunds exceed original amount.
90
- - Concurrent debit/refund attempts against the same account or transaction.
91
- - Subscription/billing cycles: trial start/end, expired subscription, canceled subscription, renewal at boundary time.
92
- - Provider failures: timeout, 4xx/5xx, malformed response, duplicate provider event, out-of-order webhook.
93
- - Side effects: ledger entry count, balance mutation, webhook/event emission, provider call arguments.
94
-
95
- ### CRUD / Repository / Database Services
96
-
97
- Generate applicable cases for:
98
-
99
- - Create with missing required fields, nullable fields set to `null`, optional fields omitted.
100
- - Duplicate unique key, generated ID collision, invalid foreign key, unknown enum value.
101
- - Read/update/delete with nonexistent ID, malformed ID, deleted/soft-deleted ID.
102
- - Partial update semantics: omitted field vs `undefined` vs `null`, empty patch object.
103
- - Optimistic locking: expected version, stale version, missing version.
104
- - Soft delete behavior: excluded by default, included only when requested, double delete idempotency.
105
- - Transaction behavior: rollback when one write fails, no partial side effects after failure.
106
- - Database dependency errors: connection failure, constraint violation, timeout.
107
- - Repository side effects: correct query arguments, write count, returned entity shape.
108
-
109
- ### Search / List / Pagination / Sorting Services
110
-
111
- Generate applicable cases for:
112
-
113
- - Empty query, whitespace query, special characters, Unicode query, injection-like strings treated as data.
114
- - No results, one result, many results, duplicate ranking/sort values.
115
- - Page/offset pagination: page 0, page 1, last page, beyond last page, negative page/offset.
116
- - Cursor pagination: missing cursor, malformed cursor, expired cursor, cursor at final item.
117
- - Limit boundaries: omitted limit, limit 0, limit 1, maximum limit, maximum + 1.
118
- - Sorting: default sort, invalid sort field, invalid direction, stable ordering with ties.
119
- - Filters: empty filter set, unknown filter, conflicting filters, inclusive/exclusive date ranges.
120
- - Total count and hasNext/hasPrevious flags at page boundaries.
121
- - Dependency failures: search index unavailable, database query error, stale index response.
122
-
123
- ### Validation / Parser / Normalizer Services
124
-
125
- Generate applicable cases for:
126
-
127
- - Missing input, `null` / `undefined` / `None`, empty string, whitespace-only string.
128
- - Minimum and maximum length; just below/above configured length boundaries.
129
- - Special characters, Unicode, emoji, newline/control characters, normalization-sensitive characters.
130
- - Wrong type, mixed-type arrays, numeric strings vs numbers, booleans represented as strings.
131
- - Unknown fields in strict mode, extra fields in permissive mode, duplicate keys/items.
132
- - Invalid formats: email, URL, date, UUID, JSON, CSV, phone number, currency.
133
- - Normalization idempotency: applying the normalizer twice yields the same result.
134
- - Error shape: path/field name, code, message, and aggregation of multiple errors.
135
- - Parser failures: malformed payload, truncated payload, unsupported version/schema.
136
-
137
- ### File Upload / Storage Services
138
-
139
- Generate applicable cases for:
140
-
141
- - Zero-byte file, one-byte file, just below max size, exactly max size, just above max size.
142
- - MIME/type mismatch between content, extension, and declared header.
143
- - Unsupported file type, missing extension, uppercase extension, compound extension.
144
- - Filename boundaries: empty name, very long name, Unicode name, duplicate name, path traversal characters.
145
- - Storage key collisions and deterministic key generation.
146
- - Checksum/hash mismatch, corrupted stream, interrupted upload, storage write failure.
147
- - Image/media metadata boundaries: dimensions too small/large, unsupported codec, malformed metadata.
148
- - Delete/read behavior for missing object, already-deleted object, permission-denied object.
149
- - Side effects: storage provider call arguments, metadata record creation, cleanup on failure.
150
-
151
- ### Notification / Email / Webhook Services
152
-
153
- Generate applicable cases for:
154
-
155
- - Missing recipient, invalid email/phone/URL, empty recipient list, duplicate recipients.
156
- - Recipient preferences: unsubscribed, disabled channel, quiet hours, blocked user.
157
- - Template variables: missing required variable, extra variable, `null` variable, special characters/HTML escaping.
158
- - Provider selection: unsupported channel, fallback provider, disabled provider.
159
- - Deduplication: duplicate notification key, repeated webhook event, retry with same id.
160
- - Provider responses: success, rate limit, timeout, 4xx/5xx, malformed response, partial multi-recipient failure.
161
- - Webhook security: missing signature, invalid signature, timestamp outside tolerance, replayed event.
162
- - Side effects: event status persisted, retry scheduled or suppressed, provider called with expected payload.
163
-
164
- ### Job / Queue / Scheduler Services
165
-
166
- Generate applicable cases for:
167
-
168
- - Missing payload, empty payload, malformed payload, unsupported job type/version.
169
- - Duplicate job ID, idempotent processing of already-completed job, requeue behavior.
170
- - Retry boundaries: first failure, retry count just below max, at max, over max, poison/dead-letter path.
171
- - Backoff calculation: zero delay, minimum delay, maximum delay, jitter bounds when deterministic/mocked.
172
- - Schedule boundaries: run in past, run now, run in future, invalid cron, daylight-saving transition.
173
- - Concurrency: two workers claim same job, lock acquisition failure, stale lock recovery.
174
- - Cancellation: before start, during processing, after completion.
175
- - Side effects: ack/nack calls, status transitions, retry scheduling, emitted events.
176
-
177
- ### Cache / Rate Limit / Idempotency Services
178
-
179
- Generate applicable cases for:
180
-
181
- - Cache miss, hit, stale hit, expired entry, invalidation, key collision.
182
- - TTL boundaries: zero TTL, negative TTL, one unit TTL, exact expiry instant, just before/after expiry.
183
- - Serialization/deserialization failure, corrupted cached value, unsupported value type.
184
- - Cache stampede: concurrent misses for the same key should not duplicate expensive work when guarded.
185
- - Rate limit boundaries: count below limit, exactly at limit, one above limit, window rollover.
186
- - Missing actor key: anonymous user, missing IP, shared tenant/global scope.
187
- - Idempotency key: missing, malformed, reused with same payload, reused with different payload, expired record.
188
- - Lock behavior: lock unavailable, lock timeout, release on success, release on failure.
189
- - Backing store failures: Redis/cache down, slow response, partial write.
190
-
191
- ### API Client / Third-Party Integration Services
192
-
193
- Generate applicable cases for:
194
-
195
- - Missing base URL, missing API key, malformed configuration, unsupported region/environment.
196
- - Request building: query encoding, header inclusion, body serialization, path parameter escaping.
197
- - Response handling: 200 with valid body, 204/no body, malformed JSON, missing required field, unknown extra field.
198
- - Error responses: 400, 401/403, 404, 409, 429, 500/503; retryable vs non-retryable classification.
199
- - Timeout, network error, DNS/connect failure, aborted request.
200
- - Retry boundaries: zero retries, one retry then success, max retries exhausted, backoff capped.
201
- - Pagination: empty page, last page, next token missing/malformed, duplicate page token.
202
- - Auth refresh/signing: expired access token, refresh failure, invalid signature, clock skew.
203
- - Circuit breaker or fallback behavior when present.
204
-
205
- ### Date / Time / Expiration Services
206
-
207
- Generate applicable cases for:
208
-
209
- - Exact boundary: `now == startsAt`, `now == expiresAt`, one millisecond/second before and after.
210
- - Inclusive vs exclusive interval ends according to the contract.
211
- - Start after end, zero-length interval, negative duration, missing timezone.
212
- - Time zones: UTC vs local time, DST jump forward/back, leap day, month end, year end.
213
- - Clock skew tolerance for tokens, webhooks, scheduled jobs, and signed requests.
214
- - Recurring schedules: first occurrence, last occurrence, skipped occurrence, invalid cron/expression.
215
- - TTL/window calculations: zero TTL, negative TTL, max TTL, rollover between windows.
216
-
217
- ## When Multiple Service Types Apply
218
-
219
- Combine the relevant sections and remove duplicate cases. Prioritize tests that exercise business invariants over tests that only repeat generic invalid input handling. For example, a wallet debit function should cover payment amount boundaries, idempotency replay, and date/expiration behavior if it also accepts an expiring authorization.
220
-
221
- ## N/A Rules
222
-
223
- Mark a category `N/A` when the unit has no corresponding input, state, or dependency. Include a short reason:
224
-
225
- - `N/A — no external provider dependency`
226
- - `N/A — function is pure and has no stateful side effects`
227
- - `N/A — pagination is handled by caller, not this unit`
228
- - `N/A — concurrency safety belongs to integration test because unit has no shared mutable state`
@@ -1,118 +0,0 @@
1
- # Test Generation and Infrastructure Steps
2
-
3
- Use this reference during `INFRA_READY`, `TEST_BUILD`, and `EXECUTE_PROVE`. The authoritative scope and completeness inputs are `scope.json`, `behavior-risk-matrix.json`, and `test-plan.json`.
4
-
5
- ## Existing-Test-First Rule
6
-
7
- Before creating a test, inspect the repository's native test files, runner configuration, and existing behavior assertions. Execute the smallest relevant existing tests first. Add, modify, or delete a repository-native test only when a concrete behavior gap is identified and recorded in the semantic preparation request. Supply project paths and behavior/layer mappings only; `prepare-tests` derives IDs, status, snapshots, and hashes from the repository and canonical Git capture. An unchanged existing test has no generated snapshot, and `generated-tests/` is never an execution workspace.
8
-
9
-
10
- For `scope=this-change`, changed files locate the affected module; they do not restrict tests to changed lines. Fill observable behavior/risk gaps across the whole affected module and execute its Regression Ring. Unrelated modules remain out of scope unless they are direct callers, consumers, shared contracts, or state dependencies.
11
-
12
- Do not infer current-change scope from a directory name alone. Unknown scope is `TEST_BLOCKED`.
13
-
14
- ## Native Infrastructure Discovery
15
-
16
- Before writing tests:
17
-
18
- 1. Inspect manifests, lockfiles, existing test files, runner configuration, scripts, fixtures, and CI commands.
19
- 2. Select the existing adequate runner and naming/import/mock conventions.
20
- 3. Let the target project's semantics determine framework, argv, cwd, timeout, attempts, concurrency, Mock tooling, layers, module roots, inventory patterns; do not import fixed ecosystem defaults from this skill.
21
- 4. If no adequate infrastructure exists, add only the smallest project-native runner/config/dependencies required for the planned layers.
22
- 5. Verify dependency versions against the package registry before writing manifests.
23
- 6. Record every file/dependency/command change and verification in `infrastructure-changes.json`.
24
- 7. Never delete existing tests or add a second framework when the current one is adequate.
25
-
26
- Create fixtures, fakes, contract-backed mock servers, browser harnesses, or isolated services only when required by `test-plan.json`. Every disposable resource needs a unique ID, bounded lifetime, and verified cleanup.
27
-
28
- ## Matrix-First Test Construction
29
-
30
- For every behavior row:
31
-
32
- 1. Read the public interface, contract source, preconditions, inputs/boundaries, outputs, side effects, transitions, and errors.
33
- 2. Inspect existing test assertions—not names—to find proven cells.
34
- 3. Assess all eight risks: functional, boundary, permission, concurrency, idempotency, time, dependency, and consumer.
35
- 4. For every applicable risk, map one or more builder-generated test units and expected execution layer; do not invent a separate AI-maintained ID for every test function.
36
- 5. For every non-applicable risk, record a concrete rationale tied to this behavior.
37
- 6. Keep unresolved truth or untestable necessary behavior explicit; it blocks rather than becoming guessed expected behavior.
38
-
39
- A module is complete when all observable behaviors and applicable risks have mapped, proven tests and the Regression Ring passes. Line/branch/function coverage may reveal an omission and trigger another matrix review, but a percentage cannot complete a cell.
40
-
41
- ## Risk-Adaptive Layers
42
-
43
- Choose only necessary test types, then assess all ordered layers in `test-plan.json`:
44
-
45
- - **Unit/focused:** pure behavior, boundaries, errors, deterministic collaborators, and fast defect localization.
46
- - **Module/component:** public module behavior with internal collaborators composed as production uses them.
47
- - **Contract:** producer/consumer or adapter compatibility against a shared/snapshotted machine-readable contract.
48
- - **Integration:** multiple local modules or isolated infrastructure when mocks cannot establish the relevant property.
49
- - **Code-level E2E:** CLI/UI/API flow through code-level boundaries when user-visible composition is at risk and a native harness is available.
50
- - **Affected-module regression:** every required test for the complete affected module.
51
- - **Regression Ring:** direct callers, consumers, shared contracts, and state dependencies.
52
-
53
- Do not require every layer mechanically. An omitted layer needs a specific reason such as “pure library has no UI or process boundary; module tests prove its public composition.” “Not tested” is not a rationale.
54
-
55
- ## Contract-Driven Doubles
56
-
57
- Read `${SKILL_DIR}/references/contract-mock-protocol.md` for external databases, queues, filesystems, SaaS, browser backends, and cross-module endpoints.
58
-
59
- Prefer shared OpenAPI/schema/types/generated models. Otherwise snapshot a locally derived contract and record derivation source. Generate success, boundary, malformed, timeout, and failure variants from that contract. Reject independently invented fields or a mock that cannot be traced to a snapshot.
60
-
61
- Code-level evidence is mock-first and always reports `environment_claim=mocked-code-level-only`. Do not connect production or a user-linked deployed test database.
62
-
63
- ## Whole-Module Legacy Gap Filling
64
-
65
- When a changed module has incomplete tests:
66
-
67
- - enumerate every observable public behavior, not only changed exports;
68
- - cover valid/minimal/complex inputs and contract-defined boundaries;
69
- - cover outputs and response shape;
70
- - assert side effects and absence of partial effects on failure;
71
- - cover state transitions and idempotency where applicable;
72
- - cover permissions, concurrency, clocks, external dependency failures, and consumer assumptions where applicable;
73
- - avoid tests for unrelated private implementation lines.
74
-
75
- Use `${SKILL_DIR}/references/service-boundary-test-catalog.md` to discover domain-specific risks, then map only contract-supported expectations.
76
-
77
- ## Testability Seams
78
-
79
- If production code cannot be controlled:
80
-
81
- 1. Prefer injecting an existing dependency abstraction.
82
- 2. Otherwise extract a pure function, add an internal adapter, or make state/clock/randomness controllable.
83
- 3. Keep the public contract and observable behavior unchanged.
84
- 4. Record the production patch and behavior-preservation evidence.
85
-
86
- If a public production contract must change, do not proceed autonomously; return `TEST_BLOCKED`.
87
-
88
- ## Valid Failures
89
-
90
- Distinguish:
91
-
92
- - **Test construction defect:** wrong import, syntax, fixture, mock contract, or runner usage. Fix the test/infrastructure within bounded attempts.
93
- - **Reliable implementation/contract failure:** valid test against resolved truth repeatedly fails. Preserve the failure and return `TEST_FAIL`; do not repair business behavior here.
94
- - **Uncertain/flaky/unavailable:** truth unresolved, required test unreliable, infrastructure unavailable, or cleanup fails. Return `TEST_BLOCKED`.
95
-
96
- Never retry until green or silently weaken an assertion/scope.
97
-
98
- ## Current-State Execution Evidence
99
-
100
- Run selected tests in order through the bundled evidence builder: focused, module/component, contract/integration, affected-module regression, Regression Ring. The model writes schema-shaped requests; the builder performs the process and appends immutable runner-generated receipts to `executions.json`.
101
-
102
- For each new/changed necessary behavior test:
103
-
104
- 1. Execute each required layer in order through the builder in the target project root.
105
- 2. Preserve runner-generated receipts and complete stdout/stderr for every selected execution.
106
- 3. Bind each applicable behavior risk to planned tests and selected execution IDs.
107
- 4. Do not create historical checkouts, controlled mutations, or worktrees; current-state conformance is the testing-domain claim.
108
-
109
-
110
- ## Completion Check
111
-
112
- Before packaging:
113
-
114
- - no applicable matrix cell lacks test/execution mapping;
115
- - all required planned layers have reliable selected executions in order;
116
- - affected-module and Regression Ring regressions are complete;
117
- - no verdict-capable unresolved edge remains for pass;
118
- - budgeted/staged work is complete, otherwise `TEST_BLOCKED`.
@@ -1,110 +0,0 @@
1
- # Trusted Evidence Execution
2
-
3
- Load this reference before `SCOPE_DISCOVER`, `EXECUTE_PROVE`, resume, or replay. It defines which choices remain with the model and which evidence mechanics are fixed.
4
-
5
- ## Two Parameter Classes
6
-
7
- ### Model-Chosen Project Parameters
8
-
9
- Inspect the target project and choose these semantically:
10
-
11
- - native test framework, runner, command, flags, and working directory;
12
- - timeout, attempt/retry policy, concurrency, resource limits, and isolation strategy;
13
- - unit/module/contract/integration/code-level E2E layers;
14
- - fixture, fake, Mock, service-double, browser, and database technology;
15
- - module roots, public behavior boundary, inventory patterns, exclusions, and Regression Ring;
16
- - environment classification and endpoint allow/deny evidence.
17
-
18
- The protocol does not prescribe Jest, pytest, a timeout number, a retry count, coverage threshold, runner flag, or mutation package. Record the reason for each choice in request and planning records.
19
-
20
- ### Protocol-Fixed Locators and Mechanisms
21
-
22
- The protocol fixes only what trustworthy, replayable packaging needs:
23
-
24
- - locator arguments: project root, evidence directory, request/manifest/inventory/receipt path;
25
- - schema-shaped request records;
26
- - project-root and evidence-root path confinement;
27
- - actual child-process execution without a shell;
28
- - complete stdout/stderr capture and output hashes;
29
- - complete effective environment and tool-probe capture;
30
- - runner-generated UUID receipt, request hash, and runner hash;
31
- - append-only execution history;
32
- - direct current-workspace native execution with runner receipts;
33
- - deterministic resume invalidation calculation;
34
- - strict schema and cross-record validation.
35
-
36
- ## Trust Boundary
37
-
38
- A receipt proves that the bundled runner produced a record with the captured inputs and outputs during this invocation. The runner snapshots its own executable bytes inside the evidence package so later validation does not silently depend on whichever PrizmKit version happens to be installed. Hashes bind bytes together and detect later drift. They do not create cryptographic non-repudiation and cannot prove that a same-OS-user, same-permission producer did not replace the runner, requests, repository, or evidence package before validation.
39
-
40
- Treat evidence as auditable project-controlled provenance, not a defense against a malicious same-privilege actor. The validator checks consistency, obvious conflicts, and replayability; it cannot mathematically prove that the model found every business behavior.
41
-
42
- ## Builder Commands
43
-
44
- Drive a new package through the canonical lifecycle. Semantic request files remain model-authored; authoritative package records remain tool-owned:
45
-
46
- ```bash
47
- python3 ${SKILL_DIR}/scripts/build_test_evidence.py --project-root <root> --evidence-dir <staging> init --baseline <commit>
48
- python3 ${SKILL_DIR}/scripts/build_test_evidence.py --project-root <root> --evidence-dir <staging> capture-change
49
- python3 ${SKILL_DIR}/scripts/build_test_evidence.py --project-root <root> --evidence-dir <staging> inventory --request requests/inventory.json
50
- python3 ${SKILL_DIR}/scripts/build_test_evidence.py --project-root <root> --evidence-dir <staging> execute --request requests/focused.json
51
- python3 ${SKILL_DIR}/scripts/build_test_evidence.py --project-root <root> --evidence-dir <staging> finalize --request requests/finalize.json
52
- python3 ${SKILL_DIR}/scripts/build_test_evidence.py --project-root <root> --evidence-dir <staging> render-report
53
- ```
54
-
55
- `capture-change` includes tracked modifications, untracked files, deletions, renames, and copies. `finalize` owns the manifest, verdict, stage hashes, evidence identity, and generated-test snapshots. The validator is the only producer of `validation.json`.
56
-
57
-
58
- Write requests beneath the evidence directory, then invoke the bundled script:
59
-
60
- ```bash
61
- python3 ${SKILL_DIR}/scripts/build_test_evidence.py \
62
- --project-root <target-project-root> \
63
- --evidence-dir <evidence-dir> \
64
- inventory --request requests/inventory.json
65
-
66
- python3 ${SKILL_DIR}/scripts/build_test_evidence.py \
67
- --project-root <target-project-root> \
68
- --evidence-dir <evidence-dir> \
69
- execute --request requests/focused.json
70
-
71
- python3 ${SKILL_DIR}/scripts/build_test_evidence.py \
72
- --project-root <target-project-root> \
73
- --evidence-dir <evidence-dir> \
74
- execute --request requests/focused.json
75
-
76
- python3 ${SKILL_DIR}/scripts/build_test_evidence.py \
77
- --project-root <target-project-root> \
78
- --evidence-dir <evidence-dir> \
79
- resume --manifest manifest.json --inventory target-inventory.json
80
- ```
81
-
82
- The subcommands cover fixed mechanics only:
83
-
84
- - `inventory`: expand model-selected project-relative patterns, independently enumerate all actual files under each declared module root, reject files that are neither categorized nor structurally excluded, hash actual files, and bind discovery inputs;
85
- - `execute`: run the declared native command in the current project workspace and append a runner-generated receipt;
86
- - `resume`: compare hashes and stage outputs to compute an invalidation decision while retaining `semantic_review_required=true`.
87
-
88
- ## Replay
89
-
90
- Replay a prior receipt through the runner rather than treating attestation as historical proof:
91
-
92
- ```bash
93
- python3 ${SKILL_DIR}/scripts/build_test_evidence.py \
94
- --project-root <target-project-root> \
95
- --evidence-dir <evidence-dir> \
96
- execute --replay-receipt receipts/<receipt>.json
97
- ```
98
-
99
- The recorded request hash must still match. Replay emits a new receipt linked with `replay_of`; it never overwrites the historical receipt. A replay can still differ because project state, installed tools, clocks, external test services, or environment values changed. Report those differences instead of claiming historical equivalence.
100
-
101
- ## External Target Safety
102
-
103
- Do not block merely because a string contains `production`; names can be misleading. Classify every external target using endpoint/configuration evidence:
104
-
105
- - block `production` and `unknown` external targets;
106
- - permit only model-classified isolated/test/staging targets with explicit allow evidence and no deny evidence;
107
- - preserve the complete evidence used for the decision;
108
- - keep local fakes and in-process contract mocks classified `local` with `external=false`.
109
-
110
- The model owns the semantic classification; the runner enforces the declared production/unknown safety invariant and rejects contradictory allow/deny records.