prizmkit 1.1.120 → 1.1.121
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bundled/VERSION.json +3 -3
- package/bundled/dev-pipeline/prizmkit_runtime/runner_classification.py +23 -25
- package/bundled/dev-pipeline/prizmkit_runtime/runner_models.py +8 -2
- package/bundled/dev-pipeline/prizmkit_runtime/runner_prompts.py +10 -1
- package/bundled/dev-pipeline/prizmkit_runtime/runners.py +43 -29
- package/bundled/dev-pipeline/scripts/continuation.py +51 -6
- package/bundled/dev-pipeline/scripts/update-bug-status.py +57 -14
- package/bundled/dev-pipeline/scripts/update-feature-status.py +54 -5
- package/bundled/dev-pipeline/scripts/update-refactor-status.py +57 -14
- package/bundled/dev-pipeline/templates/bootstrap-tier3.md +2 -2
- package/bundled/dev-pipeline/templates/bugfix-bootstrap-prompt.md +2 -1
- package/bundled/dev-pipeline/templates/refactor-bootstrap-prompt.md +2 -2
- package/bundled/dev-pipeline/templates/sections/bugfix-phase-review.md +3 -3
- package/bundled/dev-pipeline/templates/sections/phase-review-agent.md +3 -3
- package/bundled/dev-pipeline/templates/sections/phase-review-full.md +4 -4
- package/bundled/dev-pipeline/templates/sections/refactor-phase-review.md +3 -3
- package/bundled/dev-pipeline/tests/test_auto_skip.py +5 -5
- package/bundled/dev-pipeline/tests/test_generate_bootstrap_prompt.py +10 -3
- package/bundled/dev-pipeline/tests/test_generate_bugfix_prompt.py +9 -2
- package/bundled/dev-pipeline/tests/test_generate_refactor_prompt.py +9 -1
- package/bundled/dev-pipeline/tests/test_python_runner_parity.py +222 -18
- package/bundled/dev-pipeline/tests/test_unified_cli.py +14 -10
- package/bundled/skills/_metadata.json +4 -4
- package/bundled/skills/prizmkit-code-review/SKILL.md +49 -48
- package/bundled/skills/prizmkit-code-review/references/review-report-template.md +13 -8
- package/bundled/skills/prizmkit-code-review/references/reviewer-agent-prompt.md +46 -32
- package/bundled/skills/prizmkit-code-review/references/reviewer-workspace-protocol.md +90 -0
- package/bundled/skills/prizmkit-test/SKILL.md +159 -155
- package/bundled/skills/prizmkit-test/assets/authoritative-records.schema.json +420 -0
- package/bundled/skills/prizmkit-test/assets/behavior-risk-matrix.schema.json +116 -0
- package/bundled/skills/prizmkit-test/assets/evidence-manifest.schema.json +112 -0
- package/bundled/skills/prizmkit-test/assets/evidence-package-template.json +97 -0
- package/bundled/skills/prizmkit-test/references/boundary-coverage-protocol.md +76 -192
- package/bundled/skills/prizmkit-test/references/contract-mock-protocol.md +65 -0
- package/bundled/skills/prizmkit-test/references/evidence-protocol.md +194 -0
- package/bundled/skills/prizmkit-test/references/evidence-request-protocol.md +78 -0
- package/bundled/skills/prizmkit-test/references/examples.md +65 -108
- package/bundled/skills/prizmkit-test/references/service-boundary-test-catalog.md +10 -7
- package/bundled/skills/prizmkit-test/references/test-generation-steps.md +90 -82
- package/bundled/skills/prizmkit-test/references/test-report-template.md +77 -98
- package/bundled/skills/prizmkit-test/references/trusted-evidence-execution.md +97 -0
- package/bundled/skills/prizmkit-test/scripts/build_test_evidence.py +723 -0
- package/bundled/skills/prizmkit-test/scripts/validate_test_evidence.py +1426 -0
- package/package.json +1 -1
- package/bundled/skills/prizmkit-test/scripts/validate_boundary_report.py +0 -347
|
@@ -0,0 +1,194 @@
|
|
|
1
|
+
# Test Evidence Protocol
|
|
2
|
+
|
|
3
|
+
This reference is normative for behavior-changing `/prizmkit-test` runs. Structured JSON is authoritative; `test-report.md` is a derived view.
|
|
4
|
+
|
|
5
|
+
## Evidence Target and Identity
|
|
6
|
+
|
|
7
|
+
Write packages under `.prizmkit/test/evidence/<evidence-id>/`. Compute `evidence-id` as lowercase SHA-256 of canonical, key-sorted, compact JSON containing exactly:
|
|
8
|
+
|
|
9
|
+
- `baseline_commit`
|
|
10
|
+
- `working_diff_sha256`
|
|
11
|
+
- `scope_sha256`
|
|
12
|
+
|
|
13
|
+
Store those values as `evidence_id_inputs` in `manifest.json`. Record separate aggregate hashes for source, tests, contracts, lockfiles, full environment values, and plan inputs. Hash every evidence file except `manifest.json` in the manifest; do not create a self-referential manifest hash.
|
|
14
|
+
|
|
15
|
+
Hashes and validator receipts provide byte-level binding and drift detection, not cryptographic non-repudiation. A caller with the same OS permissions can replace the runner, requests, project tree, or package before validation. Preserve runner/request hashes and use replay to improve auditability without claiming resistance to a malicious same-privilege producer.
|
|
16
|
+
|
|
17
|
+
`source-change.patch` preserves the complete target diff. Snapshot generated/changed tests under `generated-tests/`, contracts under `contracts/`, and raw stdout/stderr under `raw/`. Never replace old execution/output records.
|
|
18
|
+
|
|
19
|
+
## Ordered State Machine
|
|
20
|
+
|
|
21
|
+
Run exactly these states in order:
|
|
22
|
+
|
|
23
|
+
1. `CHANGE_CLASSIFY`
|
|
24
|
+
2. `SCOPE_DISCOVER`
|
|
25
|
+
3. `CONTRACT_MODEL`
|
|
26
|
+
4. `TEST_PLAN`
|
|
27
|
+
5. `INFRA_READY`
|
|
28
|
+
6. `TEST_BUILD`
|
|
29
|
+
7. `EXECUTE_PROVE`
|
|
30
|
+
8. `EVIDENCE_PACKAGE`
|
|
31
|
+
9. `EVIDENCE_VALIDATE`
|
|
32
|
+
|
|
33
|
+
Each manifest stage records `name`, `status`, `input_sha256`, and output paths. A stage may be `not_applicable` only with a concrete rationale. A later state cannot complete until every applicable predecessor output exists, validates, and matches its recorded input hash.
|
|
34
|
+
|
|
35
|
+
### CHANGE_CLASSIFY
|
|
36
|
+
|
|
37
|
+
Classify `behavior` when executable behavior can change through code, runtime configuration, schemas, migrations, generated runtime assets, lockfiles, or dependency changes. Classify `lightweight` only when the scope is exclusively documentation, comments, or formatting with no generated/runtime behavior effect.
|
|
38
|
+
|
|
39
|
+
For lightweight changes, record deterministic checks such as patch classification, parser/format validation, and generated-output consistency. Mark behavior-only stages not applicable with rationale. `TEST_PASS` then means only that the lightweight claim was proven; never describe it as complete runtime behavior evidence.
|
|
40
|
+
|
|
41
|
+
Unknown or mixed classification enters the full protocol; unresolved classification becomes `TEST_BLOCKED`.
|
|
42
|
+
|
|
43
|
+
### SCOPE_DISCOVER
|
|
44
|
+
|
|
45
|
+
Record:
|
|
46
|
+
|
|
47
|
+
- Affected module: explicit module boundary or cohesion-derived files that jointly express one observable responsibility.
|
|
48
|
+
- Primary Scope: every public behavior of the affected module, not only changed lines.
|
|
49
|
+
- Regression Ring: directly affected callers, consumers, shared contracts, and state dependencies.
|
|
50
|
+
- Unresolved Edges: dynamic loading, reflection, generated clients, runtime registration, remote contracts, or coupling that cannot be proven statically.
|
|
51
|
+
|
|
52
|
+
Record changed files, declared module roots, contract/lockfile/test discovery, evidence-backed exclusions, and Regression Ring mappings in both scope and inventory records. Every changed file must be inventoried or explicitly excluded. The deterministic validator checks these declared relationships and obvious omissions, but inventory cannot prove that the model discovered every semantic behavior or dynamic edge.
|
|
53
|
+
|
|
54
|
+
Every edge records whether it can change the verdict. A verdict-capable unresolved edge prevents `TEST_PASS`.
|
|
55
|
+
|
|
56
|
+
### CONTRACT_MODEL
|
|
57
|
+
|
|
58
|
+
Resolve expected truth in this order:
|
|
59
|
+
|
|
60
|
+
1. specifications;
|
|
61
|
+
2. machine-readable contracts;
|
|
62
|
+
3. acceptance conditions;
|
|
63
|
+
4. existing trusted tests;
|
|
64
|
+
5. callers/consumers;
|
|
65
|
+
6. current implementation.
|
|
66
|
+
|
|
67
|
+
Record conflicts rather than silently characterizing current behavior. If higher-precedence sources conflict and cannot be resolved, issue `TEST_BLOCKED`.
|
|
68
|
+
|
|
69
|
+
Populate `behavior-risk-matrix.json` using `assets/behavior-risk-matrix.schema.json`. Enumerate observable behavior, preconditions, input classes and boundary values, outputs, side effects, state transitions, errors, and every functional, boundary, permission, concurrency, idempotency, time, dependency, and consumer risk. Every risk is applicable, not applicable with concrete rationale, or unresolved.
|
|
70
|
+
|
|
71
|
+
Coverage percentages are diagnostic omission signals. They can prompt another matrix review but cannot prove or replace behavior-risk completeness.
|
|
72
|
+
|
|
73
|
+
### TEST_PLAN
|
|
74
|
+
|
|
75
|
+
Assess all test layers:
|
|
76
|
+
|
|
77
|
+
1. `focused`
|
|
78
|
+
2. `module-component`
|
|
79
|
+
3. `contract-integration`
|
|
80
|
+
4. `affected-module-regression`
|
|
81
|
+
5. `regression-ring`
|
|
82
|
+
|
|
83
|
+
Choose unit, module/component, contract, integration, and code-level E2E tests from actual risk. Do not require every test type mechanically. Every omitted layer needs a concrete not-applicable rationale. If a necessary layer cannot be executed within the configured budget, persist progress and issue `TEST_BLOCKED` until complete.
|
|
84
|
+
|
|
85
|
+
### INFRA_READY
|
|
86
|
+
|
|
87
|
+
Discover manifests, existing tests, commands, runner configuration, fixtures, and conventions before changing infrastructure. Reuse an adequate native runner. Install only necessary dependencies after registry/version verification. Create only required configuration, fixtures, fakes, contract-backed mock servers, or isolated services. Never delete existing tests.
|
|
88
|
+
|
|
89
|
+
Record every infrastructure change, command, verification, and cleanup responsibility in `infrastructure-changes.json`. Production credentials, production databases, production APIs, and destructive real-data operations are prohibited.
|
|
90
|
+
|
|
91
|
+
### TEST_BUILD
|
|
92
|
+
|
|
93
|
+
Fill the whole affected module's observable gaps: public behavior, contracts, boundaries, invariants, failure paths, state transitions, and coupling risks. Do not stop after the changed happy path and do not mechanically exercise unrelated private lines.
|
|
94
|
+
|
|
95
|
+
When code has no controllable seam, the test work may make the minimum behavior-preserving production refactor needed for dependency injection, pure-function extraction, adapters, or controllable state. Record the patch and behavior-preservation check. If a public production contract would need to change, stop with `TEST_BLOCKED`.
|
|
96
|
+
|
|
97
|
+
A valid test that exposes a business/contract defect is evidence for `TEST_FAIL`. Do not repair the business defect inside this skill.
|
|
98
|
+
|
|
99
|
+
### EXECUTE_PROVE
|
|
100
|
+
|
|
101
|
+
Run progressively:
|
|
102
|
+
|
|
103
|
+
1. focused tests;
|
|
104
|
+
2. module/component tests;
|
|
105
|
+
3. contract/integration tests;
|
|
106
|
+
4. complete affected-module regression;
|
|
107
|
+
5. Regression Ring verification.
|
|
108
|
+
|
|
109
|
+
Only omit plan-marked non-required layers. Every N/A is structured with rationale, typed evidence, considered signals, and explanations for detected conflicts. The model retains semantic judgment; validation checks structure and obvious signal conflicts, not hidden intent. Append only runner-generated execution receipts containing:
|
|
110
|
+
|
|
111
|
+
- runner-generated receipt format, runner hash, request path/hash, and receipt-chain predecessor hash;
|
|
112
|
+
- unique selected execution identifier;
|
|
113
|
+
- layer and exact argv;
|
|
114
|
+
- confined working directory;
|
|
115
|
+
- complete effective environment key/value map used by the command;
|
|
116
|
+
- tool/runtime probe results;
|
|
117
|
+
- integer exit code (booleans are invalid);
|
|
118
|
+
- paths and hashes for complete stdout and stderr;
|
|
119
|
+
- reliability/flakiness classification.
|
|
120
|
+
|
|
121
|
+
Do not overwrite failed or superseded attempts. Do not retry until green. Use a bounded documented recovery attempt; required flakiness, unavailable infrastructure, unreliable execution, or cleanup failure is `TEST_BLOCKED`.
|
|
122
|
+
|
|
123
|
+
For each added or changed necessary behavior test, prove the test:
|
|
124
|
+
|
|
125
|
+
1. Prefer an isolated checkout or equivalent uncontaminated environment where the test fails against the pre-change baseline and passes against current code.
|
|
126
|
+
2. If baseline execution is inapplicable (for example, the public behavior did not exist), apply a minimal controlled mutation tied to the same matrix risk, observe failure, restore it completely, and observe current success.
|
|
127
|
+
3. Record `PROVEN`, justified `NOT_APPLICABLE`, or `UNPROVEN`.
|
|
128
|
+
|
|
129
|
+
Record baseline/current/mutation runner receipt IDs, request hash, baseline commit, current and isolation tree hashes, mutation apply/restore hashes, expected failure signal match, and original-tree cleanup result. Necessary `UNPROVEN` behavior prevents `TEST_PASS`.
|
|
130
|
+
|
|
131
|
+
### EVIDENCE_PACKAGE
|
|
132
|
+
|
|
133
|
+
Create the complete package from `assets/evidence-package-template.json`, then hash its records. Set:
|
|
134
|
+
|
|
135
|
+
- `sensitivity=project-controlled`
|
|
136
|
+
- `environment_claim=mocked-code-level-only`
|
|
137
|
+
- `compatibility=legacy-test-report-interface-not-supported`
|
|
138
|
+
|
|
139
|
+
Preserve complete commands, environment values, stdout, stderr, patches, hashes, and history without automatic redaction. The project owns access control, retention, and upload policy. This does not permit production secrets or services.
|
|
140
|
+
|
|
141
|
+
Generate `test-report.md` solely from structured records. Regeneration must produce the same evidence pointers and verdict. The report is not independently editable authority.
|
|
142
|
+
|
|
143
|
+
Prepare `verdict.json` and the derived report, then let the validator finalize the non-circular attestation. Callers must not predeclare validator success as evidence.
|
|
144
|
+
|
|
145
|
+
### EVIDENCE_VALIDATE
|
|
146
|
+
|
|
147
|
+
Run:
|
|
148
|
+
|
|
149
|
+
```bash
|
|
150
|
+
python3 ${SKILL_DIR}/scripts/validate_test_evidence.py \
|
|
151
|
+
--evidence-dir .prizmkit/test/evidence/<evidence-id> \
|
|
152
|
+
--project-root <target-project-root> \
|
|
153
|
+
--attest
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
`--attest` first validates the immutable payload while excluding only the final attestation/view records, atomically writes `validation.json`, refreshes final file hashes, then immediately performs a complete second validation. Subsequent checks omit `--attest` and verify the stored attestation. This avoids trusting a caller-authored `passed` marker or creating a self-referential manifest. The attestation proves package integrity and protocol consistency only; it is not proof against a same-permission producer.
|
|
157
|
+
|
|
158
|
+
When feasible, replay recorded requests through `build_test_evidence.py execute --replay-receipt ...`. Replay performs a new real execution and emits a new linked receipt; it does not pretend the attestation establishes what historically ran.
|
|
159
|
+
|
|
160
|
+
The validator checks all shipped JSON schemas, evidence-directory identity, live target-project inventories and aggregate hashes, changed-file/module-root/exclusion/Regression Ring cross-links, patch/diff binding, stage output ownership and predecessor hashes, generated-test snapshot existence, planned test/inventory/execution linkage, module/matrix completeness, successful selected runner receipts with request/raw-output binding, proof-linked isolated failures/current successes, controlled-mutation apply/restoration hashes, cleanup, structured N/A and obvious signal conflicts, production/unknown external-target safety, strict verdict semantics, and honest mocked-versus-real claims. `TEST_PASS` requires deterministic validator success.
|
|
161
|
+
|
|
162
|
+
## Resume and Invalidation
|
|
163
|
+
|
|
164
|
+
On invocation for an existing evidence target:
|
|
165
|
+
|
|
166
|
+
1. Validate manifest identity and every recorded file hash.
|
|
167
|
+
2. Recompute source, tests, contracts, lockfiles, environment, and plan target hashes.
|
|
168
|
+
3. Find the first stage whose input hash or output is invalid.
|
|
169
|
+
4. Preserve all prior executions and raw outputs immutably.
|
|
170
|
+
5. Invalidate that stage and every downstream stage; do not reuse their verdict claims.
|
|
171
|
+
6. Resume from the first invalid stage and write new execution identifiers.
|
|
172
|
+
|
|
173
|
+
Stages are idempotent with respect to the same canonical inputs. Never silently truncate a large module. Persist matrix progress and return `TEST_BLOCKED` until every necessary behavior and Regression Ring cell is complete.
|
|
174
|
+
|
|
175
|
+
## Terminal Semantics
|
|
176
|
+
|
|
177
|
+
- `TEST_PASS`: all required behavior/risk cells and Regression Ring checks are proven, all required executions are reliable and successful, no verdict-capable unresolved edge remains, cleanup succeeds, code-level requests remain replayable, and deterministic validation passes. This is protocol-validated code-level evidence, not a mathematical proof that no business behavior was omitted.
|
|
178
|
+
- `TEST_FAIL`: valid reliable evidence reproduces an implementation or resolved-contract failure. The package must remain complete enough to prove the failure.
|
|
179
|
+
- `TEST_BLOCKED`: scope/truth uncertainty, unavailable execution, necessary flakiness, incomplete evidence, failed cleanup, budget truncation, unproven behavior, conflicting truth, or deterministic validation failure.
|
|
180
|
+
|
|
181
|
+
No conditional pass exists.
|
|
182
|
+
|
|
183
|
+
## Responsibility Boundary
|
|
184
|
+
|
|
185
|
+
Return only the testing-domain verdict and evidence pointers. Do not:
|
|
186
|
+
|
|
187
|
+
- embed an independent AI reviewer or adversarial PASS judge;
|
|
188
|
+
- decide overall code quality;
|
|
189
|
+
- claim broad Spec compliance beyond tested contracts;
|
|
190
|
+
- repair a business defect exposed by a valid test;
|
|
191
|
+
- authorize commit, release, or deployment;
|
|
192
|
+
- claim mocked code-level evidence validates a deployed test or production environment.
|
|
193
|
+
|
|
194
|
+
The legacy report/verdict interface is intentionally unsupported. Do not migrate Pipeline or other Skill consumers and do not add a compatibility layer in this change.
|
|
@@ -0,0 +1,78 @@
|
|
|
1
|
+
# Evidence Request Protocol
|
|
2
|
+
|
|
3
|
+
Load this reference when writing an inventory, execution, differential, resume, or replay request. Validate every authoritative record against `${SKILL_DIR}/assets/authoritative-records.schema.json`; the validator loads the named `$defs` for each file.
|
|
4
|
+
|
|
5
|
+
## Request Ownership
|
|
6
|
+
|
|
7
|
+
The model writes project-semantic requests after inspecting manifests, runner configuration, contracts, test conventions, changed files, and module boundaries. The bundled builder owns process execution, raw output, receipts, isolation, tree hashes, and invalidation calculations.
|
|
8
|
+
|
|
9
|
+
Never hand-author `executions.json` receipt fields. Never copy a command's claimed output into `raw/`. Invoke the builder so provenance is linked to the actual process.
|
|
10
|
+
|
|
11
|
+
## Inventory Request
|
|
12
|
+
|
|
13
|
+
Place a request under `requests/` with:
|
|
14
|
+
|
|
15
|
+
- `request_version: "1.0"`;
|
|
16
|
+
- `categories`: project-relative glob arrays for source, tests, contracts, and lockfiles;
|
|
17
|
+
- nonempty `changed_files` and declared `module_roots`;
|
|
18
|
+
- structured `exclusions`, each with path, rationale, and discovery evidence;
|
|
19
|
+
- `discovery_evidence` for manifests, runner configuration, filesystem discovery, contracts, lockfiles, and boundary analysis;
|
|
20
|
+
- nonempty `plan_inputs` carrying the semantic planning inputs whose hash controls resume.
|
|
21
|
+
|
|
22
|
+
An empty inventory is not evidence of an empty module. Ensure every changed file exists and is either inventoried or explicitly excluded with evidence. The builder independently enumerates every actual file under each model-declared module root; every enumerated file must appear in one inventory category or a structured exclusion. This does not decide whether the model selected the right semantic module boundary, but it prevents silent omission after that boundary is declared.
|
|
23
|
+
|
|
24
|
+
## Execution Request
|
|
25
|
+
|
|
26
|
+
Use an argv array so the runner does not interpret a shell string. Required fields are:
|
|
27
|
+
|
|
28
|
+
- `request_version`, `purpose`, `command`, `cwd`;
|
|
29
|
+
- environment additions (the runner starts from a minimal runtime environment and the receipt records every effective value);
|
|
30
|
+
- model-selected `tool_version_commands`;
|
|
31
|
+
- selected `layer` and semantic `test_ids`;
|
|
32
|
+
- structured `external_targets`.
|
|
33
|
+
|
|
34
|
+
Optional `timeout_seconds`, `attempt_policy`, and `concurrency` remain model-chosen. The builder records them through request binding but does not invent values or silently retry.
|
|
35
|
+
|
|
36
|
+
The request path must be beneath the evidence directory. `cwd` resolves beneath the selected execution root. Missing commands produce an actual exit 127 receipt; missing cwd or escaped paths block before execution.
|
|
37
|
+
|
|
38
|
+
## Differential Request
|
|
39
|
+
|
|
40
|
+
Record:
|
|
41
|
+
|
|
42
|
+
- behavior ID and model-selected `baseline` or `controlled-mutation` method;
|
|
43
|
+
- the nested execution request;
|
|
44
|
+
- actual baseline commit and current project-tree hash;
|
|
45
|
+
- mutation patch path when applicable;
|
|
46
|
+
- concrete expected failure signals.
|
|
47
|
+
|
|
48
|
+
The builder creates isolated copies, executes the failing and current sides, records receipts, binds the differential request hash, and verifies that the original project tree is unchanged. A proof is `PROVEN` only when the intended failure signal appears, the failing side exits nonzero, current exits zero, and cleanup/tree checks pass.
|
|
49
|
+
|
|
50
|
+
## Structured N/A
|
|
51
|
+
|
|
52
|
+
Use the same structure for omitted layers, not-applicable risk cells, stage N/A, and proof N/A:
|
|
53
|
+
|
|
54
|
+
- `rationale`: behavior-specific reasoning;
|
|
55
|
+
- `evidence`: one or more typed discovery records;
|
|
56
|
+
- `considered_signals`: signals examined before deciding N/A;
|
|
57
|
+
- `conflicts`: every detected contradictory signal plus an explanation.
|
|
58
|
+
|
|
59
|
+
The model retains semantic judgment. The deterministic validator checks shape and obvious signal conflicts only. For example, auth/tenant/role signals conflict with permission N/A; lock/shared-state/worker signals conflict with concurrency N/A; retry/idempotency/dedup signals conflict with idempotency N/A; timeout/TTL/expiry signals conflict with time N/A; clients/contracts/network signals conflict with dependency N/A; callers/consumers/shared-contract mappings conflict with consumer N/A. A conflict can be accepted only when explicitly recorded and explained.
|
|
60
|
+
|
|
61
|
+
## Record-to-Schema Map
|
|
62
|
+
|
|
63
|
+
| Authoritative record | Schema definition |
|
|
64
|
+
|---|---|
|
|
65
|
+
| `change-classification.json` | `changeClassification` |
|
|
66
|
+
| `scope.json` | `scope` |
|
|
67
|
+
| `target-inventory.json` | `targetInventory` |
|
|
68
|
+
| `test-plan.json` | `testPlan` |
|
|
69
|
+
| `requests/*.execution.json` | `executionRequest` |
|
|
70
|
+
| receipt entries in `executions.json` | `executionReceipt` / `executionLog` |
|
|
71
|
+
| `requests/*differential*.json` | `differentialRequest` |
|
|
72
|
+
| `differential-proof.json` | `differentialProof` |
|
|
73
|
+
| `infrastructure-changes.json` | `infrastructureChanges` |
|
|
74
|
+
| `verdict.json` | `verdict` |
|
|
75
|
+
| `validation.json` | `validation` |
|
|
76
|
+
| `lightweight-verification.json` | `lightweightVerification` |
|
|
77
|
+
|
|
78
|
+
`manifest.json` and `behavior-risk-matrix.json` retain their dedicated schemas. All schemas reject unknown fields where the protocol owns the record.
|
|
@@ -1,145 +1,102 @@
|
|
|
1
|
-
# Test Examples
|
|
1
|
+
# Test Evidence Examples
|
|
2
2
|
|
|
3
|
-
##
|
|
3
|
+
## 1. Normally Tested Module Change — TEST_PASS
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
Input:
|
|
6
6
|
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
Detected: Fullstack (React + Express)
|
|
10
|
-
Test framework: Vitest (7 existing test files)
|
|
7
|
+
```text
|
|
8
|
+
/prizmkit-test scope=this-change artifact_dir=<change-artifact>
|
|
11
9
|
```
|
|
12
10
|
|
|
13
|
-
|
|
14
|
-
```
|
|
15
|
-
Scope: Full project (all modules, all specs)
|
|
16
|
-
```
|
|
11
|
+
A library parser changes one accepted input form. Scope discovery selects the complete parser module and two direct consumers. Existing module tests already cover most behaviors.
|
|
17
12
|
|
|
18
|
-
|
|
13
|
+
The run:
|
|
19
14
|
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
| — | Checkout flow | e2e | missing |
|
|
15
|
+
1. Classifies `behavior`.
|
|
16
|
+
2. Adds the changed parser behavior plus existing observable parser behavior to the matrix.
|
|
17
|
+
3. Selects focused, affected-module regression, and Regression Ring layers; module-component and contract-integration are N/A with concrete pure-library rationale.
|
|
18
|
+
4. Adds missing malformed-input and consumer-shape tests.
|
|
19
|
+
5. Proves changed tests fail on baseline and pass currently.
|
|
20
|
+
6. Packages all command/environment/output history.
|
|
21
|
+
7. Deterministically validates.
|
|
28
22
|
|
|
29
|
-
|
|
30
|
-
| Module | Interface | Service Type | Happy Path | Request Validation | Auth / Permission / Ownership | Domain Invariants | Dependency Failure | Final Status |
|
|
31
|
-
|--------|-----------|--------------|------------|--------------------|-------------------------------|-------------------|-------------------|--------------|
|
|
32
|
-
| auth | login() | Auth / Session | covered | boundary-missing | boundary-missing | skipped | boundary-missing | boundary-missing |
|
|
33
|
-
| payment | processPayment() | Payment / Billing | covered | boundary-missing | boundary-missing | boundary-missing | boundary-missing | boundary-missing |
|
|
23
|
+
Handoff:
|
|
34
24
|
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
Boundary gate: failed — 2 interfaces need human review
|
|
41
|
-
Report: .prizmkit/test/2026_05_23_14_30_00_testresult/test-report.md
|
|
42
|
-
Needs human review: processPayment (returns 400 for negative amounts, expected 422), refund (missing authorization header causes 500 instead of 401)
|
|
25
|
+
```text
|
|
26
|
+
TEST_PASS
|
|
27
|
+
Evidence: .prizmkit/test/evidence/<evidence-id>/
|
|
28
|
+
Validator: passed
|
|
29
|
+
Mocked code-level evidence does not verify a real deployed environment.
|
|
43
30
|
```
|
|
44
31
|
|
|
45
|
-
##
|
|
32
|
+
## 2. Incompletely Tested Legacy Module — Whole-Module Gap Filling
|
|
46
33
|
|
|
47
|
-
|
|
34
|
+
A queue worker retry change touches a legacy worker with one happy-path test. Changed lines identify the module, but Primary Scope includes parsing, claim, processing, retry, dead-letter, acknowledgment, idempotency, and dependency-failure behavior.
|
|
48
35
|
|
|
49
|
-
|
|
36
|
+
`TEST_BUILD` fills observable gaps across the module, not unrelated private utility lines. It adds deterministic fake-clock and contract-backed queue tests for retry just below/at/above maximum, duplicate delivery, malformed payload, lock failure, no partial side effect, ack/nack calls, and dead-letter transition. Regression Ring runs the scheduler and producer contract checks. Coverage percentage is recorded only as a diagnostic signal.
|
|
50
37
|
|
|
51
|
-
|
|
52
|
-
```
|
|
53
|
-
Detected: Backend (Express)
|
|
54
|
-
Test framework: Vitest
|
|
55
|
-
```
|
|
38
|
+
The run cannot pass until every applicable behavior-risk cell and Regression Ring execution is mapped and proven.
|
|
56
39
|
|
|
57
|
-
|
|
58
|
-
```
|
|
59
|
-
Scope: Single feature — 042-payment-gateway
|
|
60
|
-
Test files in scope: 2
|
|
61
|
-
```
|
|
40
|
+
## 3. Contract-Coupled Modules — Contract-Driven Mocks
|
|
62
41
|
|
|
63
|
-
|
|
42
|
+
An API client and consumer share OpenAPI. The run snapshots the OpenAPI document into `contracts/`, records its source/hash, and generates mock responses for success, schema error, 429, timeout, malformed body, and pagination token boundaries.
|
|
64
43
|
|
|
65
|
-
|
|
66
|
-
| Module | Interface | Type | Status |
|
|
67
|
-
|--------|-----------|------|--------|
|
|
68
|
-
| payment | processPayment() | unit | boundary-missing |
|
|
69
|
-
| payment | refund() | unit | boundary-missing |
|
|
44
|
+
The mock is rejected if it invents an incompatible field or cannot trace behavior to the contract. Contract and consumer executions pass, and the report states `environment_claim=mocked-code-level-only`; it makes no claim about a deployed SaaS environment.
|
|
70
45
|
|
|
71
|
-
|
|
72
|
-
| Module | Interface | Service Type | Happy Path | Request Validation | Auth / Permission / Ownership | Domain Invariants | State Transitions | Dependency Failure | Final Status |
|
|
73
|
-
|--------|-----------|--------------|------------|--------------------|-------------------------------|-------------------|-------------------|-------------------|--------------|
|
|
74
|
-
| payment | processPayment() | Payment / Billing | covered | boundary-missing | boundary-missing | boundary-missing | generated | boundary-missing | boundary-missing |
|
|
75
|
-
| payment | refund() | Payment / Refund | covered | boundary-missing | boundary-missing | boundary-missing | boundary-missing | skipped — provider sandbox unavailable | boundary-missing |
|
|
46
|
+
## 4. Reliable Business Defect — TEST_FAIL
|
|
76
47
|
|
|
77
|
-
|
|
48
|
+
Resolved specification and OpenAPI require a negative amount to return a validation error without a write. A generated contract test reliably observes success plus a database-adapter call.
|
|
78
49
|
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
50
|
+
The test is valid, current and affected-module executions reproduce the failure, and raw output is preserved. `/prizmkit-test` does not change production business behavior. It packages:
|
|
51
|
+
|
|
52
|
+
```text
|
|
53
|
+
TEST_FAIL
|
|
54
|
+
Reproduced failure: amount-boundary-negative
|
|
55
|
+
Evidence: .prizmkit/test/evidence/<evidence-id>/
|
|
85
56
|
```
|
|
86
57
|
|
|
87
|
-
##
|
|
58
|
+
## 5. Unknown Coupling or Conflicting Truth — TEST_BLOCKED
|
|
88
59
|
|
|
89
|
-
|
|
60
|
+
A dynamically loaded plugin may consume the changed result shape, but its registry is generated remotely and unavailable. The edge can change the verdict, so it remains verdict-capable and unresolved.
|
|
90
61
|
|
|
91
|
-
|
|
92
|
-
```
|
|
93
|
-
TestHandler_UpdateProfile_Success
|
|
94
|
-
TestHandler_GetProfile_Success
|
|
95
|
-
```
|
|
62
|
+
Alternatively, specification and shared schema disagree on nullability with no authoritative resolution. The skill does not freeze current behavior into a characterization test.
|
|
96
63
|
|
|
97
|
-
|
|
98
|
-
| Module | Interface | Service Type | Happy Path | Request Validation | Auth / Permission / Ownership | Domain Invariants | Dependency Failure | Final Status |
|
|
99
|
-
|--------|-----------|--------------|------------|--------------------|-------------------------------|-------------------|-------------------|--------------|
|
|
100
|
-
| user | POST /user/update-profile | Validation / CRUD | covered | boundary-missing | boundary-missing | boundary-missing | boundary-missing | boundary-missing |
|
|
101
|
-
| user | POST /user/get-profile | CRUD / Auth | covered | N/A — empty request | boundary-missing | N/A — no domain input | boundary-missing | boundary-missing |
|
|
64
|
+
Both produce:
|
|
102
65
|
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
-
|
|
106
|
-
-
|
|
107
|
-
|
|
108
|
-
- implausible height/weight when specified by service contract,
|
|
109
|
-
- user not found,
|
|
110
|
-
- repository failure,
|
|
111
|
-
- response envelope shape.
|
|
66
|
+
```text
|
|
67
|
+
TEST_BLOCKED
|
|
68
|
+
Blocker: verdict-capable unresolved edge or conflicting truth
|
|
69
|
+
Evidence: .prizmkit/test/evidence/<evidence-id>/
|
|
70
|
+
```
|
|
112
71
|
|
|
113
|
-
|
|
114
|
-
Do not report `user` as covered just because both endpoints have success tests.
|
|
72
|
+
## 6. Baseline Inapplicable — Controlled Mutation
|
|
115
73
|
|
|
74
|
+
A newly added CLI command has no pre-change interface, so baseline execution is justified `NOT_APPLICABLE` as a direct proof method. In an uncontaminated checkout, the run applies a minimal mutation that removes the command registration, observes the command test fail for that risk, restores the patch, verifies source hashes and cleanup, then observes current success. The behavior is `PROVEN` by `controlled-mutation`.
|
|
116
75
|
|
|
117
|
-
|
|
76
|
+
If restoration cannot be proven, the verdict is `TEST_BLOCKED`.
|
|
118
77
|
|
|
119
|
-
|
|
78
|
+
## 7. Interrupted or Budgeted Run
|
|
120
79
|
|
|
121
|
-
|
|
80
|
+
On recovery, the skill validates stage outputs and target hashes. A changed lockfile invalidates `INFRA_READY` and every downstream stage but preserves earlier execution history immutably. The run resumes there.
|
|
122
81
|
|
|
123
|
-
|
|
124
|
-
- Reads the change artifact's `spec.md`, `plan.md`, and optional `context-snapshot.md`.
|
|
125
|
-
- Builds in-scope files from changed files and the change artifact.
|
|
126
|
-
- Generates or updates only tests for this change's files, public interfaces, and acceptance criteria.
|
|
127
|
-
- Records unrelated historical gaps under `Out of Scope Gaps`.
|
|
128
|
-
- Does not generate tests for unrelated modules.
|
|
129
|
-
- The change artifact may represent a feature, bug fix, refactor, migration, or other scoped change.
|
|
82
|
+
If the budget ends after half a large module's required matrix is tested, progress persists and the result is `TEST_BLOCKED`; no scope is silently truncated.
|
|
130
83
|
|
|
131
|
-
|
|
132
|
-
```markdown
|
|
133
|
-
## Scope
|
|
134
|
-
- Mode: this-change
|
|
135
|
-
- Artifact Dir: .prizmkit/specs/{{CHANGE_SLUG}}/
|
|
136
|
-
- Generation Policy: in-scope-only
|
|
84
|
+
## 8. Tampering
|
|
137
85
|
|
|
138
|
-
|
|
139
|
-
- payment: missing legacy tests, but this change did not affect payment code.
|
|
86
|
+
After packaging, changing `raw/execution-003.stdout`, a contract snapshot, generated test, source patch, or derived report produces a content-hash mismatch. Changing identity inputs without recomputing the canonical evidence ID also fails.
|
|
140
87
|
|
|
141
|
-
|
|
142
|
-
|
|
88
|
+
```bash
|
|
89
|
+
python3 ${SKILL_DIR}/scripts/validate_test_evidence.py \
|
|
90
|
+
--evidence-dir .prizmkit/test/evidence/<evidence-id> \
|
|
91
|
+
--project-root <target-project-root>
|
|
143
92
|
```
|
|
144
93
|
|
|
145
|
-
|
|
94
|
+
Result: non-zero with `Test evidence validation: FAIL`. A tampered package cannot retain `TEST_PASS`.
|
|
95
|
+
|
|
96
|
+
## 9. Lightweight Change
|
|
97
|
+
|
|
98
|
+
A comments-only diff is deterministically classified and parser/format/generated-output checks pass. Behavior-only stages are marked not applicable with reasons. `TEST_PASS` means the lightweight claim is proven; the report explicitly does not claim full runtime test evidence.
|
|
99
|
+
|
|
100
|
+
## Compatibility Notice
|
|
101
|
+
|
|
102
|
+
The legacy `PASS | NEEDS_FIXES | BLOCKED` Markdown report interface is intentionally not preserved. Existing Pipeline and Skill consumers are not migrated here and may be temporarily incompatible after installation.
|
|
@@ -1,14 +1,17 @@
|
|
|
1
|
-
# Service
|
|
1
|
+
# Service Behavior-Risk Catalog
|
|
2
2
|
|
|
3
|
-
Use this catalog during
|
|
3
|
+
Use this catalog during `CONTRACT_MODEL` and `TEST_BUILD` to discover domain-specific observable risks. It supplies questions, not truth: every expected value must come from the recorded truth precedence and every external dependency double must follow `references/contract-mock-protocol.md`.
|
|
4
4
|
|
|
5
5
|
## How to Use This Catalog
|
|
6
6
|
|
|
7
|
-
1. Classify the
|
|
8
|
-
2. Select every service type that applies
|
|
9
|
-
3.
|
|
10
|
-
4.
|
|
11
|
-
5. Record
|
|
7
|
+
1. Classify the affected module by its public behavior, dependencies, state changes, and resolved contracts.
|
|
8
|
+
2. Select every service type that applies; one behavior can combine several types.
|
|
9
|
+
3. Add applicable cases to `behavior-risk-matrix.json` and map them to test/execution IDs.
|
|
10
|
+
4. Derive mock requests/responses from shared machine-readable or locally snapshotted contracts. Never independently invent provider/database/queue behavior.
|
|
11
|
+
5. Record a concrete rationale for every non-applicable risk.
|
|
12
|
+
6. Treat coverage metrics as diagnostic only; complete observable behavior and risk rather than private lines.
|
|
13
|
+
|
|
14
|
+
Code-level tests are mock-first and must state that they do not verify a real deployed environment. Production credentials, services, and data are prohibited.
|
|
12
15
|
|
|
13
16
|
## Service Type Detection
|
|
14
17
|
|