prizmkit 1.1.120 → 1.1.122
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bundled/VERSION.json +3 -3
- package/bundled/dev-pipeline/prizmkit_runtime/cli.py +1 -0
- package/bundled/dev-pipeline/prizmkit_runtime/runner_classification.py +23 -25
- package/bundled/dev-pipeline/prizmkit_runtime/runner_models.py +8 -2
- package/bundled/dev-pipeline/prizmkit_runtime/runner_prompts.py +10 -1
- package/bundled/dev-pipeline/prizmkit_runtime/runners.py +43 -29
- package/bundled/dev-pipeline/scripts/continuation.py +51 -6
- package/bundled/dev-pipeline/scripts/update-bug-status.py +57 -14
- package/bundled/dev-pipeline/scripts/update-feature-status.py +54 -5
- package/bundled/dev-pipeline/scripts/update-refactor-status.py +57 -14
- package/bundled/dev-pipeline/templates/bootstrap-tier3.md +2 -2
- package/bundled/dev-pipeline/templates/bugfix-bootstrap-prompt.md +2 -1
- package/bundled/dev-pipeline/templates/refactor-bootstrap-prompt.md +2 -2
- package/bundled/dev-pipeline/templates/sections/bugfix-phase-review.md +3 -3
- package/bundled/dev-pipeline/templates/sections/phase-review-agent.md +3 -3
- package/bundled/dev-pipeline/templates/sections/phase-review-full.md +4 -4
- package/bundled/dev-pipeline/templates/sections/refactor-phase-review.md +3 -3
- package/bundled/dev-pipeline/tests/test_auto_skip.py +5 -5
- package/bundled/dev-pipeline/tests/test_generate_bootstrap_prompt.py +10 -3
- package/bundled/dev-pipeline/tests/test_generate_bugfix_prompt.py +9 -2
- package/bundled/dev-pipeline/tests/test_generate_refactor_prompt.py +9 -1
- package/bundled/dev-pipeline/tests/test_python_runner_parity.py +222 -18
- package/bundled/dev-pipeline/tests/test_unified_cli.py +24 -11
- package/bundled/skills/_metadata.json +4 -4
- package/bundled/skills/prizmkit-code-review/SKILL.md +49 -48
- package/bundled/skills/prizmkit-code-review/references/review-report-template.md +13 -8
- package/bundled/skills/prizmkit-code-review/references/reviewer-agent-prompt.md +46 -32
- package/bundled/skills/prizmkit-code-review/references/reviewer-workspace-protocol.md +90 -0
- package/bundled/skills/prizmkit-test/SKILL.md +160 -155
- package/bundled/skills/prizmkit-test/assets/authoritative-records.schema.json +420 -0
- package/bundled/skills/prizmkit-test/assets/behavior-risk-matrix.schema.json +116 -0
- package/bundled/skills/prizmkit-test/assets/evidence-manifest.schema.json +112 -0
- package/bundled/skills/prizmkit-test/assets/evidence-package-template.json +97 -0
- package/bundled/skills/prizmkit-test/references/boundary-coverage-protocol.md +76 -192
- package/bundled/skills/prizmkit-test/references/contract-mock-protocol.md +65 -0
- package/bundled/skills/prizmkit-test/references/evidence-protocol.md +201 -0
- package/bundled/skills/prizmkit-test/references/evidence-request-protocol.md +78 -0
- package/bundled/skills/prizmkit-test/references/examples.md +65 -108
- package/bundled/skills/prizmkit-test/references/service-boundary-test-catalog.md +10 -7
- package/bundled/skills/prizmkit-test/references/test-generation-steps.md +90 -82
- package/bundled/skills/prizmkit-test/references/test-report-template.md +77 -98
- package/bundled/skills/prizmkit-test/references/trusted-evidence-execution.md +97 -0
- package/bundled/skills/prizmkit-test/scripts/build_test_evidence.py +739 -0
- package/bundled/skills/prizmkit-test/scripts/validate_test_evidence.py +1442 -0
- package/package.json +1 -1
- package/bundled/skills/prizmkit-test/scripts/validate_boundary_report.py +0 -347
|
@@ -1,224 +1,229 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: "prizmkit-test"
|
|
3
|
-
description: "
|
|
3
|
+
description: "Auditable test-evidence protocol for AI-generated behavior changes. Classifies changes, discovers affected modules and regression rings, models observable behavior and risk, builds project-native tests and contract-driven mocks, proves tests differentially, preserves content-addressed raw evidence, and deterministically returns TEST_PASS, TEST_FAIL, or TEST_BLOCKED. Use after code/config/schema/dependency changes or when users ask to test, verify, add complete module tests, inspect boundaries, or produce auditable evidence. Supports full-project, module, feature, and scope=this-change. (project)"
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# PrizmKit Test
|
|
7
7
|
|
|
8
|
-
|
|
8
|
+
`/prizmkit-test` converts code changes into auditable testing evidence. Completeness means that the affected module's observable behavior, contracts, boundaries, invariants, errors, state transitions, and coupling risks are represented and proven. It does not mean mechanically reaching a line-coverage threshold.
|
|
9
9
|
|
|
10
|
-
|
|
10
|
+
This skill orchestrates each target project's native runners and conventions. It does not install a cross-language test runtime.
|
|
11
11
|
|
|
12
|
-
|
|
13
|
-
- Behavior changes that need verification
|
|
14
|
-
- Public API/interface changes
|
|
15
|
-
- Data model, schema, migration, persistence, or serialization changes
|
|
16
|
-
- Security, permission, authentication, payment, billing, entitlement, or ownership logic
|
|
17
|
-
- Deployment readiness checks
|
|
18
|
-
- First-time test generation for a project with a test framework
|
|
19
|
-
- User says "test", "run tests", "verify", "quality check", "add tests", or asks about coverage/boundaries
|
|
12
|
+
## Model Choice vs Fixed Evidence Mechanics
|
|
20
13
|
|
|
21
|
-
|
|
22
|
-
- Pure documentation edits
|
|
23
|
-
- Formatting-only changes
|
|
24
|
-
- Internal renames with no behavior impact
|
|
25
|
-
- Tiny config tweaks with no runtime behavior change
|
|
26
|
-
- Project has no test framework and no code to test
|
|
14
|
+
Choose project-semantic parameters from the target project: framework, commands, cwd, timeout, retry/attempt policy, concurrency, Mock tools, test layers, module boundary, inventory patterns, mutation technology, and environment classification. Do not hard-code ecosystem-specific defaults when the project can decide them.
|
|
27
15
|
|
|
28
|
-
|
|
16
|
+
Fix only the protocol mechanisms needed for safe, replayable evidence: project/evidence/request locators, schema-shaped requests, path confinement, real process execution, complete raw capture, runner-generated chained receipts, hash binding, append-only history, differential isolation/cleanup, and resume invalidation. Read `${SKILL_DIR}/references/trusted-evidence-execution.md` before inventory, execution, differential proof, resume, or replay; it defines the trust boundary and builder commands. Read `${SKILL_DIR}/references/evidence-request-protocol.md` when authoring any request or authoritative JSON record; it defines request ownership, structured N/A, and the schema map.
|
|
29
17
|
|
|
30
|
-
|
|
31
|
-
|---|---|---|
|
|
32
|
-
| `.prizmkit/prizm-docs/root.prizm` exists | File exists | Run `/prizmkit-init` first |
|
|
33
|
-
| Test framework installed | Dependency in package.json or equivalent | Offer to skip, install manually, or generate a plan to add tests |
|
|
18
|
+
## When to Use
|
|
34
19
|
|
|
35
|
-
|
|
20
|
+
Use for:
|
|
36
21
|
|
|
37
|
-
|
|
22
|
+
- behavior-changing source code;
|
|
23
|
+
- runtime configuration, schema, migration, generated-runtime, lockfile, or dependency changes;
|
|
24
|
+
- public interfaces and data contracts;
|
|
25
|
+
- frontend, backend, library, CLI, adapter, and multi-module changes;
|
|
26
|
+
- legacy modules whose observable test coverage is incomplete;
|
|
27
|
+
- explicit test, boundary, evidence, confidence, or audit requests.
|
|
38
28
|
|
|
39
|
-
|
|
40
|
-
|---|---|
|
|
41
|
-
| `full-project` | Analyze and test the whole project. |
|
|
42
|
-
| `module:<name>` | Analyze one module from Prizm docs. |
|
|
43
|
-
| `feature:<slug>` | Interactive selection of a historical feature/spec directory. |
|
|
44
|
-
| `this-change` | Analyze the current change artifact and changed files only. |
|
|
29
|
+
Documentation, comment, and formatting-only scopes use deterministic lightweight verification. They do not receive an unjustified full behavior-test claim.
|
|
45
30
|
|
|
46
|
-
|
|
31
|
+
## Preconditions and Safety
|
|
47
32
|
|
|
48
|
-
|
|
33
|
+
1. Load `root.prizm`, relevant L1/L2 docs, project manifests, runner configuration, and changed scope once.
|
|
34
|
+
2. Never use production credentials, production databases, production APIs, production queues/storage, or destructive operations against real data.
|
|
35
|
+
3. Preserve complete command/environment/output values as requested. Mark the package `sensitivity=project-controlled`; the project owns access control, retention, and upload policy.
|
|
36
|
+
4. Treat code-level dependencies as mock-first. Real deployed test-environment validation is a separate authorized activity.
|
|
37
|
+
5. Never delete existing tests.
|
|
49
38
|
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
Before execution, load context once:
|
|
53
|
-
|
|
54
|
-
1. Architecture context: `root.prizm`, relevant L1 docs, and relevant L2 docs when scope requires interface or boundary detail.
|
|
55
|
-
2. Project config: `.prizmkit/config.json` if present.
|
|
56
|
-
3. Dependency manifest: `package.json`, `pyproject.toml`, `requirements.txt`, `go.mod`, or equivalent.
|
|
57
|
-
4. For `scope=this-change`:
|
|
58
|
-
- `artifact_dir/spec.md`
|
|
59
|
-
- `artifact_dir/plan.md`
|
|
60
|
-
- `artifact_dir/context-snapshot.md` if present
|
|
61
|
-
- `artifact_dir/review-report.md` if present
|
|
62
|
-
- changed source files and directly related tests
|
|
63
|
-
|
|
64
|
-
Do not load every module or every spec merely because they exist.
|
|
39
|
+
If test infrastructure is missing, the infrastructure preparation state establishes only the necessary project-native infrastructure autonomously. Do not ask the user to install routine test dependencies in a headless run; verify dependency versions before editing manifests.
|
|
65
40
|
|
|
66
41
|
## Input
|
|
67
42
|
|
|
68
43
|
| Parameter | Required | Description |
|
|
69
44
|
|-----------|----------|-------------|
|
|
70
|
-
| `scope` | No | `full-project`, `module:<name>`, `feature:<slug>`, or `this-change`. |
|
|
71
|
-
| `artifact_dir` | No | Change artifact
|
|
72
|
-
| `
|
|
73
|
-
| `
|
|
74
|
-
| `
|
|
45
|
+
| `scope` | No | `full-project`, `module:<name>`, `feature:<slug>`, or `this-change`. Defaults to `this-change` with a valid artifact directory, otherwise `full-project`. |
|
|
46
|
+
| `artifact_dir` | No | Change artifact containing `spec.md` + `plan.md`; used to resolve acceptance conditions and changed scope. |
|
|
47
|
+
| `changed_files` | No | Explicit changed files; highest-priority scope input. |
|
|
48
|
+
| `diff_base` | No | Git baseline used when `changed_files` is absent. |
|
|
49
|
+
| `evidence_target` | No | Existing evidence ID/directory to validate and resume. |
|
|
50
|
+
| `execution_budget` | No | Bounded time/attempt budget. Incomplete necessary work is `TEST_BLOCKED`, never truncated to pass. |
|
|
51
|
+
| `test_commands` | No | Explicit project-native commands when discovery is ambiguous. |
|
|
52
|
+
|
|
53
|
+
For `scope=this-change`, determine changed files from `changed_files`, `diff_base`, current diff, artifact plan/context manifest, then spec mapping. Unknown scope is `TEST_BLOCKED`.
|
|
54
|
+
|
|
55
|
+
Do not infer a current change artifact from a legacy bug/refactor directory name alone; require explicit `artifact_dir` and changed-file evidence.
|
|
56
|
+
|
|
57
|
+
## Authoritative Resources
|
|
58
|
+
|
|
59
|
+
Load each resource only on its branch:
|
|
60
|
+
|
|
61
|
+
- Read `${SKILL_DIR}/references/evidence-protocol.md` before a behavior evidence run or validation/resume decision — it defines state contracts, identity, proof, and verdict semantics.
|
|
62
|
+
- Load `${SKILL_DIR}/assets/evidence-manifest.schema.json` when creating or validating `manifest.json`.
|
|
63
|
+
- Load `${SKILL_DIR}/assets/behavior-risk-matrix.schema.json` during `CONTRACT_MODEL` or matrix validation.
|
|
64
|
+
- Load `${SKILL_DIR}/assets/authoritative-records.schema.json` when creating or validating any other authoritative request/record.
|
|
65
|
+
- Load `${SKILL_DIR}/assets/evidence-package-template.json` when initializing or resuming an evidence directory.
|
|
66
|
+
- Read `${SKILL_DIR}/references/contract-mock-protocol.md` only when dependency, contract, or external-boundary risks require doubles.
|
|
67
|
+
- Read `${SKILL_DIR}/references/test-generation-steps.md` during `INFRA_READY`, `TEST_BUILD`, or `EXECUTE_PROVE`.
|
|
68
|
+
- Read `${SKILL_DIR}/references/service-boundary-test-catalog.md` only when domain boundary signals need risk discovery.
|
|
69
|
+
- Read `${SKILL_DIR}/references/boundary-coverage-protocol.md` when defining module roots, exclusions, Regression Ring, or behavior-risk completeness.
|
|
70
|
+
|
|
71
|
+
## Required State Machine
|
|
72
|
+
|
|
73
|
+
Execute in this exact order: CHANGE_CLASSIFY, SCOPE_DISCOVER, CONTRACT_MODEL, TEST_PLAN, INFRA_READY, TEST_BUILD, EXECUTE_PROVE, EVIDENCE_PACKAGE, EVIDENCE_VALIDATE. Do not claim a later state complete without valid applicable predecessor outputs.
|
|
75
74
|
|
|
76
|
-
|
|
77
|
-
- If `scope=this-change` is provided, require a valid `artifact_dir` with `spec.md` + `plan.md`.
|
|
78
|
-
- If a valid `artifact_dir` is provided without `scope`, treat it as `scope=this-change`.
|
|
79
|
-
- If neither scope nor artifact directory is provided, default to `full-project`.
|
|
75
|
+
### 1. CHANGE_CLASSIFY
|
|
80
76
|
|
|
81
|
-
|
|
77
|
+
Classify the scope as `behavior` or `lightweight` and write `change-classification.json`.
|
|
82
78
|
|
|
83
|
-
|
|
79
|
+
- `behavior`: code, runtime config, schema, migration, generated runtime, dependency, or mixed/uncertain scope; full protocol required.
|
|
80
|
+
- `lightweight`: exclusively docs/comments/formatting with deterministic evidence that runtime behavior cannot change.
|
|
84
81
|
|
|
85
|
-
|
|
86
|
-
- Frontend
|
|
87
|
-
- Backend
|
|
88
|
-
- Fullstack
|
|
89
|
-
- Library/CLI
|
|
90
|
-
- Unknown
|
|
91
|
-
2. Detect test framework:
|
|
92
|
-
- Jest, Vitest, pytest, Go test, Playwright, or custom `npm test`
|
|
93
|
-
- If multiple frameworks exist, use the one with the most relevant test files and list all detected frameworks in the report
|
|
94
|
-
3. Interactive mode: show detected architecture/framework and ask for correction when uncertain.
|
|
95
|
-
4. Headless mode: proceed with detected assumptions and record them in the report.
|
|
82
|
+
Unknown classification follows the full protocol or becomes `TEST_BLOCKED`.
|
|
96
83
|
|
|
97
|
-
###
|
|
84
|
+
### 2. SCOPE_DISCOVER
|
|
98
85
|
|
|
99
|
-
|
|
86
|
+
Write `scope.json`, `target-inventory.json`, and `source-change.patch`. Use a model-authored inventory request with the bundled builder so actual discovered files are hashed. Record:
|
|
100
87
|
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
88
|
+
- changed files, all declared module roots, and evidence-backed exclusions;
|
|
89
|
+
- affected module boundary (`explicit` or `cohesion-derived`);
|
|
90
|
+
- Primary Scope containing all observable behavior of that module;
|
|
91
|
+
- Regression Ring containing direct callers, consumers, shared contracts, and state dependencies, each mapped to planned tests;
|
|
92
|
+
- contract, lockfile, test, and module-boundary discovery evidence;
|
|
93
|
+
- Unresolved Edges for dynamic/unprovable coupling.
|
|
105
94
|
|
|
106
|
-
|
|
95
|
+
Require every changed file to appear in inventory or a structured exclusion. Inventory and static discovery constrain obvious omissions but cannot deterministically prove semantic completeness; preserve that residual model judgment. Any verdict-capable unresolved edge blocks `TEST_PASS`.
|
|
107
96
|
|
|
108
|
-
|
|
109
|
-
2. `git diff --name-only <diff_base>`
|
|
110
|
-
3. files/tasks listed in `artifact_dir/plan.md`
|
|
111
|
-
4. file manifest in `artifact_dir/context-snapshot.md`
|
|
112
|
-
5. file/module mapping from `artifact_dir/spec.md`
|
|
97
|
+
### 3. CONTRACT_MODEL
|
|
113
98
|
|
|
114
|
-
|
|
99
|
+
Write `behavior-risk-matrix.json`. Resolve test truth in strict precedence:
|
|
115
100
|
|
|
116
|
-
|
|
101
|
+
1. specifications;
|
|
102
|
+
2. machine-readable contracts;
|
|
103
|
+
3. acceptance conditions;
|
|
104
|
+
4. trusted existing tests;
|
|
105
|
+
5. callers/consumers;
|
|
106
|
+
6. current implementation.
|
|
117
107
|
|
|
118
|
-
|
|
119
|
-
2. Run the detected test command in scope.
|
|
120
|
-
3. Capture pass/fail counts, failed details, and raw output in the report directory.
|
|
121
|
-
4. For `scope=this-change`, classify failures:
|
|
122
|
-
- In-Scope Failures: changed tests, tests mapped to in-scope source, or acceptance-criteria tests without baseline evidence.
|
|
123
|
-
- Baseline Failures: unrelated pre-existing failures.
|
|
124
|
-
5. If `scope=this-change` has unexplained In-Scope Failures, final verdict must be `NEEDS_FIXES` or `BLOCKED`.
|
|
108
|
+
Unresolved conflicting truth is `TEST_BLOCKED`, not a characterization test.
|
|
125
109
|
|
|
126
|
-
|
|
110
|
+
For every observable behavior, record contract source, preconditions, input classes/boundaries, outputs, side effects, transitions, errors, and functional/boundary/permission/concurrency/idempotency/time/dependency/consumer risks. Every applicable cell maps to tests and executions or remains explicitly unresolved.
|
|
127
111
|
|
|
128
|
-
|
|
112
|
+
Coverage metrics are diagnostic signals only and never substitute for this matrix.
|
|
129
113
|
|
|
130
|
-
|
|
131
|
-
- acceptance criteria from the change artifact
|
|
132
|
-
- directly affected integration boundaries
|
|
133
|
-
- directly affected UI/E2E flows
|
|
114
|
+
### 4. TEST_PLAN
|
|
134
115
|
|
|
135
|
-
|
|
116
|
+
Write `test-plan.json`. Select necessary unit, module/component, contract, integration, and code-level E2E tests by risk. Assess the ordered execution layers:
|
|
136
117
|
|
|
137
|
-
|
|
118
|
+
1. focused;
|
|
119
|
+
2. module/component;
|
|
120
|
+
3. contract/integration;
|
|
121
|
+
4. complete affected-module regression;
|
|
122
|
+
5. Regression Ring.
|
|
138
123
|
|
|
139
|
-
|
|
140
|
-
- Integration tests for module boundaries, API endpoints, DB operations, and external service seams
|
|
141
|
-
- E2E tests for user-visible acceptance criteria when a UI exists
|
|
124
|
+
Every omitted layer uses a structured N/A decision containing rationale, typed evidence, considered signals, and explicit explanations for detected conflicts. The model owns semantic judgment; deterministic validation checks shape and obvious conflicts.
|
|
142
125
|
|
|
143
|
-
###
|
|
126
|
+
### 5. INFRA_READY
|
|
144
127
|
|
|
145
|
-
|
|
128
|
+
Discover and reuse native runners and conventions. Install only required dependencies, configure commands, and create fixtures/fakes/contract mock servers/isolated services as needed. Avoid duplicate frameworks and record every infrastructure change and verification in `infrastructure-changes.json`.
|
|
146
129
|
|
|
147
|
-
|
|
130
|
+
External dependencies use `${SKILL_DIR}/references/contract-mock-protocol.md`. Shared machine-readable contracts are preferred; locally derived contract fixtures must record their source. Independently invented incompatible mocks are rejected.
|
|
148
131
|
|
|
149
|
-
|
|
132
|
+
### 6. TEST_BUILD
|
|
150
133
|
|
|
151
|
-
|
|
152
|
-
- auth / permission / ownership
|
|
153
|
-
- domain invariants
|
|
154
|
-
- collection / pagination
|
|
155
|
-
- date/time boundaries
|
|
156
|
-
- state transitions
|
|
157
|
-
- dependency failures
|
|
158
|
-
- response contract
|
|
134
|
+
Fill gaps across the whole affected module's observable behavior, contracts, boundaries, invariants, failure paths, transitions, and coupling risks. Do not test only the changed happy path or unrelated private implementation lines.
|
|
159
135
|
|
|
160
|
-
|
|
136
|
+
Minimal behavior-preserving production refactoring is allowed only for test seams such as dependency injection, pure-function extraction, adapters, or controllable state. If testing requires changing a public production contract, issue `TEST_BLOCKED`.
|
|
161
137
|
|
|
162
|
-
|
|
163
|
-
- `generated`
|
|
164
|
-
- `needs-review`
|
|
165
|
-
- `skipped` with reason
|
|
166
|
-
- `boundary-missing`
|
|
138
|
+
If a valid test reveals a business/contract defect, do not fix production behavior in this skill. Preserve the evidence for `TEST_FAIL`.
|
|
167
139
|
|
|
168
|
-
###
|
|
140
|
+
### 7. EXECUTE_PROVE
|
|
169
141
|
|
|
170
|
-
|
|
142
|
+
Run required layers in plan order through `${SKILL_DIR}/scripts/build_test_evidence.py execute`. The model selects project-native argv, cwd, timeout, attempts, concurrency, layer, environment classification, and tool probes in schema-shaped requests. The runner performs the actual process execution and appends complete environment/tool versions, raw stdout/stderr hashes, request hash, runner hash, unique receipt, and receipt-chain binding to `executions.json`. Do not accept caller-authored execution JSON and do not retry until green.
|
|
171
143
|
|
|
172
|
-
|
|
144
|
+
Differentially prove each added/changed necessary behavior test:
|
|
173
145
|
|
|
174
|
-
|
|
175
|
-
-
|
|
176
|
-
-
|
|
177
|
-
-
|
|
178
|
-
-
|
|
179
|
-
-
|
|
146
|
+
- Prefer baseline failure plus current success in an isolated/uncontaminated environment.
|
|
147
|
+
- If baseline is inapplicable, use a minimal controlled mutation tied to the same risk.
|
|
148
|
+
- Bind proof to runner receipts, baseline commit, a stable source-snapshot hash, mutation apply/restore hashes, and complete cleanup. The source snapshot excludes runtime-managed `.prizmkit/state/`, evidence output under `.prizmkit/test/evidence/`, `.claude/worktrees/`, Git metadata, and Python bytecode caches because those paths change while the tested code remains identical.
|
|
149
|
+
- Keep source identity separate from evidence integrity: the stable snapshot detects behavior-bearing project drift, while `manifest.json` hashes every evidence file and detects report/output tampering.
|
|
150
|
+
- Completely remove mutations and verify the stable source snapshot is unchanged. Never rebind completed proof to the mutable live project tree.
|
|
151
|
+
- Classify each proof as `PROVEN`, structured `NOT_APPLICABLE`, or `UNPROVEN`.
|
|
180
152
|
|
|
181
|
-
|
|
153
|
+
Necessary `UNPROVEN` behavior, required flakiness, unavailable infrastructure after bounded recovery, failed cleanup, unreliable execution, or budget truncation yields `TEST_BLOCKED`.
|
|
182
154
|
|
|
183
|
-
|
|
155
|
+
### 8. EVIDENCE_PACKAGE
|
|
184
156
|
|
|
185
|
-
|
|
157
|
+
Create `.prizmkit/test/evidence/<evidence-id>/` from the package template with:
|
|
186
158
|
|
|
187
|
-
-
|
|
188
|
-
-
|
|
189
|
-
-
|
|
190
|
-
-
|
|
191
|
-
-
|
|
192
|
-
-
|
|
193
|
-
-
|
|
194
|
-
-
|
|
159
|
+
- `manifest.json`
|
|
160
|
+
- `change-classification.json`
|
|
161
|
+
- `scope.json`
|
|
162
|
+
- `behavior-risk-matrix.json`
|
|
163
|
+
- `test-plan.json`
|
|
164
|
+
- `infrastructure-changes.json`
|
|
165
|
+
- `differential-proof.json`
|
|
166
|
+
- `executions.json`
|
|
167
|
+
- `verdict.json`
|
|
168
|
+
- `source-change.patch`
|
|
169
|
+
- complete `raw/` stdout/stderr
|
|
170
|
+
- `generated-tests/` snapshots
|
|
171
|
+
- `contracts/` snapshots
|
|
172
|
+
- derived `test-report.md`
|
|
195
173
|
|
|
196
|
-
|
|
174
|
+
Hash baseline, diff, the stable source snapshot, inventoried source/tests/contracts/lockfiles, environment, plan, and every evidence file. Runtime state, evidence output, linked Agent worktrees, Git metadata, and bytecode caches are outside source identity; evidence files remain independently content-addressed through `manifest.json`. Set `environment_claim=mocked-code-level-only` and `compatibility=legacy-test-report-interface-not-supported`.
|
|
175
|
+
|
|
176
|
+
### 9. EVIDENCE_VALIDATE
|
|
177
|
+
|
|
178
|
+
Run:
|
|
197
179
|
|
|
198
180
|
```bash
|
|
199
|
-
python3 ${SKILL_DIR}/scripts/
|
|
200
|
-
--
|
|
201
|
-
--
|
|
181
|
+
python3 ${SKILL_DIR}/scripts/validate_test_evidence.py \
|
|
182
|
+
--evidence-dir .prizmkit/test/evidence/<evidence-id> \
|
|
183
|
+
--project-root <target-project-root> \
|
|
184
|
+
--attest
|
|
202
185
|
```
|
|
203
186
|
|
|
204
|
-
|
|
187
|
+
The validator checks every authoritative record against its shipped schema, live target inventories, patch/diff binding, stage dependencies, changed-file/module-root/exclusion/Regression Ring cross-links, generated-test snapshots, planned-test execution links, runner-generated receipts/request/raw-output hashes, successful behavior-risk mappings, proof-linked isolated failures/current successes, production/unknown external-target safety, cleanup, and package integrity. `--attest` writes an integrity/protocol attestation, not proof against a malicious same-permission producer. Use builder replay when feasible to rerun recorded requests and create new linked receipts. Store validation output. Tampering, schema/identity/hash mismatch, source/test/lockfile drift, missing matrix mappings, wrong execution order, invalid differential proof, cleanup failure, unjustified structured N/A, unresolved risk, or a false real-environment claim fails validation. `TEST_PASS` requires validator success and means code-level evidence is replayable and validated under this protocol.
|
|
188
|
+
|
|
189
|
+
## Resume and Idempotency
|
|
190
|
+
|
|
191
|
+
When interrupted or re-invoked:
|
|
192
|
+
|
|
193
|
+
1. validate existing manifest, hashes, and stage outputs;
|
|
194
|
+
2. recompute code/test/contract/dependency/environment/plan input hashes;
|
|
195
|
+
3. preserve prior executions and raw output immutably;
|
|
196
|
+
4. invalidate the first changed stage and every downstream result;
|
|
197
|
+
5. resume from the last valid predecessor.
|
|
198
|
+
|
|
199
|
+
For a large module or bounded budget, persist matrix/checkpoint progress and return `TEST_BLOCKED` until complete evidence exists. Never silently reduce scope.
|
|
200
|
+
|
|
201
|
+
## Terminal Verdicts
|
|
202
|
+
|
|
203
|
+
Only these testing-domain outcomes are valid:
|
|
204
|
+
|
|
205
|
+
- `TEST_PASS`: every required module behavior/risk and Regression Ring check is proven, required executions pass reliably, cleanup succeeds, no verdict-capable unresolved edge remains, and deterministic validation passes.
|
|
206
|
+
- `TEST_FAIL`: a valid reliable test reproduces an implementation or resolved-contract failure.
|
|
207
|
+
- `TEST_BLOCKED`: unknown scope, conflicting truth, unavailable/unreliable/flaky necessary execution, incomplete evidence, unproven tests, failed cleanup, budget truncation, or deterministic validation failure.
|
|
208
|
+
|
|
209
|
+
No conditional pass exists.
|
|
205
210
|
|
|
206
|
-
##
|
|
211
|
+
## Derived Report and Handoff
|
|
207
212
|
|
|
208
|
-
-
|
|
209
|
-
- Generated or updated test files
|
|
210
|
-
- Raw existing-test output, generated-test copies, E2E artifacts, and boundary validation output in the report directory
|
|
213
|
+
Generate `test-report.md` from structured evidence using `${SKILL_DIR}/references/test-report-template.md`. It is a replaceable view, not the source of truth. Report only:
|
|
211
214
|
|
|
212
|
-
|
|
215
|
+
- testing-domain verdict;
|
|
216
|
+
- evidence directory and ID;
|
|
217
|
+
- validator result and key execution pointers;
|
|
218
|
+
- sensitivity and mocked-versus-real warning;
|
|
219
|
+
- explicit notice that the legacy test-report interface is not supported.
|
|
213
220
|
|
|
214
|
-
|
|
221
|
+
Do not embed an independent AI reviewer, decide overall code quality, claim broad Spec compliance, repair a validly detected business defect, authorize commit/release, or perform deployed test/production validation.
|
|
215
222
|
|
|
216
|
-
-
|
|
217
|
-
- Re-run `/prizmkit-test`; it starts fresh but can reuse passing tests and existing generated files.
|
|
218
|
-
- Preserve any `needs-review` findings in the next report.
|
|
223
|
+
Existing Pipeline and Skill consumers are not migrated by this protocol. Temporary post-install incompatibility with callers expecting the legacy report is accepted and must be stated, not hidden.
|
|
219
224
|
|
|
220
225
|
## Examples
|
|
221
226
|
|
|
222
|
-
Read `${SKILL_DIR}/references/examples.md`
|
|
227
|
+
Read `${SKILL_DIR}/references/examples.md` only when a concrete normal, legacy gap-filling, contract-coupled, failure, blocked, recovery, or tampering example is needed.
|
|
223
228
|
|
|
224
|
-
**HANDOFF:**
|
|
229
|
+
**HANDOFF:** Return the strict testing verdict and structured evidence pointers only.
|