prizmkit 1.1.120 → 1.1.121

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (45) hide show
  1. package/bundled/VERSION.json +3 -3
  2. package/bundled/dev-pipeline/prizmkit_runtime/runner_classification.py +23 -25
  3. package/bundled/dev-pipeline/prizmkit_runtime/runner_models.py +8 -2
  4. package/bundled/dev-pipeline/prizmkit_runtime/runner_prompts.py +10 -1
  5. package/bundled/dev-pipeline/prizmkit_runtime/runners.py +43 -29
  6. package/bundled/dev-pipeline/scripts/continuation.py +51 -6
  7. package/bundled/dev-pipeline/scripts/update-bug-status.py +57 -14
  8. package/bundled/dev-pipeline/scripts/update-feature-status.py +54 -5
  9. package/bundled/dev-pipeline/scripts/update-refactor-status.py +57 -14
  10. package/bundled/dev-pipeline/templates/bootstrap-tier3.md +2 -2
  11. package/bundled/dev-pipeline/templates/bugfix-bootstrap-prompt.md +2 -1
  12. package/bundled/dev-pipeline/templates/refactor-bootstrap-prompt.md +2 -2
  13. package/bundled/dev-pipeline/templates/sections/bugfix-phase-review.md +3 -3
  14. package/bundled/dev-pipeline/templates/sections/phase-review-agent.md +3 -3
  15. package/bundled/dev-pipeline/templates/sections/phase-review-full.md +4 -4
  16. package/bundled/dev-pipeline/templates/sections/refactor-phase-review.md +3 -3
  17. package/bundled/dev-pipeline/tests/test_auto_skip.py +5 -5
  18. package/bundled/dev-pipeline/tests/test_generate_bootstrap_prompt.py +10 -3
  19. package/bundled/dev-pipeline/tests/test_generate_bugfix_prompt.py +9 -2
  20. package/bundled/dev-pipeline/tests/test_generate_refactor_prompt.py +9 -1
  21. package/bundled/dev-pipeline/tests/test_python_runner_parity.py +222 -18
  22. package/bundled/dev-pipeline/tests/test_unified_cli.py +14 -10
  23. package/bundled/skills/_metadata.json +4 -4
  24. package/bundled/skills/prizmkit-code-review/SKILL.md +49 -48
  25. package/bundled/skills/prizmkit-code-review/references/review-report-template.md +13 -8
  26. package/bundled/skills/prizmkit-code-review/references/reviewer-agent-prompt.md +46 -32
  27. package/bundled/skills/prizmkit-code-review/references/reviewer-workspace-protocol.md +90 -0
  28. package/bundled/skills/prizmkit-test/SKILL.md +159 -155
  29. package/bundled/skills/prizmkit-test/assets/authoritative-records.schema.json +420 -0
  30. package/bundled/skills/prizmkit-test/assets/behavior-risk-matrix.schema.json +116 -0
  31. package/bundled/skills/prizmkit-test/assets/evidence-manifest.schema.json +112 -0
  32. package/bundled/skills/prizmkit-test/assets/evidence-package-template.json +97 -0
  33. package/bundled/skills/prizmkit-test/references/boundary-coverage-protocol.md +76 -192
  34. package/bundled/skills/prizmkit-test/references/contract-mock-protocol.md +65 -0
  35. package/bundled/skills/prizmkit-test/references/evidence-protocol.md +194 -0
  36. package/bundled/skills/prizmkit-test/references/evidence-request-protocol.md +78 -0
  37. package/bundled/skills/prizmkit-test/references/examples.md +65 -108
  38. package/bundled/skills/prizmkit-test/references/service-boundary-test-catalog.md +10 -7
  39. package/bundled/skills/prizmkit-test/references/test-generation-steps.md +90 -82
  40. package/bundled/skills/prizmkit-test/references/test-report-template.md +77 -98
  41. package/bundled/skills/prizmkit-test/references/trusted-evidence-execution.md +97 -0
  42. package/bundled/skills/prizmkit-test/scripts/build_test_evidence.py +723 -0
  43. package/bundled/skills/prizmkit-test/scripts/validate_test_evidence.py +1426 -0
  44. package/package.json +1 -1
  45. package/bundled/skills/prizmkit-test/scripts/validate_boundary_report.py +0 -347
@@ -1,109 +1,117 @@
1
- # Test Generation Steps (Phase 5)
1
+ # Test Generation and Infrastructure Steps
2
+
3
+ Use this reference during `INFRA_READY`, `TEST_BUILD`, and `EXECUTE_PROVE`. The authoritative scope and completeness inputs are `scope.json`, `behavior-risk-matrix.json`, and `test-plan.json`.
2
4
 
3
5
  ## Scope Guard
4
6
 
5
- Before generating any test, determine whether the run is `scope=this-change`.
7
+ For `scope=this-change`, changed files locate the affected module; they do not restrict tests to changed lines. Fill observable behavior/risk gaps across the whole affected module and execute its Regression Ring. Unrelated modules remain out of scope unless they are direct callers, consumers, shared contracts, or state dependencies.
8
+
9
+ Do not infer current-change scope from a directory name alone. Unknown scope is `TEST_BLOCKED`.
10
+
11
+ ## Native Infrastructure Discovery
12
+
13
+ Before writing tests:
14
+
15
+ 1. Inspect manifests, lockfiles, existing test files, runner configuration, scripts, fixtures, and CI commands.
16
+ 2. Select the existing adequate runner and naming/import/mock conventions.
17
+ 3. Let the target project's semantics determine framework, argv, cwd, timeout, attempts, concurrency, Mock tooling, layers, module roots, inventory patterns, and mutation technology; do not import fixed ecosystem defaults from this skill.
18
+ 4. If no adequate infrastructure exists, add only the smallest project-native runner/config/dependencies required for the planned layers.
19
+ 5. Verify dependency versions against the package registry before writing manifests.
20
+ 6. Record every file/dependency/command change and verification in `infrastructure-changes.json`.
21
+ 7. Never delete existing tests or add a second framework when the current one is adequate.
22
+
23
+ Create fixtures, fakes, contract-backed mock servers, browser harnesses, or isolated services only when required by `test-plan.json`. Every disposable resource needs a unique ID, bounded lifetime, and verified cleanup.
24
+
25
+ ## Matrix-First Test Construction
26
+
27
+ For every behavior row:
28
+
29
+ 1. Read the public interface, contract source, preconditions, inputs/boundaries, outputs, side effects, transitions, and errors.
30
+ 2. Inspect existing test assertions—not names—to find proven cells.
31
+ 3. Assess all eight risks: functional, boundary, permission, concurrency, idempotency, time, dependency, and consumer.
32
+ 4. For every applicable risk, create a distinct test ID and expected execution layer.
33
+ 5. For every non-applicable risk, record a concrete rationale tied to this behavior.
34
+ 6. Keep unresolved truth or untestable necessary behavior explicit; it blocks rather than becoming guessed expected behavior.
35
+
36
+ A module is complete when all observable behaviors and applicable risks have mapped, proven tests and the Regression Ring passes. Line/branch/function coverage may reveal an omission and trigger another matrix review, but a percentage cannot complete a cell.
37
+
38
+ ## Risk-Adaptive Layers
39
+
40
+ Choose only necessary test types, then assess all ordered layers in `test-plan.json`:
41
+
42
+ - **Unit/focused:** pure behavior, boundaries, errors, deterministic collaborators, and fast defect localization.
43
+ - **Module/component:** public module behavior with internal collaborators composed as production uses them.
44
+ - **Contract:** producer/consumer or adapter compatibility against a shared/snapshotted machine-readable contract.
45
+ - **Integration:** multiple local modules or isolated infrastructure when mocks cannot establish the relevant property.
46
+ - **Code-level E2E:** CLI/UI/API flow through code-level boundaries when user-visible composition is at risk and a native harness is available.
47
+ - **Affected-module regression:** every required test for the complete affected module.
48
+ - **Regression Ring:** direct callers, consumers, shared contracts, and state dependencies.
49
+
50
+ Do not require every layer mechanically. An omitted layer needs a specific reason such as “pure library has no UI or process boundary; module tests prove its public composition.” “Not tested” is not a rationale.
6
51
 
7
- For `scope=this-change`, use `generation_policy=in-scope-only` by default. Generate tests only for the in-scope source/test set built from the change artifact and changed files. If a missing test is unrelated to the current change, record it in `Out of Scope Gaps` and do not generate it. A change artifact is any directory with `spec.md` and `plan.md`; it may represent a feature, bug fix, refactor, migration, or other scoped change. Do not infer this-change scope from `.prizmkit/bugfix/` or `.prizmkit/refactor/` by directory name alone; require an explicit artifact directory and changed-files source.
52
+ ## Contract-Driven Doubles
8
53
 
54
+ Read `${SKILL_DIR}/references/contract-mock-protocol.md` for external databases, queues, filesystems, SaaS, browser backends, and cross-module endpoints.
9
55
 
10
- Generate tests from the Phase 4 Boundary Coverage Matrix in priority order: unit integration E2E. After each batch, run immediately.
56
+ Prefer shared OpenAPI/schema/types/generated models. Otherwise snapshot a locally derived contract and record derivation source. Generate success, boundary, malformed, timeout, and failure variants from that contract. Reject independently invented fields or a mock that cannot be traced to a snapshot.
11
57
 
12
- When generated tests fail, distinguish two cases:
13
- - **Test bug** (syntax error, wrong import, wrong mock, wrong framework API usage) → fix the test and re-run
14
- - **Assertion failure** (test is valid but code returns unexpected result) → mark as "needs review" in report; do NOT modify production code. This is a potential bug discovered by the generated test.
58
+ Code-level evidence is mock-first and always reports `environment_claim=mocked-code-level-only`. Do not connect production or a user-linked deployed test database.
15
59
 
16
- If a test fails repeatedly after 2 fix attempts, skip it and mark as "unresolved" in the report.
60
+ ## Whole-Module Legacy Gap Filling
17
61
 
18
- ## 5a. Boundary Matrix First
62
+ When a changed module has incomplete tests:
19
63
 
20
- Before writing any new test, create or update the per-interface Boundary Test Matrix from `references/boundary-coverage-protocol.md`. This step matters because a test suite with broad happy-path coverage can still miss the conditions that cause production failures: invalid input, unauthorized access, ownership mistakes, state transitions, dependency errors, and product-specific invariants.
64
+ - enumerate every observable public behavior, not only changed exports;
65
+ - cover valid/minimal/complex inputs and contract-defined boundaries;
66
+ - cover outputs and response shape;
67
+ - assert side effects and absence of partial effects on failure;
68
+ - cover state transitions and idempotency where applicable;
69
+ - cover permissions, concurrency, clocks, external dependency failures, and consumer assumptions where applicable;
70
+ - avoid tests for unrelated private implementation lines.
21
71
 
22
- For each in-scope interface or endpoint:
23
- - Identify existing happy-path coverage.
24
- - Identify existing boundary coverage by reading test assertions, not just test names.
25
- - Classify applicable categories: request validation, auth/permission/ownership, domain invariants, collection/pagination, date/time, state transitions, dependency failure, and response contract.
26
- - Mark non-applicable categories as `N/A` with a concrete reason.
27
- - Mark applicable uncovered categories as `boundary-missing`.
72
+ Use `${SKILL_DIR}/references/service-boundary-test-catalog.md` to discover domain-specific risks, then map only contract-supported expectations.
28
73
 
29
- Do not generate only happy-path tests. If happy path is missing, generate it, then continue with applicable boundary tests in the same batch.
74
+ ## Testability Seams
30
75
 
31
- ### Existing Test Evaluation
76
+ If production code cannot be controlled:
32
77
 
33
- An existing test counts as boundary coverage only when it:
34
- - targets the same interface/endpoint,
35
- - sends the relevant boundary input or creates the relevant state/dependency failure,
36
- - asserts the expected business code, exception, response shape, state change, or side effect,
37
- - verifies behavior beyond "HTTP 200" or "non-empty response".
78
+ 1. Prefer injecting an existing dependency abstraction.
79
+ 2. Otherwise extract a pure function, add an internal adapter, or make state/clock/randomness controllable.
80
+ 3. Keep the public contract and observable behavior unchanged.
81
+ 4. Record the production patch and behavior-preservation evidence.
38
82
 
39
- A test that only checks the route returns successfully is happy-path coverage, not boundary coverage.
83
+ If a public production contract must change, do not proceed autonomously; return `TEST_BLOCKED`.
40
84
 
41
- ### API Endpoint Boundary Discovery
85
+ ## Valid Failures
42
86
 
43
- For API endpoints, derive boundary cases from:
44
- - OpenAPI request schemas and response schemas,
45
- - handler binding tags and request parsing behavior,
46
- - service validation logic,
47
- - repository/service error handling,
48
- - Prizm L2 TRAPS and DECISIONS,
49
- - existing tests and source behavior,
50
- - product rules in project brief/specs.
87
+ Distinguish:
51
88
 
52
- Map each endpoint to business-specific cases. Examples:
53
- - `POST /auth/refresh`: missing token, malformed token, reused/rotated token, revoked token, token-version race if testable.
54
- - `POST /meals/update`: missing ID, missing foods, invalid meal type, invalid timestamp, not found, wrong owner, recomputed totals, confirmed state transition.
55
- - `POST /stats/trends`: invalid type, invalid period, invalid date range, empty data, each supported trend type, inclusive end date behavior.
56
- - `POST /upload/sign-url`: unsupported content type, file_count 0/default/max/max+1, scene handling, storage provider failure.
89
+ - **Test construction defect:** wrong import, syntax, fixture, mock contract, or runner usage. Fix the test/infrastructure within bounded attempts.
90
+ - **Reliable implementation/contract failure:** valid test against resolved truth repeatedly fails. Preserve the failure and return `TEST_FAIL`; do not repair business behavior here.
91
+ - **Uncertain/flaky/unavailable:** truth unresolved, required test unreliable, infrastructure unavailable, or cleanup fails. Return `TEST_BLOCKED`.
57
92
 
58
- ## 5b. Unit Tests (always applicable)
93
+ Never retry until green or silently weaken an assertion/scope.
59
94
 
60
- For each in-scope uncovered interface:
61
- - Read the source file to understand function signature, branch logic, dependencies, state changes, side effects, and failure paths.
62
- - Classify the function/class method by service responsibility using names, module path, parameters, dependencies, and related Prizm docs/spec acceptance criteria.
63
- - Read `references/service-boundary-test-catalog.md` when the unit is service-like or has business-specific inputs — it maps service responsibilities to boundary cases such as auth/session, payment/wallet, repository, pagination, file storage, notification, queue, cache/idempotency, API client, and date/expiration behavior.
64
- - Build a Unit Test Case Matrix before writing tests. Include generic boundaries plus every applicable service-specific boundary; mark non-applicable categories as `N/A` with a short reason instead of silently skipping them.
65
- - Generate test file matching project conventions (framework, naming, directory, import style, mock/fixture patterns)
66
- - Common naming patterns to match:
67
- - `src/foo.ts` → `src/__tests__/foo.test.ts` or `src/foo.spec.ts` or `tests/foo_test.py`
68
- - Mirror the existing pattern; if no tests exist, use the framework's default convention
69
- - Cover: happy path, service-specific boundary cases, edge cases (null/undefined/empty), error conditions, branch paths, boundary values, state/side effects, and dependency failures.
70
- - Verify mocks for dependency method calls, call count/order/arguments when the behavior depends on collaborators.
71
- - Do NOT test framework internals or third-party library behavior
72
- - Run tests immediately after generating
95
+ ## Execution and Differential Proof
73
96
 
74
- ## 5c. Integration Tests
97
+ Run selected tests in order through the bundled evidence builder: focused, module/component, contract/integration, affected-module regression, Regression Ring. The model writes schema-shaped requests; the builder performs the process and appends immutable runner-generated receipts to `executions.json`.
75
98
 
76
- For each in-scope module with dependencies:
77
- - Generate tests for the module's primary interface exercising its real dependencies.
78
- - If API endpoints exist, generate request/response tests for every applicable boundary category in the Boundary Matrix, not only valid input. Cover auth, missing params, invalid formats/enums, ownership/not-found, state transitions, and response envelope or schema.
79
- - If database operations exist, generate CRUD tests using the project's existing test database config when available. Cover transactions, duplicate keys, soft-delete visibility, rollback, and query boundary behavior when applicable.
80
- - If a dependency failure is not practical at integration level, mark it `N/A` with the reason and cover it in a unit test where mocks are available.
81
- - Run tests immediately after generating.
99
+ For each new/changed necessary behavior test:
82
100
 
83
- ## 5d. E2E Tests (conditional)
101
+ 1. In an isolated checkout/equivalent, run it against the actual baseline commit and require failure for the intended recorded signal, then current success.
102
+ 2. If baseline proof is genuinely inapplicable, apply one model-selected minimal controlled mutation tied to the risk, require failure, restore it, and verify request/receipt, mutation apply/restore, isolation tree, and original project tree hashes.
103
+ 3. Record `PROVEN`, structured `NOT_APPLICABLE`, or `UNPROVEN` with runner receipt IDs.
84
104
 
85
- Preconditions (ALL must be met):
86
- - Playwright is available (`@playwright/test` or `playwright` in package.json dependencies, or `npx playwright --version` succeeds as fallback)
87
- - Playwright browsers are installed (check `npx playwright install --dry-run 2>/dev/null` exits cleanly, or `node_modules/.cache/ms-playwright/` directory exists; if missing, warn in report and skip E2E)
88
- - Project has a UI layer
89
- - Project can be started (start/dev script in package.json)
90
- - Acceptance criteria exist in spec.md files
105
+ Necessary `UNPROVEN` evidence blocks. Store complete stdout/stderr in `raw/`; snapshots in `generated-tests/`; do not overwrite previous attempts.
91
106
 
92
- If ALL met:
93
- - **Before starting dev server**: detect whether it's already running: (1) check `package.json` scripts for port flags (`--port`, `-p`, `PORT=`), (2) fall back to framework defaults (3000 for React/Next/Express, 5173 for Vite, 8000 for Django, 5000 for Flask), (3) if still unknown, ask the user. Use the detected port in `lsof -i :<port>` to check. If running, tell user: "Dev server appears to be already running on port {N}. Use this running instance for E2E tests?" If user confirms, use existing instance. If user declines, skip E2E.
94
- - Start the project dev server (only if not already running). Wait for it to be ready.
95
- - For each in-scope uncovered acceptance criterion, generate a Playwright test script.
96
- - Include boundary user journeys when acceptance criteria mention validation, permissions, empty states, error messages, or lifecycle transitions.
97
- - Run generated E2E tests, capture screenshots of failure.
98
- - **Only stop dev server if you started it.** Never stop a server that was already running.
107
+ ## Completion Check
99
108
 
100
- If NOT met:
101
- - Note what was skipped and why in the report
102
- - If only the server wasn't startable: still generate E2E script files but mark as "not run — manual execution needed"
109
+ Before packaging:
103
110
 
104
- ## 5e. Boundary Completion Check
105
-
106
- After generating and running tests:
107
- - Update the Boundary Matrix statuses based on passing tests and `needs-review` findings.
108
- - If any interface remains `boundary-missing` or happy-path-only, run another targeted generation pass for those categories before final reporting.
109
- - Stop only when all in-scope interfaces are `boundary-covered`, `generated-needs-review`, or `skipped-with-reason`, or when remaining gaps are explicitly listed as unresolved.
111
+ - no applicable matrix cell lacks test/execution mapping;
112
+ - all required planned layers have reliable selected executions in order;
113
+ - affected-module and Regression Ring regressions are complete;
114
+ - every new/changed necessary behavior has valid differential proof;
115
+ - every isolated resource/mutation is cleaned up;
116
+ - no verdict-capable unresolved edge remains for pass;
117
+ - budgeted/staged work is complete, otherwise `TEST_BLOCKED`.
@@ -1,111 +1,90 @@
1
- # Test Report Output Template
1
+ # Derived Test Evidence Report Template
2
2
 
3
- Used in Phase 6 of `/prizmkit-test`. Create `.prizmkit/test/{YYYY_MM_DD_HH_MM_SS}_testresult/` directory and write `test-report.md`.
4
-
5
- ## test-report.md Format
3
+ Generate this report solely from the structured, hashed evidence package. It is not independently editable authority. Any change to this file or another linked artifact without manifest regeneration causes deterministic validation failure.
6
4
 
7
5
  ```markdown
8
- # Test Report — {timestamp}
6
+ # Test Evidence Report
9
7
 
10
- ## Summary
11
- - Scope: {full / module: name / feature: ###-name / this-change}
12
- - Architecture: {frontend / backend / fullstack}
13
- - Test framework: {name}
14
- - Mode: {interactive / headless}
8
+ Derived from structured evidence. Authoritative package: `{evidence_dir}`
15
9
 
16
- ## Scope
17
- - Mode: {full-project / module:<name> / feature:<slug> / this-change}
18
- - Artifact Dir: {none / path to change artifact containing spec.md + plan.md}
19
- - Diff Base: {ref or none}
20
- - Changed Files Source: {changed_files / diff_base / plan.md / context-snapshot.md / spec.md / not applicable}
21
- - Generation Policy: {in-scope-only / full-project}
22
- - In-Scope Source Files:
23
- - {...}
24
- - In-Scope Test Files:
25
- - {...}
26
-
27
-
28
- ## Existing Tests Run
29
- - Total: {N} | Passed: {N} | Failed: {N}
30
- {failed test details or "All passing"}
31
- {or "No existing tests found — generated first batch"}
32
-
33
- ## Coverage Gaps Found
34
- | Module | Interface | Type | Status |
35
- |--------|-----------|------|--------|
36
- | ... | ... | unit/integration/e2e | covered / generated / needs-review / unresolved / skipped / boundary-missing |
37
-
38
- ## Boundary Matrix
39
- | Module | Interface | Service Type | Happy Path | Request Validation | Auth / Permission / Ownership | Domain Invariants | Collection / Pagination | Date / Time | State Transitions | Dependency Failure | Response Contract | N/A Reasons | Final Status |
40
- |--------|-----------|--------------|------------|--------------------|-------------------------------|-------------------|-------------------------|-------------|-------------------|-------------------|-------------------|-------------|--------------|
41
- | ... | ... | auth/repository/upload/etc. | covered/generated/skipped | covered/generated/needs-review/skipped/boundary-missing | ... | ... | ... | ... | ... | ... | ... | category — reason | boundary-covered / generated-needs-review / skipped-with-reason / boundary-missing |
42
-
43
- ## Boundary Completion Gate
44
- - Interfaces total: {N}
45
- - Boundary-covered: {N}
46
- - Generated-needs-review: {N}
47
- - Skipped-with-reason: {N}
48
- - Boundary-missing: {N}
49
- - Happy-path-only: {N}
50
- - Completion gate passed: {yes / no}
51
-
52
- If `Boundary-missing > 0` or `Happy-path-only > 0`, write:
53
- `Tests may pass, but boundary coverage is incomplete.`
54
-
55
- ## Boundary Coverage Notes
56
- | Module | Interface | Boundary Categories Covered | Missing Applicable Boundaries | N/A Categories with Reasons |
57
- |--------|-----------|-----------------------------|-------------------------------|-----------------------------|
58
- | ... | ... | request validation, ownership, dependency failure | invalid timestamp, repo failure | pagination — endpoint has no list semantics |
59
-
60
- ## Generated / Updated Tests
61
- - Unit: {N} generated, {N} needs-review, {N} unresolved
62
- - Integration: {N} generated, {N} needs-review, {N} unresolved
63
- - E2E: {N} generated, {N} skipped ({reason})
64
-
65
- ## Boundary Validation
66
- - Validator command: `{command or "not run — reason"}`
67
- - Validator result: {passed / failed / skipped}
68
- - Validator output: {summary or path to raw output}
69
-
70
- ## In-Scope Failures
71
- - {failures caused by this-change scope, or "None"}
72
-
73
- ## Baseline Failures
74
- - {pre-existing failures, or "None"}
75
-
76
- ## Out of Scope Gaps
77
- - {unrelated historical gaps; informational only for scope=this-change}
10
+ ## Evidence
11
+ - Evidence ID: {64-character evidence_id}
12
+ - Protocol Version: {protocol_version}
13
+ - Classification: {behavior / lightweight}
14
+ - Baseline Commit: {baseline_commit}
15
+ - Working Diff SHA-256: {working_diff_sha256}
16
+ - Sensitivity: sensitivity=project-controlled
17
+ - Environment Claim: mocked-code-level-only
78
18
 
79
- ## Verdict
80
- PASS | NEEDS_FIXES | BLOCKED
81
-
82
- ## Final Status
83
- - All tests passing: {yes / no}
84
- - Boundary completion gate passed: {yes / no}
85
- - Ready for commit: {yes / no — include reason}
86
- - Needs human review: {list of tests/interfaces marked needs-review}
87
- {remaining failures if any}
88
- ```
19
+ The evidence package preserves complete command, environment, stdout, stderr, patch, hash, and execution-history values without automatic redaction. The project owns access control, retention, and upload policy. Production credentials, production services, and destructive real-data operations are prohibited.
89
20
 
90
- ## Artifacts
21
+ Mocked code-level evidence does not verify a real deployed environment.
91
22
 
92
- Also save alongside `test-report.md`:
93
- - `existing-test-output.txt` raw output from Phase 2
94
- - `generated-tests/` copies of all generated test files
95
- - `e2e-output/` E2E logs and failure screenshots (if applicable)
96
- - `boundary-validation-output.txt` output from `scripts/validate_boundary_report.py` when run
23
+ ## Scope
24
+ - Affected Module: {name} ({explicit / cohesion-derived})
25
+ - Primary Scope: {count and pointers}
26
+ - Regression Ring: {count and pointers}
27
+ - Verdict-capable Unresolved Edges: {count}
28
+
29
+ ## Behavior-Risk Completeness
30
+ - Observable Behaviors: {count}
31
+ - Applicable Risk Cells: {count}
32
+ - Proven/Mapped Risk Cells: {count}
33
+ - Unresolved Risk Cells: {count}
34
+ - Diagnostic Coverage Signals: {values or not collected; never the completion basis}
35
+ - Matrix: `behavior-risk-matrix.json`
36
+
37
+ ## Risk-Adaptive Test Plan
38
+ | Layer | Required | Rationale | Tests | Selected Execution IDs |
39
+ |-------|----------|-----------|-------|------------------------|
40
+ | focused | {yes/no} | {...} | {...} | {...} |
41
+ | module-component | {yes/no} | {...} | {...} | {...} |
42
+ | contract-integration | {yes/no} | {...} | {...} | {...} |
43
+ | affected-module-regression | {yes/no} | {...} | {...} | {...} |
44
+ | regression-ring | {yes/no} | {...} | {...} | {...} |
45
+
46
+ ## Infrastructure and Contracts
47
+ - Native Runner/Conventions: {...}
48
+ - Infrastructure Changes: `infrastructure-changes.json`
49
+ - Contract Snapshots: `contracts/`
50
+ - Production Resources Used: no
51
+ - Cleanup Succeeded: {yes/no}
52
+
53
+ ## Differential Proof
54
+ - PROVEN: {count}
55
+ - NOT_APPLICABLE: {count with rationale pointers}
56
+ - UNPROVEN: {count}
57
+ - Proof Record: `differential-proof.json`
58
+
59
+ ## Executions
60
+ - Attempts Preserved: {count}
61
+ - Selected Required Executions: {IDs in order}
62
+ - Complete History: `executions.json`
63
+ - Raw Outputs: `raw/`
64
+
65
+ ## Deterministic Validation
66
+ - Command: `python3 ${SKILL_DIR}/scripts/validate_test_evidence.py --evidence-dir {evidence_dir} --project-root {target_project_root}`
67
+ - Result: {passed / failed}
68
+ - Validation Record: `validation.json`
97
69
 
98
- ## Final Status Rules
70
+ ## Verdict
71
+ Verdict: TEST_PASS | TEST_FAIL | TEST_BLOCKED
99
72
 
100
- Use these rules so the report does not overstate quality:
73
+ ## Responsibility Boundary
74
+ - Testing-domain verdict only: yes
75
+ - Overall code-quality or broad Spec verdict: no
76
+ - Business defects repaired by this skill: no
77
+ - Commit/release authorized: no
78
+ - Real deployed environment validated: no
101
79
 
102
- | Tests Passing | Boundary Gate | Needs Review | Ready for Commit |
103
- |---------------|---------------|--------------|------------------|
104
- | yes | yes | no | yes |
105
- | yes | yes | yes | no — human review required |
106
- | yes | no | any | no — boundary coverage incomplete |
107
- | no | any | any | no — tests failing |
80
+ ## Compatibility
81
+ The legacy test-report interface is not supported. Existing Pipeline or other Skill consumers were not migrated and may be temporarily incompatible after installation.
82
+ ```
108
83
 
109
- Do not write "All tests passing. Ready for commit." if any interface is `boundary-missing`, `happy-path-only`, `unresolved`, or `needs-review`.
84
+ ## Verdict Rendering Rules
110
85
 
111
- Out-of-scope gaps are informational only. They must not trigger generated tests and must not fail a `scope=this-change` run.
86
+ - Render `TEST_PASS` only after deterministic validation passes and all required evidence is proven.
87
+ - Render `TEST_FAIL` only with reliable reproduced failure execution pointers.
88
+ - Render `TEST_BLOCKED` with explicit blocker pointers.
89
+ - Never render `PASS`, `NEEDS_FIXES`, conditional pass, “ready for commit,” or release authorization.
90
+ - A lightweight `TEST_PASS` must say it proves only the deterministic lightweight claim, not runtime behavior.
@@ -0,0 +1,97 @@
1
+ # Trusted Evidence Execution
2
+
3
+ Load this reference before `SCOPE_DISCOVER`, `EXECUTE_PROVE`, resume, or replay. It defines which choices remain with the model and which evidence mechanics are fixed.
4
+
5
+ ## Two Parameter Classes
6
+
7
+ ### Model-Chosen Project Parameters
8
+
9
+ Inspect the target project and choose these semantically:
10
+
11
+ - native test framework, runner, command, flags, and working directory;
12
+ - timeout, attempt/retry policy, concurrency, resource limits, and isolation strategy;
13
+ - unit/module/contract/integration/code-level E2E layers;
14
+ - fixture, fake, Mock, service-double, browser, database, and mutation technology;
15
+ - module roots, public behavior boundary, inventory patterns, exclusions, Regression Ring, and differential method;
16
+ - environment classification and endpoint allow/deny evidence.
17
+
18
+ The protocol does not prescribe Jest, pytest, a timeout number, a retry count, coverage threshold, runner flag, or mutation package. Record the reason for each choice in request and planning records.
19
+
20
+ ### Protocol-Fixed Locators and Mechanisms
21
+
22
+ The protocol fixes only what trustworthy, replayable packaging needs:
23
+
24
+ - locator arguments: project root, evidence directory, request/manifest/inventory/receipt path;
25
+ - schema-shaped request records;
26
+ - project-root and evidence-root path confinement;
27
+ - actual child-process execution without a shell;
28
+ - complete stdout/stderr capture and output hashes;
29
+ - complete effective environment and tool-probe capture;
30
+ - runner-generated UUID receipt, request hash, runner hash, and receipt-chain hash;
31
+ - append-only execution history;
32
+ - differential isolation, cleanup, and tree hashes;
33
+ - deterministic resume invalidation calculation;
34
+ - strict schema and cross-record validation.
35
+
36
+ ## Trust Boundary
37
+
38
+ A receipt proves that the bundled runner produced a record with the captured inputs and outputs during this invocation. The runner snapshots its own executable bytes inside the evidence package so later validation does not silently depend on whichever PrizmKit version happens to be installed. Hashes bind bytes together and detect later drift. They do not create cryptographic non-repudiation and cannot prove that a same-OS-user, same-permission producer did not replace the runner, requests, repository, or evidence package before validation.
39
+
40
+ Treat evidence as auditable project-controlled provenance, not a defense against a malicious same-privilege actor. The validator checks consistency, obvious conflicts, and replayability; it cannot mathematically prove that the model found every business behavior.
41
+
42
+ ## Builder Commands
43
+
44
+ Write requests beneath the evidence directory, then invoke the bundled script:
45
+
46
+ ```bash
47
+ python3 ${SKILL_DIR}/scripts/build_test_evidence.py \
48
+ --project-root <target-project-root> \
49
+ --evidence-dir <evidence-dir> \
50
+ inventory --request requests/inventory.json
51
+
52
+ python3 ${SKILL_DIR}/scripts/build_test_evidence.py \
53
+ --project-root <target-project-root> \
54
+ --evidence-dir <evidence-dir> \
55
+ execute --request requests/focused.json
56
+
57
+ python3 ${SKILL_DIR}/scripts/build_test_evidence.py \
58
+ --project-root <target-project-root> \
59
+ --evidence-dir <evidence-dir> \
60
+ differential --request requests/differential.json
61
+
62
+ python3 ${SKILL_DIR}/scripts/build_test_evidence.py \
63
+ --project-root <target-project-root> \
64
+ --evidence-dir <evidence-dir> \
65
+ resume --manifest manifest.json --inventory target-inventory.json
66
+ ```
67
+
68
+ The subcommands cover fixed mechanics only:
69
+
70
+ - `inventory`: expand model-selected project-relative patterns, independently enumerate all actual files under each declared module root, reject files that are neither categorized nor structurally excluded, hash actual files, and bind discovery inputs;
71
+ - `execute`: run the declared argv in the declared safe cwd and append a runner-generated receipt;
72
+ - `differential`: execute baseline or controlled-mutation proof in isolated copies and verify cleanup/tree binding;
73
+ - `resume`: compare hashes and stage outputs to compute an invalidation decision while retaining `semantic_review_required=true`.
74
+
75
+ ## Replay
76
+
77
+ Replay a prior receipt through the runner rather than treating attestation as historical proof:
78
+
79
+ ```bash
80
+ python3 ${SKILL_DIR}/scripts/build_test_evidence.py \
81
+ --project-root <target-project-root> \
82
+ --evidence-dir <evidence-dir> \
83
+ execute --replay-receipt receipts/<receipt>.json
84
+ ```
85
+
86
+ The recorded request hash must still match. Replay emits a new receipt linked with `replay_of`; it never overwrites the historical receipt. A replay can still differ because project state, installed tools, clocks, external test services, or environment values changed. Report those differences instead of claiming historical equivalence.
87
+
88
+ ## External Target Safety
89
+
90
+ Do not block merely because a string contains `production`; names can be misleading. Classify every external target using endpoint/configuration evidence:
91
+
92
+ - block `production` and `unknown` external targets;
93
+ - permit only model-classified isolated/test/staging targets with explicit allow evidence and no deny evidence;
94
+ - preserve the complete evidence used for the decision;
95
+ - keep local fakes and in-process contract mocks classified `local` with `external=false`.
96
+
97
+ The model owns the semantic classification; the runner enforces the declared production/unknown safety invariant and rejects contradictory allow/deny records.