prizmkit 1.1.120 → 1.1.122
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bundled/VERSION.json +3 -3
- package/bundled/dev-pipeline/prizmkit_runtime/cli.py +1 -0
- package/bundled/dev-pipeline/prizmkit_runtime/runner_classification.py +23 -25
- package/bundled/dev-pipeline/prizmkit_runtime/runner_models.py +8 -2
- package/bundled/dev-pipeline/prizmkit_runtime/runner_prompts.py +10 -1
- package/bundled/dev-pipeline/prizmkit_runtime/runners.py +43 -29
- package/bundled/dev-pipeline/scripts/continuation.py +51 -6
- package/bundled/dev-pipeline/scripts/update-bug-status.py +57 -14
- package/bundled/dev-pipeline/scripts/update-feature-status.py +54 -5
- package/bundled/dev-pipeline/scripts/update-refactor-status.py +57 -14
- package/bundled/dev-pipeline/templates/bootstrap-tier3.md +2 -2
- package/bundled/dev-pipeline/templates/bugfix-bootstrap-prompt.md +2 -1
- package/bundled/dev-pipeline/templates/refactor-bootstrap-prompt.md +2 -2
- package/bundled/dev-pipeline/templates/sections/bugfix-phase-review.md +3 -3
- package/bundled/dev-pipeline/templates/sections/phase-review-agent.md +3 -3
- package/bundled/dev-pipeline/templates/sections/phase-review-full.md +4 -4
- package/bundled/dev-pipeline/templates/sections/refactor-phase-review.md +3 -3
- package/bundled/dev-pipeline/tests/test_auto_skip.py +5 -5
- package/bundled/dev-pipeline/tests/test_generate_bootstrap_prompt.py +10 -3
- package/bundled/dev-pipeline/tests/test_generate_bugfix_prompt.py +9 -2
- package/bundled/dev-pipeline/tests/test_generate_refactor_prompt.py +9 -1
- package/bundled/dev-pipeline/tests/test_python_runner_parity.py +222 -18
- package/bundled/dev-pipeline/tests/test_unified_cli.py +24 -11
- package/bundled/skills/_metadata.json +4 -4
- package/bundled/skills/prizmkit-code-review/SKILL.md +49 -48
- package/bundled/skills/prizmkit-code-review/references/review-report-template.md +13 -8
- package/bundled/skills/prizmkit-code-review/references/reviewer-agent-prompt.md +46 -32
- package/bundled/skills/prizmkit-code-review/references/reviewer-workspace-protocol.md +90 -0
- package/bundled/skills/prizmkit-test/SKILL.md +160 -155
- package/bundled/skills/prizmkit-test/assets/authoritative-records.schema.json +420 -0
- package/bundled/skills/prizmkit-test/assets/behavior-risk-matrix.schema.json +116 -0
- package/bundled/skills/prizmkit-test/assets/evidence-manifest.schema.json +112 -0
- package/bundled/skills/prizmkit-test/assets/evidence-package-template.json +97 -0
- package/bundled/skills/prizmkit-test/references/boundary-coverage-protocol.md +76 -192
- package/bundled/skills/prizmkit-test/references/contract-mock-protocol.md +65 -0
- package/bundled/skills/prizmkit-test/references/evidence-protocol.md +201 -0
- package/bundled/skills/prizmkit-test/references/evidence-request-protocol.md +78 -0
- package/bundled/skills/prizmkit-test/references/examples.md +65 -108
- package/bundled/skills/prizmkit-test/references/service-boundary-test-catalog.md +10 -7
- package/bundled/skills/prizmkit-test/references/test-generation-steps.md +90 -82
- package/bundled/skills/prizmkit-test/references/test-report-template.md +77 -98
- package/bundled/skills/prizmkit-test/references/trusted-evidence-execution.md +97 -0
- package/bundled/skills/prizmkit-test/scripts/build_test_evidence.py +739 -0
- package/bundled/skills/prizmkit-test/scripts/validate_test_evidence.py +1442 -0
- package/package.json +1 -1
- package/bundled/skills/prizmkit-test/scripts/validate_boundary_report.py +0 -347
|
@@ -1,109 +1,117 @@
|
|
|
1
|
-
# Test Generation
|
|
1
|
+
# Test Generation and Infrastructure Steps
|
|
2
|
+
|
|
3
|
+
Use this reference during `INFRA_READY`, `TEST_BUILD`, and `EXECUTE_PROVE`. The authoritative scope and completeness inputs are `scope.json`, `behavior-risk-matrix.json`, and `test-plan.json`.
|
|
2
4
|
|
|
3
5
|
## Scope Guard
|
|
4
6
|
|
|
5
|
-
|
|
7
|
+
For `scope=this-change`, changed files locate the affected module; they do not restrict tests to changed lines. Fill observable behavior/risk gaps across the whole affected module and execute its Regression Ring. Unrelated modules remain out of scope unless they are direct callers, consumers, shared contracts, or state dependencies.
|
|
8
|
+
|
|
9
|
+
Do not infer current-change scope from a directory name alone. Unknown scope is `TEST_BLOCKED`.
|
|
10
|
+
|
|
11
|
+
## Native Infrastructure Discovery
|
|
12
|
+
|
|
13
|
+
Before writing tests:
|
|
14
|
+
|
|
15
|
+
1. Inspect manifests, lockfiles, existing test files, runner configuration, scripts, fixtures, and CI commands.
|
|
16
|
+
2. Select the existing adequate runner and naming/import/mock conventions.
|
|
17
|
+
3. Let the target project's semantics determine framework, argv, cwd, timeout, attempts, concurrency, Mock tooling, layers, module roots, inventory patterns, and mutation technology; do not import fixed ecosystem defaults from this skill.
|
|
18
|
+
4. If no adequate infrastructure exists, add only the smallest project-native runner/config/dependencies required for the planned layers.
|
|
19
|
+
5. Verify dependency versions against the package registry before writing manifests.
|
|
20
|
+
6. Record every file/dependency/command change and verification in `infrastructure-changes.json`.
|
|
21
|
+
7. Never delete existing tests or add a second framework when the current one is adequate.
|
|
22
|
+
|
|
23
|
+
Create fixtures, fakes, contract-backed mock servers, browser harnesses, or isolated services only when required by `test-plan.json`. Every disposable resource needs a unique ID, bounded lifetime, and verified cleanup.
|
|
24
|
+
|
|
25
|
+
## Matrix-First Test Construction
|
|
26
|
+
|
|
27
|
+
For every behavior row:
|
|
28
|
+
|
|
29
|
+
1. Read the public interface, contract source, preconditions, inputs/boundaries, outputs, side effects, transitions, and errors.
|
|
30
|
+
2. Inspect existing test assertions—not names—to find proven cells.
|
|
31
|
+
3. Assess all eight risks: functional, boundary, permission, concurrency, idempotency, time, dependency, and consumer.
|
|
32
|
+
4. For every applicable risk, create a distinct test ID and expected execution layer.
|
|
33
|
+
5. For every non-applicable risk, record a concrete rationale tied to this behavior.
|
|
34
|
+
6. Keep unresolved truth or untestable necessary behavior explicit; it blocks rather than becoming guessed expected behavior.
|
|
35
|
+
|
|
36
|
+
A module is complete when all observable behaviors and applicable risks have mapped, proven tests and the Regression Ring passes. Line/branch/function coverage may reveal an omission and trigger another matrix review, but a percentage cannot complete a cell.
|
|
37
|
+
|
|
38
|
+
## Risk-Adaptive Layers
|
|
39
|
+
|
|
40
|
+
Choose only necessary test types, then assess all ordered layers in `test-plan.json`:
|
|
41
|
+
|
|
42
|
+
- **Unit/focused:** pure behavior, boundaries, errors, deterministic collaborators, and fast defect localization.
|
|
43
|
+
- **Module/component:** public module behavior with internal collaborators composed as production uses them.
|
|
44
|
+
- **Contract:** producer/consumer or adapter compatibility against a shared/snapshotted machine-readable contract.
|
|
45
|
+
- **Integration:** multiple local modules or isolated infrastructure when mocks cannot establish the relevant property.
|
|
46
|
+
- **Code-level E2E:** CLI/UI/API flow through code-level boundaries when user-visible composition is at risk and a native harness is available.
|
|
47
|
+
- **Affected-module regression:** every required test for the complete affected module.
|
|
48
|
+
- **Regression Ring:** direct callers, consumers, shared contracts, and state dependencies.
|
|
49
|
+
|
|
50
|
+
Do not require every layer mechanically. An omitted layer needs a specific reason such as “pure library has no UI or process boundary; module tests prove its public composition.” “Not tested” is not a rationale.
|
|
6
51
|
|
|
7
|
-
|
|
52
|
+
## Contract-Driven Doubles
|
|
8
53
|
|
|
54
|
+
Read `${SKILL_DIR}/references/contract-mock-protocol.md` for external databases, queues, filesystems, SaaS, browser backends, and cross-module endpoints.
|
|
9
55
|
|
|
10
|
-
|
|
56
|
+
Prefer shared OpenAPI/schema/types/generated models. Otherwise snapshot a locally derived contract and record derivation source. Generate success, boundary, malformed, timeout, and failure variants from that contract. Reject independently invented fields or a mock that cannot be traced to a snapshot.
|
|
11
57
|
|
|
12
|
-
|
|
13
|
-
- **Test bug** (syntax error, wrong import, wrong mock, wrong framework API usage) → fix the test and re-run
|
|
14
|
-
- **Assertion failure** (test is valid but code returns unexpected result) → mark as "needs review" in report; do NOT modify production code. This is a potential bug discovered by the generated test.
|
|
58
|
+
Code-level evidence is mock-first and always reports `environment_claim=mocked-code-level-only`. Do not connect production or a user-linked deployed test database.
|
|
15
59
|
|
|
16
|
-
|
|
60
|
+
## Whole-Module Legacy Gap Filling
|
|
17
61
|
|
|
18
|
-
|
|
62
|
+
When a changed module has incomplete tests:
|
|
19
63
|
|
|
20
|
-
|
|
64
|
+
- enumerate every observable public behavior, not only changed exports;
|
|
65
|
+
- cover valid/minimal/complex inputs and contract-defined boundaries;
|
|
66
|
+
- cover outputs and response shape;
|
|
67
|
+
- assert side effects and absence of partial effects on failure;
|
|
68
|
+
- cover state transitions and idempotency where applicable;
|
|
69
|
+
- cover permissions, concurrency, clocks, external dependency failures, and consumer assumptions where applicable;
|
|
70
|
+
- avoid tests for unrelated private implementation lines.
|
|
21
71
|
|
|
22
|
-
|
|
23
|
-
- Identify existing happy-path coverage.
|
|
24
|
-
- Identify existing boundary coverage by reading test assertions, not just test names.
|
|
25
|
-
- Classify applicable categories: request validation, auth/permission/ownership, domain invariants, collection/pagination, date/time, state transitions, dependency failure, and response contract.
|
|
26
|
-
- Mark non-applicable categories as `N/A` with a concrete reason.
|
|
27
|
-
- Mark applicable uncovered categories as `boundary-missing`.
|
|
72
|
+
Use `${SKILL_DIR}/references/service-boundary-test-catalog.md` to discover domain-specific risks, then map only contract-supported expectations.
|
|
28
73
|
|
|
29
|
-
|
|
74
|
+
## Testability Seams
|
|
30
75
|
|
|
31
|
-
|
|
76
|
+
If production code cannot be controlled:
|
|
32
77
|
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
- verifies behavior beyond "HTTP 200" or "non-empty response".
|
|
78
|
+
1. Prefer injecting an existing dependency abstraction.
|
|
79
|
+
2. Otherwise extract a pure function, add an internal adapter, or make state/clock/randomness controllable.
|
|
80
|
+
3. Keep the public contract and observable behavior unchanged.
|
|
81
|
+
4. Record the production patch and behavior-preservation evidence.
|
|
38
82
|
|
|
39
|
-
|
|
83
|
+
If a public production contract must change, do not proceed autonomously; return `TEST_BLOCKED`.
|
|
40
84
|
|
|
41
|
-
|
|
85
|
+
## Valid Failures
|
|
42
86
|
|
|
43
|
-
|
|
44
|
-
- OpenAPI request schemas and response schemas,
|
|
45
|
-
- handler binding tags and request parsing behavior,
|
|
46
|
-
- service validation logic,
|
|
47
|
-
- repository/service error handling,
|
|
48
|
-
- Prizm L2 TRAPS and DECISIONS,
|
|
49
|
-
- existing tests and source behavior,
|
|
50
|
-
- product rules in project brief/specs.
|
|
87
|
+
Distinguish:
|
|
51
88
|
|
|
52
|
-
|
|
53
|
-
-
|
|
54
|
-
-
|
|
55
|
-
- `POST /stats/trends`: invalid type, invalid period, invalid date range, empty data, each supported trend type, inclusive end date behavior.
|
|
56
|
-
- `POST /upload/sign-url`: unsupported content type, file_count 0/default/max/max+1, scene handling, storage provider failure.
|
|
89
|
+
- **Test construction defect:** wrong import, syntax, fixture, mock contract, or runner usage. Fix the test/infrastructure within bounded attempts.
|
|
90
|
+
- **Reliable implementation/contract failure:** valid test against resolved truth repeatedly fails. Preserve the failure and return `TEST_FAIL`; do not repair business behavior here.
|
|
91
|
+
- **Uncertain/flaky/unavailable:** truth unresolved, required test unreliable, infrastructure unavailable, or cleanup fails. Return `TEST_BLOCKED`.
|
|
57
92
|
|
|
58
|
-
|
|
93
|
+
Never retry until green or silently weaken an assertion/scope.
|
|
59
94
|
|
|
60
|
-
|
|
61
|
-
- Read the source file to understand function signature, branch logic, dependencies, state changes, side effects, and failure paths.
|
|
62
|
-
- Classify the function/class method by service responsibility using names, module path, parameters, dependencies, and related Prizm docs/spec acceptance criteria.
|
|
63
|
-
- Read `references/service-boundary-test-catalog.md` when the unit is service-like or has business-specific inputs — it maps service responsibilities to boundary cases such as auth/session, payment/wallet, repository, pagination, file storage, notification, queue, cache/idempotency, API client, and date/expiration behavior.
|
|
64
|
-
- Build a Unit Test Case Matrix before writing tests. Include generic boundaries plus every applicable service-specific boundary; mark non-applicable categories as `N/A` with a short reason instead of silently skipping them.
|
|
65
|
-
- Generate test file matching project conventions (framework, naming, directory, import style, mock/fixture patterns)
|
|
66
|
-
- Common naming patterns to match:
|
|
67
|
-
- `src/foo.ts` → `src/__tests__/foo.test.ts` or `src/foo.spec.ts` or `tests/foo_test.py`
|
|
68
|
-
- Mirror the existing pattern; if no tests exist, use the framework's default convention
|
|
69
|
-
- Cover: happy path, service-specific boundary cases, edge cases (null/undefined/empty), error conditions, branch paths, boundary values, state/side effects, and dependency failures.
|
|
70
|
-
- Verify mocks for dependency method calls, call count/order/arguments when the behavior depends on collaborators.
|
|
71
|
-
- Do NOT test framework internals or third-party library behavior
|
|
72
|
-
- Run tests immediately after generating
|
|
95
|
+
## Execution and Differential Proof
|
|
73
96
|
|
|
74
|
-
|
|
97
|
+
Run selected tests in order through the bundled evidence builder: focused, module/component, contract/integration, affected-module regression, Regression Ring. The model writes schema-shaped requests; the builder performs the process and appends immutable runner-generated receipts to `executions.json`.
|
|
75
98
|
|
|
76
|
-
For each
|
|
77
|
-
- Generate tests for the module's primary interface exercising its real dependencies.
|
|
78
|
-
- If API endpoints exist, generate request/response tests for every applicable boundary category in the Boundary Matrix, not only valid input. Cover auth, missing params, invalid formats/enums, ownership/not-found, state transitions, and response envelope or schema.
|
|
79
|
-
- If database operations exist, generate CRUD tests using the project's existing test database config when available. Cover transactions, duplicate keys, soft-delete visibility, rollback, and query boundary behavior when applicable.
|
|
80
|
-
- If a dependency failure is not practical at integration level, mark it `N/A` with the reason and cover it in a unit test where mocks are available.
|
|
81
|
-
- Run tests immediately after generating.
|
|
99
|
+
For each new/changed necessary behavior test:
|
|
82
100
|
|
|
83
|
-
|
|
101
|
+
1. In an isolated checkout/equivalent, run it against the actual baseline commit and require failure for the intended recorded signal, then current success.
|
|
102
|
+
2. If baseline proof is genuinely inapplicable, apply one model-selected minimal controlled mutation tied to the risk, require failure, restore it, and verify request/receipt, mutation apply/restore, isolation tree, and original project tree hashes.
|
|
103
|
+
3. Record `PROVEN`, structured `NOT_APPLICABLE`, or `UNPROVEN` with runner receipt IDs.
|
|
84
104
|
|
|
85
|
-
|
|
86
|
-
- Playwright is available (`@playwright/test` or `playwright` in package.json dependencies, or `npx playwright --version` succeeds as fallback)
|
|
87
|
-
- Playwright browsers are installed (check `npx playwright install --dry-run 2>/dev/null` exits cleanly, or `node_modules/.cache/ms-playwright/` directory exists; if missing, warn in report and skip E2E)
|
|
88
|
-
- Project has a UI layer
|
|
89
|
-
- Project can be started (start/dev script in package.json)
|
|
90
|
-
- Acceptance criteria exist in spec.md files
|
|
105
|
+
Necessary `UNPROVEN` evidence blocks. Store complete stdout/stderr in `raw/`; snapshots in `generated-tests/`; do not overwrite previous attempts.
|
|
91
106
|
|
|
92
|
-
|
|
93
|
-
- **Before starting dev server**: detect whether it's already running: (1) check `package.json` scripts for port flags (`--port`, `-p`, `PORT=`), (2) fall back to framework defaults (3000 for React/Next/Express, 5173 for Vite, 8000 for Django, 5000 for Flask), (3) if still unknown, ask the user. Use the detected port in `lsof -i :<port>` to check. If running, tell user: "Dev server appears to be already running on port {N}. Use this running instance for E2E tests?" If user confirms, use existing instance. If user declines, skip E2E.
|
|
94
|
-
- Start the project dev server (only if not already running). Wait for it to be ready.
|
|
95
|
-
- For each in-scope uncovered acceptance criterion, generate a Playwright test script.
|
|
96
|
-
- Include boundary user journeys when acceptance criteria mention validation, permissions, empty states, error messages, or lifecycle transitions.
|
|
97
|
-
- Run generated E2E tests, capture screenshots of failure.
|
|
98
|
-
- **Only stop dev server if you started it.** Never stop a server that was already running.
|
|
107
|
+
## Completion Check
|
|
99
108
|
|
|
100
|
-
|
|
101
|
-
- Note what was skipped and why in the report
|
|
102
|
-
- If only the server wasn't startable: still generate E2E script files but mark as "not run — manual execution needed"
|
|
109
|
+
Before packaging:
|
|
103
110
|
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
-
|
|
108
|
-
-
|
|
109
|
-
-
|
|
111
|
+
- no applicable matrix cell lacks test/execution mapping;
|
|
112
|
+
- all required planned layers have reliable selected executions in order;
|
|
113
|
+
- affected-module and Regression Ring regressions are complete;
|
|
114
|
+
- every new/changed necessary behavior has valid differential proof;
|
|
115
|
+
- every isolated resource/mutation is cleaned up;
|
|
116
|
+
- no verdict-capable unresolved edge remains for pass;
|
|
117
|
+
- budgeted/staged work is complete, otherwise `TEST_BLOCKED`.
|
|
@@ -1,111 +1,90 @@
|
|
|
1
|
-
# Test Report
|
|
1
|
+
# Derived Test Evidence Report Template
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
## test-report.md Format
|
|
3
|
+
Generate this report solely from the structured, hashed evidence package. It is not independently editable authority. Any change to this file or another linked artifact without manifest regeneration causes deterministic validation failure.
|
|
6
4
|
|
|
7
5
|
```markdown
|
|
8
|
-
# Test Report
|
|
6
|
+
# Test Evidence Report
|
|
9
7
|
|
|
10
|
-
|
|
11
|
-
- Scope: {full / module: name / feature: ###-name / this-change}
|
|
12
|
-
- Architecture: {frontend / backend / fullstack}
|
|
13
|
-
- Test framework: {name}
|
|
14
|
-
- Mode: {interactive / headless}
|
|
8
|
+
Derived from structured evidence. Authoritative package: `{evidence_dir}`
|
|
15
9
|
|
|
16
|
-
##
|
|
17
|
-
-
|
|
18
|
-
-
|
|
19
|
-
-
|
|
20
|
-
-
|
|
21
|
-
-
|
|
22
|
-
-
|
|
23
|
-
|
|
24
|
-
- In-Scope Test Files:
|
|
25
|
-
- {...}
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
## Existing Tests Run
|
|
29
|
-
- Total: {N} | Passed: {N} | Failed: {N}
|
|
30
|
-
{failed test details or "All passing"}
|
|
31
|
-
{or "No existing tests found — generated first batch"}
|
|
32
|
-
|
|
33
|
-
## Coverage Gaps Found
|
|
34
|
-
| Module | Interface | Type | Status |
|
|
35
|
-
|--------|-----------|------|--------|
|
|
36
|
-
| ... | ... | unit/integration/e2e | covered / generated / needs-review / unresolved / skipped / boundary-missing |
|
|
37
|
-
|
|
38
|
-
## Boundary Matrix
|
|
39
|
-
| Module | Interface | Service Type | Happy Path | Request Validation | Auth / Permission / Ownership | Domain Invariants | Collection / Pagination | Date / Time | State Transitions | Dependency Failure | Response Contract | N/A Reasons | Final Status |
|
|
40
|
-
|--------|-----------|--------------|------------|--------------------|-------------------------------|-------------------|-------------------------|-------------|-------------------|-------------------|-------------------|-------------|--------------|
|
|
41
|
-
| ... | ... | auth/repository/upload/etc. | covered/generated/skipped | covered/generated/needs-review/skipped/boundary-missing | ... | ... | ... | ... | ... | ... | ... | category — reason | boundary-covered / generated-needs-review / skipped-with-reason / boundary-missing |
|
|
42
|
-
|
|
43
|
-
## Boundary Completion Gate
|
|
44
|
-
- Interfaces total: {N}
|
|
45
|
-
- Boundary-covered: {N}
|
|
46
|
-
- Generated-needs-review: {N}
|
|
47
|
-
- Skipped-with-reason: {N}
|
|
48
|
-
- Boundary-missing: {N}
|
|
49
|
-
- Happy-path-only: {N}
|
|
50
|
-
- Completion gate passed: {yes / no}
|
|
51
|
-
|
|
52
|
-
If `Boundary-missing > 0` or `Happy-path-only > 0`, write:
|
|
53
|
-
`Tests may pass, but boundary coverage is incomplete.`
|
|
54
|
-
|
|
55
|
-
## Boundary Coverage Notes
|
|
56
|
-
| Module | Interface | Boundary Categories Covered | Missing Applicable Boundaries | N/A Categories with Reasons |
|
|
57
|
-
|--------|-----------|-----------------------------|-------------------------------|-----------------------------|
|
|
58
|
-
| ... | ... | request validation, ownership, dependency failure | invalid timestamp, repo failure | pagination — endpoint has no list semantics |
|
|
59
|
-
|
|
60
|
-
## Generated / Updated Tests
|
|
61
|
-
- Unit: {N} generated, {N} needs-review, {N} unresolved
|
|
62
|
-
- Integration: {N} generated, {N} needs-review, {N} unresolved
|
|
63
|
-
- E2E: {N} generated, {N} skipped ({reason})
|
|
64
|
-
|
|
65
|
-
## Boundary Validation
|
|
66
|
-
- Validator command: `{command or "not run — reason"}`
|
|
67
|
-
- Validator result: {passed / failed / skipped}
|
|
68
|
-
- Validator output: {summary or path to raw output}
|
|
69
|
-
|
|
70
|
-
## In-Scope Failures
|
|
71
|
-
- {failures caused by this-change scope, or "None"}
|
|
72
|
-
|
|
73
|
-
## Baseline Failures
|
|
74
|
-
- {pre-existing failures, or "None"}
|
|
75
|
-
|
|
76
|
-
## Out of Scope Gaps
|
|
77
|
-
- {unrelated historical gaps; informational only for scope=this-change}
|
|
10
|
+
## Evidence
|
|
11
|
+
- Evidence ID: {64-character evidence_id}
|
|
12
|
+
- Protocol Version: {protocol_version}
|
|
13
|
+
- Classification: {behavior / lightweight}
|
|
14
|
+
- Baseline Commit: {baseline_commit}
|
|
15
|
+
- Working Diff SHA-256: {working_diff_sha256}
|
|
16
|
+
- Sensitivity: sensitivity=project-controlled
|
|
17
|
+
- Environment Claim: mocked-code-level-only
|
|
78
18
|
|
|
79
|
-
|
|
80
|
-
PASS | NEEDS_FIXES | BLOCKED
|
|
81
|
-
|
|
82
|
-
## Final Status
|
|
83
|
-
- All tests passing: {yes / no}
|
|
84
|
-
- Boundary completion gate passed: {yes / no}
|
|
85
|
-
- Ready for commit: {yes / no — include reason}
|
|
86
|
-
- Needs human review: {list of tests/interfaces marked needs-review}
|
|
87
|
-
{remaining failures if any}
|
|
88
|
-
```
|
|
19
|
+
The evidence package preserves complete command, environment, stdout, stderr, patch, hash, and execution-history values without automatic redaction. The project owns access control, retention, and upload policy. Production credentials, production services, and destructive real-data operations are prohibited.
|
|
89
20
|
|
|
90
|
-
|
|
21
|
+
Mocked code-level evidence does not verify a real deployed environment.
|
|
91
22
|
|
|
92
|
-
|
|
93
|
-
-
|
|
94
|
-
-
|
|
95
|
-
-
|
|
96
|
-
-
|
|
23
|
+
## Scope
|
|
24
|
+
- Affected Module: {name} ({explicit / cohesion-derived})
|
|
25
|
+
- Primary Scope: {count and pointers}
|
|
26
|
+
- Regression Ring: {count and pointers}
|
|
27
|
+
- Verdict-capable Unresolved Edges: {count}
|
|
28
|
+
|
|
29
|
+
## Behavior-Risk Completeness
|
|
30
|
+
- Observable Behaviors: {count}
|
|
31
|
+
- Applicable Risk Cells: {count}
|
|
32
|
+
- Proven/Mapped Risk Cells: {count}
|
|
33
|
+
- Unresolved Risk Cells: {count}
|
|
34
|
+
- Diagnostic Coverage Signals: {values or not collected; never the completion basis}
|
|
35
|
+
- Matrix: `behavior-risk-matrix.json`
|
|
36
|
+
|
|
37
|
+
## Risk-Adaptive Test Plan
|
|
38
|
+
| Layer | Required | Rationale | Tests | Selected Execution IDs |
|
|
39
|
+
|-------|----------|-----------|-------|------------------------|
|
|
40
|
+
| focused | {yes/no} | {...} | {...} | {...} |
|
|
41
|
+
| module-component | {yes/no} | {...} | {...} | {...} |
|
|
42
|
+
| contract-integration | {yes/no} | {...} | {...} | {...} |
|
|
43
|
+
| affected-module-regression | {yes/no} | {...} | {...} | {...} |
|
|
44
|
+
| regression-ring | {yes/no} | {...} | {...} | {...} |
|
|
45
|
+
|
|
46
|
+
## Infrastructure and Contracts
|
|
47
|
+
- Native Runner/Conventions: {...}
|
|
48
|
+
- Infrastructure Changes: `infrastructure-changes.json`
|
|
49
|
+
- Contract Snapshots: `contracts/`
|
|
50
|
+
- Production Resources Used: no
|
|
51
|
+
- Cleanup Succeeded: {yes/no}
|
|
52
|
+
|
|
53
|
+
## Differential Proof
|
|
54
|
+
- PROVEN: {count}
|
|
55
|
+
- NOT_APPLICABLE: {count with rationale pointers}
|
|
56
|
+
- UNPROVEN: {count}
|
|
57
|
+
- Proof Record: `differential-proof.json`
|
|
58
|
+
|
|
59
|
+
## Executions
|
|
60
|
+
- Attempts Preserved: {count}
|
|
61
|
+
- Selected Required Executions: {IDs in order}
|
|
62
|
+
- Complete History: `executions.json`
|
|
63
|
+
- Raw Outputs: `raw/`
|
|
64
|
+
|
|
65
|
+
## Deterministic Validation
|
|
66
|
+
- Command: `python3 ${SKILL_DIR}/scripts/validate_test_evidence.py --evidence-dir {evidence_dir} --project-root {target_project_root}`
|
|
67
|
+
- Result: {passed / failed}
|
|
68
|
+
- Validation Record: `validation.json`
|
|
97
69
|
|
|
98
|
-
##
|
|
70
|
+
## Verdict
|
|
71
|
+
Verdict: TEST_PASS | TEST_FAIL | TEST_BLOCKED
|
|
99
72
|
|
|
100
|
-
|
|
73
|
+
## Responsibility Boundary
|
|
74
|
+
- Testing-domain verdict only: yes
|
|
75
|
+
- Overall code-quality or broad Spec verdict: no
|
|
76
|
+
- Business defects repaired by this skill: no
|
|
77
|
+
- Commit/release authorized: no
|
|
78
|
+
- Real deployed environment validated: no
|
|
101
79
|
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
| yes | yes | yes | no — human review required |
|
|
106
|
-
| yes | no | any | no — boundary coverage incomplete |
|
|
107
|
-
| no | any | any | no — tests failing |
|
|
80
|
+
## Compatibility
|
|
81
|
+
The legacy test-report interface is not supported. Existing Pipeline or other Skill consumers were not migrated and may be temporarily incompatible after installation.
|
|
82
|
+
```
|
|
108
83
|
|
|
109
|
-
|
|
84
|
+
## Verdict Rendering Rules
|
|
110
85
|
|
|
111
|
-
|
|
86
|
+
- Render `TEST_PASS` only after deterministic validation passes and all required evidence is proven.
|
|
87
|
+
- Render `TEST_FAIL` only with reliable reproduced failure execution pointers.
|
|
88
|
+
- Render `TEST_BLOCKED` with explicit blocker pointers.
|
|
89
|
+
- Never render `PASS`, `NEEDS_FIXES`, conditional pass, “ready for commit,” or release authorization.
|
|
90
|
+
- A lightweight `TEST_PASS` must say it proves only the deterministic lightweight claim, not runtime behavior.
|
|
@@ -0,0 +1,97 @@
|
|
|
1
|
+
# Trusted Evidence Execution
|
|
2
|
+
|
|
3
|
+
Load this reference before `SCOPE_DISCOVER`, `EXECUTE_PROVE`, resume, or replay. It defines which choices remain with the model and which evidence mechanics are fixed.
|
|
4
|
+
|
|
5
|
+
## Two Parameter Classes
|
|
6
|
+
|
|
7
|
+
### Model-Chosen Project Parameters
|
|
8
|
+
|
|
9
|
+
Inspect the target project and choose these semantically:
|
|
10
|
+
|
|
11
|
+
- native test framework, runner, command, flags, and working directory;
|
|
12
|
+
- timeout, attempt/retry policy, concurrency, resource limits, and isolation strategy;
|
|
13
|
+
- unit/module/contract/integration/code-level E2E layers;
|
|
14
|
+
- fixture, fake, Mock, service-double, browser, database, and mutation technology;
|
|
15
|
+
- module roots, public behavior boundary, inventory patterns, exclusions, Regression Ring, and differential method;
|
|
16
|
+
- environment classification and endpoint allow/deny evidence.
|
|
17
|
+
|
|
18
|
+
The protocol does not prescribe Jest, pytest, a timeout number, a retry count, coverage threshold, runner flag, or mutation package. Record the reason for each choice in request and planning records.
|
|
19
|
+
|
|
20
|
+
### Protocol-Fixed Locators and Mechanisms
|
|
21
|
+
|
|
22
|
+
The protocol fixes only what trustworthy, replayable packaging needs:
|
|
23
|
+
|
|
24
|
+
- locator arguments: project root, evidence directory, request/manifest/inventory/receipt path;
|
|
25
|
+
- schema-shaped request records;
|
|
26
|
+
- project-root and evidence-root path confinement;
|
|
27
|
+
- actual child-process execution without a shell;
|
|
28
|
+
- complete stdout/stderr capture and output hashes;
|
|
29
|
+
- complete effective environment and tool-probe capture;
|
|
30
|
+
- runner-generated UUID receipt, request hash, runner hash, and receipt-chain hash;
|
|
31
|
+
- append-only execution history;
|
|
32
|
+
- differential isolation, cleanup, and tree hashes;
|
|
33
|
+
- deterministic resume invalidation calculation;
|
|
34
|
+
- strict schema and cross-record validation.
|
|
35
|
+
|
|
36
|
+
## Trust Boundary
|
|
37
|
+
|
|
38
|
+
A receipt proves that the bundled runner produced a record with the captured inputs and outputs during this invocation. The runner snapshots its own executable bytes inside the evidence package so later validation does not silently depend on whichever PrizmKit version happens to be installed. Hashes bind bytes together and detect later drift. They do not create cryptographic non-repudiation and cannot prove that a same-OS-user, same-permission producer did not replace the runner, requests, repository, or evidence package before validation.
|
|
39
|
+
|
|
40
|
+
Treat evidence as auditable project-controlled provenance, not a defense against a malicious same-privilege actor. The validator checks consistency, obvious conflicts, and replayability; it cannot mathematically prove that the model found every business behavior.
|
|
41
|
+
|
|
42
|
+
## Builder Commands
|
|
43
|
+
|
|
44
|
+
Write requests beneath the evidence directory, then invoke the bundled script:
|
|
45
|
+
|
|
46
|
+
```bash
|
|
47
|
+
python3 ${SKILL_DIR}/scripts/build_test_evidence.py \
|
|
48
|
+
--project-root <target-project-root> \
|
|
49
|
+
--evidence-dir <evidence-dir> \
|
|
50
|
+
inventory --request requests/inventory.json
|
|
51
|
+
|
|
52
|
+
python3 ${SKILL_DIR}/scripts/build_test_evidence.py \
|
|
53
|
+
--project-root <target-project-root> \
|
|
54
|
+
--evidence-dir <evidence-dir> \
|
|
55
|
+
execute --request requests/focused.json
|
|
56
|
+
|
|
57
|
+
python3 ${SKILL_DIR}/scripts/build_test_evidence.py \
|
|
58
|
+
--project-root <target-project-root> \
|
|
59
|
+
--evidence-dir <evidence-dir> \
|
|
60
|
+
differential --request requests/differential.json
|
|
61
|
+
|
|
62
|
+
python3 ${SKILL_DIR}/scripts/build_test_evidence.py \
|
|
63
|
+
--project-root <target-project-root> \
|
|
64
|
+
--evidence-dir <evidence-dir> \
|
|
65
|
+
resume --manifest manifest.json --inventory target-inventory.json
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
The subcommands cover fixed mechanics only:
|
|
69
|
+
|
|
70
|
+
- `inventory`: expand model-selected project-relative patterns, independently enumerate all actual files under each declared module root, reject files that are neither categorized nor structurally excluded, hash actual files, and bind discovery inputs;
|
|
71
|
+
- `execute`: run the declared argv in the declared safe cwd and append a runner-generated receipt;
|
|
72
|
+
- `differential`: execute baseline or controlled-mutation proof in isolated copies and verify cleanup/tree binding;
|
|
73
|
+
- `resume`: compare hashes and stage outputs to compute an invalidation decision while retaining `semantic_review_required=true`.
|
|
74
|
+
|
|
75
|
+
## Replay
|
|
76
|
+
|
|
77
|
+
Replay a prior receipt through the runner rather than treating attestation as historical proof:
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
python3 ${SKILL_DIR}/scripts/build_test_evidence.py \
|
|
81
|
+
--project-root <target-project-root> \
|
|
82
|
+
--evidence-dir <evidence-dir> \
|
|
83
|
+
execute --replay-receipt receipts/<receipt>.json
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
The recorded request hash must still match. Replay emits a new receipt linked with `replay_of`; it never overwrites the historical receipt. A replay can still differ because project state, installed tools, clocks, external test services, or environment values changed. Report those differences instead of claiming historical equivalence.
|
|
87
|
+
|
|
88
|
+
## External Target Safety
|
|
89
|
+
|
|
90
|
+
Do not block merely because a string contains `production`; names can be misleading. Classify every external target using endpoint/configuration evidence:
|
|
91
|
+
|
|
92
|
+
- block `production` and `unknown` external targets;
|
|
93
|
+
- permit only model-classified isolated/test/staging targets with explicit allow evidence and no deny evidence;
|
|
94
|
+
- preserve the complete evidence used for the decision;
|
|
95
|
+
- keep local fakes and in-process contract mocks classified `local` with `external=false`.
|
|
96
|
+
|
|
97
|
+
The model owns the semantic classification; the runner enforces the declared production/unknown safety invariant and rejects contradictory allow/deny records.
|