prizmkit 1.1.140 → 1.1.142
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bundled/VERSION.json +3 -3
- package/bundled/dev-pipeline/README.md +1 -1
- package/bundled/dev-pipeline/prizmkit_runtime/checkpoint_state.py +165 -125
- package/bundled/dev-pipeline/prizmkit_runtime/runner_classification.py +13 -1
- package/bundled/dev-pipeline/prizmkit_runtime/runner_models.py +0 -6
- package/bundled/dev-pipeline/prizmkit_runtime/runners.py +5 -3
- package/bundled/dev-pipeline/prizmkit_runtime/test_result.py +160 -0
- package/bundled/dev-pipeline/scripts/check-session-status.py +11 -1
- package/bundled/dev-pipeline/scripts/generate-bootstrap-prompt.py +23 -806
- package/bundled/dev-pipeline/scripts/generate-bugfix-prompt.py +6 -8
- package/bundled/dev-pipeline/scripts/generate-recovery-prompt.py +1 -1
- package/bundled/dev-pipeline/scripts/generate-refactor-prompt.py +6 -8
- package/bundled/dev-pipeline/scripts/prompt_framework.py +68 -56
- package/bundled/dev-pipeline/scripts/update-bug-status.py +21 -2
- package/bundled/dev-pipeline/scripts/update-checkpoint.py +52 -131
- package/bundled/dev-pipeline/scripts/update-feature-status.py +21 -2
- package/bundled/dev-pipeline/scripts/update-refactor-status.py +21 -2
- package/bundled/dev-pipeline/templates/bootstrap-prompt.md +3 -3
- package/bundled/dev-pipeline/templates/bootstrap-tier3.md +8 -19
- package/bundled/dev-pipeline/templates/bugfix-bootstrap-prompt.md +6 -6
- package/bundled/dev-pipeline/templates/refactor-bootstrap-prompt.md +5 -5
- package/bundled/dev-pipeline/templates/sections/bugfix-phase-commit-report.md +1 -1
- package/bundled/dev-pipeline/templates/sections/bugfix-phase-diagnose-plan.md +1 -1
- package/bundled/dev-pipeline/templates/sections/bugfix-phase-manual-verification.md +1 -1
- package/bundled/dev-pipeline/templates/sections/bugfix-phase-review.md +1 -1
- package/bundled/dev-pipeline/templates/sections/bugfix-phase-test.md +13 -10
- package/bundled/dev-pipeline/templates/sections/bugfix-reminders.md +1 -1
- package/bundled/dev-pipeline/templates/sections/checkpoint-system.md +18 -9
- package/bundled/dev-pipeline/templates/sections/headless-commit-authorization.md +1 -1
- package/bundled/dev-pipeline/templates/sections/phase-commit-full.md +1 -1
- package/bundled/dev-pipeline/templates/sections/phase-commit.md +1 -1
- package/bundled/dev-pipeline/templates/sections/phase-context-snapshot-agent-suffix.md +2 -2
- package/bundled/dev-pipeline/templates/sections/phase-context-snapshot-base.md +1 -1
- package/bundled/dev-pipeline/templates/sections/phase-context-snapshot-lite-suffix.md +1 -1
- package/bundled/dev-pipeline/templates/sections/phase-implement-lite.md +1 -1
- package/bundled/dev-pipeline/templates/sections/phase-prizmkit-test.md +14 -26
- package/bundled/dev-pipeline/templates/sections/phase-review-full.md +2 -2
- package/bundled/dev-pipeline/templates/sections/phase-specify-plan-full.md +3 -3
- package/bundled/dev-pipeline/templates/sections/refactor-phase-review.md +1 -1
- package/bundled/dev-pipeline/templates/sections/refactor-phase-test.md +13 -10
- package/bundled/dev-pipeline/templates/sections/refactor-reminders.md +1 -1
- package/bundled/dev-pipeline/templates/session-status-schema.json +1 -1
- package/bundled/dev-pipeline/tests/test_checkpoint_state.py +494 -413
- package/bundled/dev-pipeline/tests/test_generate_bootstrap_prompt.py +144 -684
- package/bundled/dev-pipeline/tests/test_generate_bugfix_prompt.py +20 -41
- package/bundled/dev-pipeline/tests/test_generate_refactor_prompt.py +24 -24
- package/bundled/dev-pipeline/tests/test_python_runner_parity.py +176 -36
- package/bundled/skills/_metadata.json +6 -6
- package/bundled/skills/prizmkit/SKILL.md +13 -18
- package/bundled/skills/prizmkit/references/workflow-state-protocol.md +73 -80
- package/bundled/skills/prizmkit-code-review/SKILL.md +7 -7
- package/bundled/skills/prizmkit-code-review/references/workflow-state-protocol.md +73 -80
- package/bundled/skills/prizmkit-committer/SKILL.md +8 -7
- package/bundled/skills/prizmkit-committer/references/workflow-state-protocol.md +73 -80
- package/bundled/skills/prizmkit-implement/SKILL.md +3 -2
- package/bundled/skills/prizmkit-implement/references/workflow-state-protocol.md +73 -80
- package/bundled/skills/prizmkit-plan/SKILL.md +2 -1
- package/bundled/skills/prizmkit-plan/references/workflow-state-protocol.md +73 -80
- package/bundled/skills/prizmkit-retrospective/SKILL.md +8 -6
- package/bundled/skills/prizmkit-retrospective/references/workflow-state-protocol.md +73 -80
- package/bundled/skills/prizmkit-test/SKILL.md +217 -189
- package/bundled/skills/prizmkit-test/references/external-contract-mock-guidance.md +119 -0
- package/bundled/skills/prizmkit-test/references/independent-test-review.md +269 -0
- package/bundled/skills/prizmkit-test/references/test-coverage-model.md +193 -0
- package/bundled/skills/prizmkit-test/references/test-report-template.md +164 -79
- package/bundled/skills/prizmkit-workflow/SKILL.md +28 -36
- package/bundled/skills/prizmkit-workflow/references/workflow-state-protocol.md +73 -80
- package/package.json +1 -1
- package/bundled/skills/prizmkit-test/assets/authoritative-records.schema.json +0 -477
- package/bundled/skills/prizmkit-test/assets/behavior-risk-matrix.schema.json +0 -116
- package/bundled/skills/prizmkit-test/assets/evidence-manifest.schema.json +0 -111
- package/bundled/skills/prizmkit-test/assets/evidence-package-template.json +0 -66
- package/bundled/skills/prizmkit-test/references/boundary-coverage-protocol.md +0 -104
- package/bundled/skills/prizmkit-test/references/contract-mock-protocol.md +0 -65
- package/bundled/skills/prizmkit-test/references/evidence-protocol.md +0 -208
- package/bundled/skills/prizmkit-test/references/evidence-request-protocol.md +0 -80
- package/bundled/skills/prizmkit-test/references/examples.md +0 -100
- package/bundled/skills/prizmkit-test/references/service-boundary-test-catalog.md +0 -228
- package/bundled/skills/prizmkit-test/references/test-generation-steps.md +0 -118
- package/bundled/skills/prizmkit-test/references/trusted-evidence-execution.md +0 -107
- package/bundled/skills/prizmkit-test/references/workflow-state-protocol.md +0 -181
- package/bundled/skills/prizmkit-test/scripts/build_test_evidence.py +0 -1930
- package/bundled/skills/prizmkit-test/scripts/validate_test_evidence.py +0 -1847
|
@@ -1,285 +1,313 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: "prizmkit-test"
|
|
3
|
-
description: "
|
|
3
|
+
description: "AI-led project-native testing for the complete affected business module and its Regression Ring. Adds comprehensive behavior, boundary, error, state, side-effect, permission, concurrency, idempotency, time, dependency, and consumer tests; repairs valid failures in a bounded loop; performs mandatory Main-Agent review plus one optional strict independent Test Reviewer; and returns TEST_PASS, TEST_NEEDS_FIXES, or TEST_BLOCKED with a human report and minimal terminal result. (project)"
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# PrizmKit Test
|
|
7
7
|
|
|
8
|
-
`/prizmkit-test`
|
|
8
|
+
`/prizmkit-test` is the independent testing stage for the current workspace. It understands the affected business behavior, fills project-native test gaps, executes the tests, repairs valid failures, reviews the final tests, and reports a truthful testing-domain result.
|
|
9
9
|
|
|
10
|
-
|
|
10
|
+
It does not build an evidence package, prove historical TDD order, validate a previous checkout, or manage pipeline/session state.
|
|
11
11
|
|
|
12
|
-
##
|
|
12
|
+
## Atomic Skill Boundary
|
|
13
13
|
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
Fix only the protocol mechanisms needed for safe, replayable evidence: project/evidence/request locators, schema-shaped requests, path confinement, real process execution, complete raw capture, runner-generated receipts, append-only history, and resume invalidation. Read `${SKILL_DIR}/references/trusted-evidence-execution.md` before inventory, execution, resume, or replay; it defines the trust boundary and builder commands. Read `${SKILL_DIR}/references/evidence-request-protocol.md` when authoring any request or authoritative JSON record; it defines request ownership, structured N/A, and the schema map.
|
|
17
|
-
|
|
18
|
-
## Canonical Evidence Builder Lifecycle
|
|
19
|
-
|
|
20
|
-
The Main Agent supplies semantic requests only; it does not author authoritative package records. Initialize and drive one evidence directory through the builder in this order:
|
|
14
|
+
The Main Agent owns this invocation from analysis through final report:
|
|
21
15
|
|
|
22
16
|
```text
|
|
23
|
-
|
|
17
|
+
resolve current change and requirement context
|
|
18
|
+
→ discover affected module and Regression Ring
|
|
19
|
+
→ model observable behavior and applicable risks
|
|
20
|
+
→ inspect and add project-native tests
|
|
21
|
+
→ execute and repair valid failures: maximum three rounds
|
|
22
|
+
→ mandatory Main-Agent test review: maximum ten completed rounds
|
|
23
|
+
→ optional independent Test Reviewer: one unit, maximum five responses
|
|
24
|
+
→ final native regression on the exact final state
|
|
25
|
+
→ write test-report.md and test-result.json
|
|
26
|
+
→ return TEST_PASS | TEST_NEEDS_FIXES | TEST_BLOCKED
|
|
24
27
|
```
|
|
25
28
|
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
- `init`: creates the package directories and pins the baseline identity.
|
|
29
|
-
- `capture-change`: captures the complete canonical change, including untracked, deleted, renamed, and copied files; never substitute plain `git diff`. `inventory` then requires its changed-file set to match the capture exactly and rejects stale captures or unaccounted live paths.
|
|
30
|
-
- `inventory`: hashes current source/tests/contracts/lockfiles, rejects category overlap, excludes runtime-only paths, and validates directory roots plus the complete capture-derived changed-file set.
|
|
31
|
-
- `prepare-tests`: discovers repository-native test files from the canonical capture and inventory, generates stable builder-owned test IDs, derives existing/added/modified status, and snapshots only added/modified live test sources.
|
|
32
|
-
- `execute`: runs native project commands and owns `executions.json`, `receipts/`, and `raw/`. Each request uses exactly one binding mode: `test_ids`, `test_paths`, or `auto_bind=true`. The builder infers recognized package-wide or explicit file/directory/glob command scope, blocks opaque commands or mismatched declarations, and persists verified canonical `test_ids` only.
|
|
33
|
-
- `finalize`: derives builder-owned target hashes, runs semantic-record and required-layer checks, rechecks canonical capture freshness, blocks every selected unreliable execution, and owns the payload manifest, verdict, generated-test snapshots, and evidence identity. It leaves the package in a mutable `payload-finalized` state until validator attestation; malformed semantic records fail before package relocation.
|
|
34
|
-
- `render-report`: recovery/debugging command that derives `test-report.md`; normal finalization and attestation render it automatically, and it is never authoritative.
|
|
35
|
-
- `resume`: preserves prior receipts and refuses to rebind an identity-changing package; create a new package after scope/source identity changes.
|
|
29
|
+
After writing its result, this skill returns control to its caller. It must not:
|
|
36
30
|
|
|
37
|
-
|
|
31
|
+
- invoke implementation, Code Review, retrospective, or committer;
|
|
32
|
+
- update a workflow checkpoint or pipeline runtime state;
|
|
33
|
+
- classify an AI CLI session as success, crash, retry, or continuation;
|
|
34
|
+
- decide whether a workflow terminates or resumes;
|
|
35
|
+
- call a testing-domain non-pass result a runtime failure.
|
|
38
36
|
|
|
39
|
-
|
|
37
|
+
## Inputs and Outputs
|
|
40
38
|
|
|
39
|
+
| Parameter | Required | Description |
|
|
40
|
+
|---|---|---|
|
|
41
|
+
| `scope` | No | `full-project`, `module:<name>`, `feature:<slug>`, or `this-change`. Defaults to `this-change` when current change context exists, otherwise `full-project`. |
|
|
42
|
+
| `artifact_dir` | No | Active requirement directory containing available specification, plan, and prior review context. Reuse the caller-provided path. |
|
|
43
|
+
| `changed_files` | No | Explicit current changed paths. Highest-priority scope locator. |
|
|
44
|
+
| `diff_base` | No | Git comparison base when explicit changed paths are absent. |
|
|
45
|
+
| `test_commands` | No | Project-native commands when repository conventions are ambiguous. |
|
|
41
46
|
|
|
42
|
-
|
|
47
|
+
Write exactly two terminal artifacts in `artifact_dir` when it is available; otherwise use a clearly reported caller-visible artifact directory selected from the active request context:
|
|
43
48
|
|
|
44
49
|
```text
|
|
45
|
-
|
|
50
|
+
{artifact_dir}/
|
|
51
|
+
├── test-report.md
|
|
52
|
+
└── test-result.json
|
|
46
53
|
```
|
|
47
54
|
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
## Atomic Stage Boundary
|
|
51
|
-
|
|
52
|
-
`prizmkit-test` owns only auditable test evidence generation, validation, and the testing-domain verdict. It writes the authoritative `TEST_*` result and `next_stage`, then returns control. When an external orchestrator is active, it must not invoke retrospective or implementation itself. The active orchestrator owns the next-stage invocation and repair routing.
|
|
53
|
-
|
|
55
|
+
Do not create `.prizmkit/test/evidence/`, manifests, hashes, attestations, package lifecycle records, or test-internal checkpoints.
|
|
54
56
|
|
|
55
|
-
|
|
56
|
-
- runtime configuration, schema, migration, generated-runtime, lockfile, or dependency changes;
|
|
57
|
-
- public interfaces and data contracts;
|
|
58
|
-
- frontend, backend, library, CLI, adapter, and multi-module changes;
|
|
59
|
-
- legacy modules whose observable test coverage is incomplete;
|
|
60
|
-
- explicit test, boundary, evidence, confidence, or audit requests.
|
|
57
|
+
## Context Loading and Scope Resolution
|
|
61
58
|
|
|
62
|
-
|
|
59
|
+
1. Reuse current requirement and workspace context already loaded by the Main Agent.
|
|
60
|
+
2. Load `root.prizm`, the relevant module indexes, and the applicable detail docs before modifying source or tests.
|
|
61
|
+
3. Inspect current staged, unstaged, untracked, deleted, and renamed paths. Changed lines locate impact but never define completeness.
|
|
62
|
+
4. Read the available specification, acceptance criteria, plan, and prior review only as needed to establish expected behavior.
|
|
63
|
+
5. Inspect manifests, runner configuration, CI conventions, existing test assertions, fixtures, fakes, mocks, contracts, and coverage support.
|
|
64
|
+
6. If necessary scope cannot be determined safely, report the precise missing input and return `TEST_BLOCKED`.
|
|
63
65
|
|
|
64
|
-
|
|
66
|
+
Do not infer the active requirement solely from the name of an old artifact directory. Do not use a historical worktree, second checkout, mutation proof, baseline-failure proof, or test overlay.
|
|
65
67
|
|
|
66
|
-
|
|
67
|
-
2. Never use production credentials, production databases, production APIs, production queues/storage, or destructive operations against real data.
|
|
68
|
-
3. Preserve complete command/environment/output values as requested. Mark the package `sensitivity=project-controlled`; the project owns access control, retention, and upload policy.
|
|
69
|
-
4. Treat code-level dependencies as mock-first. Real deployed test-environment validation is a separate authorized activity.
|
|
70
|
-
5. Never delete existing tests.
|
|
71
|
-
6. Evidence-protocol recovery is separately bounded: allow at most two package initializations and two semantic preflight failures for one canonical change identity. After the budget is exhausted, return `TEST_BLOCKED`; do not create another versioned package or hand-edit finalized evidence.
|
|
72
|
-
7. For `TEST_FAIL`, classify and persist an evidence-backed `repair_scope` of `test-infrastructure`, `production`, `runtime`, `schema`, `dependency`, `public-interface`, or `unknown`. Test-infrastructure scope routes to implement then test; production-family scope routes to implement, code-review, then test. If evidence cannot support a safe classification, return `TEST_BLOCKED`.
|
|
68
|
+
## Coverage Model
|
|
73
69
|
|
|
74
|
-
|
|
70
|
+
Read `${SKILL_DIR}/references/test-coverage-model.md` before designing or reviewing tests.
|
|
75
71
|
|
|
76
|
-
|
|
72
|
+
### Affected Business Module
|
|
77
73
|
|
|
78
|
-
|
|
79
|
-
|-----------|----------|-------------|
|
|
80
|
-
| `scope` | No | `full-project`, `module:<name>`, `feature:<slug>`, or `this-change`. Defaults to `this-change` with a valid artifact directory, otherwise `full-project`. |
|
|
81
|
-
| `artifact_dir` | No | Change artifact containing `spec.md` + `plan.md`; used to resolve acceptance conditions and changed scope. |
|
|
82
|
-
| `changed_files` | No | Explicit changed files; highest-priority scope input. |
|
|
83
|
-
| `diff_base` | No | Git baseline used when `changed_files` is absent. |
|
|
84
|
-
| `evidence_target` | No | Existing evidence ID/directory to validate and resume. |
|
|
85
|
-
| `execution_budget` | No | Bounded time/attempt budget. Incomplete necessary work is `TEST_BLOCKED`, never truncated to pass. |
|
|
86
|
-
| `test_commands` | No | Explicit project-native commands when discovery is ambiguous. |
|
|
87
|
-
|
|
88
|
-
For `scope=this-change`, determine changed files from `changed_files`, `diff_base`, current diff, artifact plan/context manifest, then spec mapping. Unknown scope is `TEST_BLOCKED`.
|
|
74
|
+
Prefer an explicit project module. If none exists, derive one cohesive boundary from files that jointly implement the same observable responsibility. Cover every discoverable observable behavior of that module, including relevant legacy behavior whose tests are incomplete.
|
|
89
75
|
|
|
90
|
-
|
|
76
|
+
### Regression Ring
|
|
91
77
|
|
|
92
|
-
|
|
78
|
+
Expand only along concrete coupling evidence:
|
|
93
79
|
|
|
94
|
-
|
|
80
|
+
- direct callers;
|
|
81
|
+
- consumers of values, errors, events, files, or ordering;
|
|
82
|
+
- shared schemas, types, protocols, generated assets, and adapters;
|
|
83
|
+
- persistence, cache, lock, queue, transaction, and other shared state dependencies.
|
|
95
84
|
|
|
96
|
-
|
|
97
|
-
- Load `${SKILL_DIR}/assets/evidence-manifest.schema.json` when creating or validating `manifest.json`.
|
|
98
|
-
- Load `${SKILL_DIR}/assets/behavior-risk-matrix.schema.json` during `CONTRACT_MODEL` or matrix validation.
|
|
99
|
-
- Load `${SKILL_DIR}/assets/authoritative-records.schema.json` when creating or validating any other authoritative request/record.
|
|
100
|
-
- Load `${SKILL_DIR}/assets/evidence-package-template.json` when initializing or resuming an evidence directory.
|
|
101
|
-
- Read `${SKILL_DIR}/references/contract-mock-protocol.md` only when dependency, contract, or external-boundary risks require doubles.
|
|
102
|
-
- Read `${SKILL_DIR}/references/test-generation-steps.md` during `INFRA_READY`, `TEST_BUILD`, or `EXECUTE_PROVE`.
|
|
103
|
-
- Read `${SKILL_DIR}/references/service-boundary-test-catalog.md` only when domain boundary signals need risk discovery.
|
|
104
|
-
- Read `${SKILL_DIR}/references/boundary-coverage-protocol.md` when defining module roots, exclusions, Regression Ring, or behavior-risk completeness.
|
|
85
|
+
Do not perform an unconditional repository-wide sweep.
|
|
105
86
|
|
|
106
|
-
|
|
87
|
+
### Behavior and Risks
|
|
107
88
|
|
|
108
|
-
|
|
89
|
+
Model outside-in from business capability or acceptance criterion to public entry point, rules, invariants, state, side effects, low-level logic, dependencies, and consumers. For each behavior, assess applicable:
|
|
109
90
|
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
91
|
+
```text
|
|
92
|
+
functional | boundary | error | state | side-effect | permission
|
|
93
|
+
concurrency | idempotency | time | dependency | consumer
|
|
94
|
+
```
|
|
113
95
|
|
|
114
|
-
|
|
115
|
-
- `lightweight`: exclusively docs/comments/formatting with deterministic evidence that runtime behavior cannot change.
|
|
96
|
+
A dimension may be omitted only with a concise behavior-specific reason. Coverage percentages are diagnostic signals, never a completion threshold.
|
|
116
97
|
|
|
117
|
-
|
|
98
|
+
### Test Layers
|
|
118
99
|
|
|
119
|
-
|
|
100
|
+
Use the lowest layer that proves a property without losing relevant composition, then add higher layers for composition risk:
|
|
120
101
|
|
|
121
|
-
|
|
102
|
+
```text
|
|
103
|
+
focused/unit
|
|
104
|
+
→ module/component
|
|
105
|
+
→ contract/integration
|
|
106
|
+
→ code-level E2E when applicable
|
|
107
|
+
→ complete affected-module regression
|
|
108
|
+
→ Regression Ring verification
|
|
109
|
+
```
|
|
122
110
|
|
|
123
|
-
-
|
|
124
|
-
- affected module boundary (`explicit` or `cohesion-derived`);
|
|
125
|
-
- Primary Scope containing all observable behavior of that module;
|
|
126
|
-
- Regression Ring containing direct callers, consumers, shared contracts, and state dependencies, each mapped to planned tests;
|
|
127
|
-
- contract, lockfile, test, and module-boundary discovery evidence;
|
|
128
|
-
- Unresolved Edges for dynamic/unprovable coupling.
|
|
111
|
+
Do not mechanically test every private function. Add direct focused tests for critical low-level logic when complexity, boundaries, risk, or failure localization justify them; otherwise prove it through public behavior.
|
|
129
112
|
|
|
130
|
-
|
|
113
|
+
## Test Construction
|
|
131
114
|
|
|
132
|
-
|
|
115
|
+
Read `${SKILL_DIR}/references/test-coverage-model.md` and, when external or cross-module contracts apply, `${SKILL_DIR}/references/external-contract-mock-guidance.md`.
|
|
133
116
|
|
|
134
|
-
|
|
117
|
+
Expected behavior follows this precedence:
|
|
135
118
|
|
|
136
|
-
1.
|
|
137
|
-
2. machine-readable
|
|
138
|
-
3. acceptance
|
|
119
|
+
1. confirmed specification;
|
|
120
|
+
2. machine-readable contract;
|
|
121
|
+
3. acceptance criteria;
|
|
139
122
|
4. trusted existing tests;
|
|
140
|
-
5. callers
|
|
123
|
+
5. callers and consumers;
|
|
141
124
|
6. current implementation.
|
|
142
125
|
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
For every observable behavior, record contract source, preconditions, input classes/boundaries, outputs, side effects, transitions, errors, and functional/boundary/permission/concurrency/idempotency/time/dependency/consumer risks. Every applicable cell maps to tests and executions or remains explicitly unresolved.
|
|
126
|
+
Conflicting higher-precedence truth that cannot be resolved is `TEST_BLOCKED`; do not preserve a possible implementation defect as a characterization test.
|
|
146
127
|
|
|
147
|
-
|
|
128
|
+
Tests must:
|
|
148
129
|
|
|
149
|
-
|
|
130
|
+
- assert observable outputs, errors, state transitions, side effects, ordering, counts, and absence of partial effects;
|
|
131
|
+
- isolate only dependencies irrelevant to the property being proved;
|
|
132
|
+
- follow project-native naming, location, framework, fixture, and helper conventions;
|
|
133
|
+
- control clocks, randomness, concurrency, synchronization, and fixtures deterministically;
|
|
134
|
+
- verify failure cleanup and useful diagnostics;
|
|
135
|
+
- avoid vacuous assertions and assertions that merely duplicate implementation steps;
|
|
136
|
+
- never delete or weaken a valid existing test just to obtain green output.
|
|
150
137
|
|
|
151
|
-
|
|
138
|
+
Reuse an adequate existing framework. Add only the smallest necessary project-native infrastructure when a required property cannot otherwise be expressed.
|
|
152
139
|
|
|
153
|
-
|
|
154
|
-
2. module/component;
|
|
155
|
-
3. contract/integration;
|
|
156
|
-
4. complete affected-module regression;
|
|
157
|
-
5. Regression Ring.
|
|
140
|
+
Minimal behavior-preserving testability seams are allowed, including dependency injection through an existing abstraction, pure-function extraction, an internal adapter, or controllable clock/randomness/state. Classify any production change under the repair boundary below.
|
|
158
141
|
|
|
159
|
-
|
|
142
|
+
## Native Execution and Failure Repair
|
|
160
143
|
|
|
161
|
-
|
|
144
|
+
The target project owns its runner, command, working directory, timeout, concurrency, retry behavior already encoded by the project, coverage tooling, and isolated integration environment. Do not impose an ecosystem-specific command policy.
|
|
162
145
|
|
|
163
|
-
|
|
146
|
+
During construction and repair, run the smallest relevant tests first. Before a pass decision, run all required layers, the complete affected-module regression, the Regression Ring, and project-wide regression when concrete coupling or project convention requires it.
|
|
164
147
|
|
|
165
|
-
|
|
148
|
+
A nonzero execution is development feedback, not an immediate workflow stop. Classify it:
|
|
166
149
|
|
|
167
|
-
|
|
150
|
+
| Classification | Action |
|
|
151
|
+
|---|---|
|
|
152
|
+
| `test-defect` | Repair the test, fixture, mock, import, syntax, or test infrastructure. |
|
|
153
|
+
| `local-production-defect` | Repair the proven internal implementation defect when public and cross-module contracts remain unchanged. |
|
|
154
|
+
| `high-risk-production-defect` | Repair only when safe, then require delta Code Review and return `TEST_NEEDS_FIXES`. |
|
|
155
|
+
| `environment-unavailable` | Return `TEST_BLOCKED` with the unavailable safe prerequisite. |
|
|
156
|
+
| `truth-unresolved` | Return `TEST_BLOCKED`; never guess the assertion. |
|
|
157
|
+
| `flaky-or-unreliable` | Diagnose within budget; otherwise return `TEST_BLOCKED`. |
|
|
168
158
|
|
|
169
|
-
|
|
159
|
+
Use at most three execution-failure repair rounds. One round is reproduce → establish truth → repair → focused verification → affected-module and Regression Ring verification. Initial test construction is not a repair round. After the third unsuccessful round:
|
|
170
160
|
|
|
171
|
-
|
|
161
|
+
- known remaining correction → `TEST_NEEDS_FIXES`;
|
|
162
|
+
- unresolved truth, reliability, safety, or environment → `TEST_BLOCKED`.
|
|
172
163
|
|
|
173
|
-
|
|
164
|
+
Never retry until green or weaken scope/assertions to manufacture a pass.
|
|
174
165
|
|
|
175
|
-
|
|
166
|
+
## Production Repair Boundary
|
|
176
167
|
|
|
177
|
-
|
|
168
|
+
A repair may close inside this skill only when evidence establishes that it is limited to private/internal implementation, a behavior-preserving algorithm correction, test infrastructure, a behavior-preserving testability seam, or local error handling with unchanged public obligations.
|
|
178
169
|
|
|
179
|
-
|
|
170
|
+
Set `review_required=true` and `review_scope=delta` when a repair affects or may affect:
|
|
180
171
|
|
|
181
|
-
-
|
|
182
|
-
-
|
|
183
|
-
-
|
|
184
|
-
-
|
|
172
|
+
- public API or observable public behavior;
|
|
173
|
+
- schema, migration, serialization, or generated protocol shape;
|
|
174
|
+
- dependency or lockfile;
|
|
175
|
+
- authorization, identity, secrets, permissions, or tenant isolation;
|
|
176
|
+
- persistence, transaction, data integrity, or concurrency semantics;
|
|
177
|
+
- cross-module contracts or consumer obligations;
|
|
178
|
+
- compatibility guarantees;
|
|
179
|
+
- any change whose locality cannot be established confidently.
|
|
185
180
|
|
|
186
|
-
|
|
181
|
+
Record completed repairs and test results, return `TEST_NEEDS_FIXES`, and stop. Do not invoke Code Review or prescribe caller routing from this skill.
|
|
187
182
|
|
|
188
|
-
|
|
183
|
+
## Mandatory Main-Agent Test Review
|
|
189
184
|
|
|
190
|
-
|
|
185
|
+
The Main Agent must personally review the complete current production-and-test state after construction and required execution. This responsibility may not be delegated.
|
|
191
186
|
|
|
192
|
-
|
|
193
|
-
- `change-classification.json`
|
|
194
|
-
- `scope.json`
|
|
195
|
-
- `behavior-risk-matrix.json`
|
|
196
|
-
- `test-plan.json`
|
|
197
|
-
- `infrastructure-changes.json`
|
|
198
|
-
- `executions.json`
|
|
199
|
-
- `verdict.json`
|
|
200
|
-
- `source-change.patch`
|
|
201
|
-
- complete `raw/` stdout/stderr
|
|
202
|
-
- `generated-tests/` snapshots
|
|
203
|
-
- `contracts/` snapshots
|
|
204
|
-
- derived `test-report.md`
|
|
187
|
+
Use at most ten completed rounds. Track report facts:
|
|
205
188
|
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
**First pass (pre-check, no `--attest`):**
|
|
213
|
-
|
|
214
|
-
```bash
|
|
215
|
-
python3 ${SKILL_DIR}/scripts/validate_test_evidence.py \
|
|
216
|
-
--evidence-dir .prizmkit/test/evidence/<evidence-id> \
|
|
217
|
-
--project-root <target-project-root>
|
|
189
|
+
```yaml
|
|
190
|
+
main_review_rounds: 0
|
|
191
|
+
accepted_findings: 0
|
|
192
|
+
fixed_findings: 0
|
|
193
|
+
rejected_findings: 0
|
|
194
|
+
unresolved_findings: 0
|
|
218
195
|
```
|
|
219
196
|
|
|
220
|
-
|
|
197
|
+
Every complete round checks:
|
|
221
198
|
|
|
222
|
-
|
|
199
|
+
- missing behavior, acceptance criteria, Regression Ring edges, and applicable risk dimensions;
|
|
200
|
+
- critical low-level boundary logic;
|
|
201
|
+
- incorrect truth precedence or expected behavior;
|
|
202
|
+
- weak, vacuous, implementation-only, nondeterministic, or flaky assertions;
|
|
203
|
+
- over-mocking, invented contracts, and missing negative side-effect assertions;
|
|
204
|
+
- unjustified layer omissions or tests absent from actual native execution;
|
|
205
|
+
- production repairs outside this skill's safe boundary.
|
|
223
206
|
|
|
224
|
-
|
|
207
|
+
For each candidate finding record exactly one decision:
|
|
225
208
|
|
|
226
|
-
```
|
|
227
|
-
|
|
228
|
-
--evidence-dir .prizmkit/test/evidence/<evidence-id> \
|
|
229
|
-
--project-root <target-project-root> \
|
|
230
|
-
--attest
|
|
209
|
+
```text
|
|
210
|
+
accepted | rejected | unresolved
|
|
231
211
|
```
|
|
232
212
|
|
|
233
|
-
|
|
213
|
+
Round behavior:
|
|
234
214
|
|
|
235
|
-
|
|
215
|
+
```text
|
|
216
|
+
accepted = 0 and unresolved = 0
|
|
217
|
+
→ Main-Agent review converged
|
|
236
218
|
|
|
237
|
-
|
|
219
|
+
accepted > 0 and rounds remain
|
|
220
|
+
→ repair directly
|
|
221
|
+
→ targeted native verification
|
|
222
|
+
→ next complete review round
|
|
238
223
|
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
3. preserve prior executions and raw output immutably;
|
|
242
|
-
4. invalidate the first changed stage and every downstream result;
|
|
243
|
-
5. resume from the last valid predecessor.
|
|
224
|
+
known correction remains after round ten
|
|
225
|
+
→ TEST_NEEDS_FIXES
|
|
244
226
|
|
|
245
|
-
|
|
227
|
+
truth, safety, input, or environment remains unresolved
|
|
228
|
+
→ TEST_BLOCKED
|
|
229
|
+
```
|
|
246
230
|
|
|
247
|
-
|
|
231
|
+
All evidence-based rejections with no unresolved item are normal convergence. Any mutation invalidates prior review and execution of that content.
|
|
248
232
|
|
|
249
|
-
|
|
233
|
+
## Optional Independent Test Reviewer
|
|
250
234
|
|
|
251
|
-
|
|
252
|
-
- `TEST_FAIL`: a valid reliable test reproduces an implementation or resolved-contract failure. The authoritative verdict record must include `repair_scope` and enough evidence for the next stage to route safely.
|
|
253
|
-
- `TEST_BLOCKED`: unknown scope, conflicting truth, unavailable/unreliable/flaky necessary execution, incomplete evidence, unmapped tests, budget truncation, or deterministic validation failure.
|
|
235
|
+
Only after Main-Agent review converges, read `${SKILL_DIR}/references/independent-test-review.md` and follow its complete contract.
|
|
254
236
|
|
|
255
|
-
|
|
237
|
+
- Create at most one Reviewer for this invocation.
|
|
238
|
+
- Permit at most five responses.
|
|
239
|
+
- Use native continuation of the exact same Reviewer after an accepted correction.
|
|
240
|
+
- The Reviewer is optional and available only when every structural capability in the reference is proven.
|
|
241
|
+
- If capability is unavailable or unproven, create no weaker replacement; record a visible strict downgrade and preserve the converged mandatory Main-Agent review.
|
|
242
|
+
- The Main Agent retains adjudication, mutation, execution, final result, and reporting authority.
|
|
243
|
+
- If response five still contains an accepted correction, return `TEST_NEEDS_FIXES`.
|
|
244
|
+
- If response five still contains unresolved truth, input, or safety, return `TEST_BLOCKED`.
|
|
245
|
+
- Never claim `TEST_PASS` for a final mutation that was not independently rechecked when independent review had already begun, unless the reference's strict downgrade rule establishes the permitted Main-Agent fallback.
|
|
256
246
|
|
|
257
|
-
##
|
|
247
|
+
## External Services and Safety
|
|
258
248
|
|
|
259
|
-
|
|
249
|
+
Read `${SKILL_DIR}/references/external-contract-mock-guidance.md` whenever an external service, persistence boundary, queue, filesystem, generated client, or cross-module endpoint affects the tested property.
|
|
260
250
|
|
|
261
|
-
|
|
251
|
+
- Prefer project-owned schemas, generated types, SDK interfaces, fixtures, and documentation.
|
|
252
|
+
- Use approved retrieval of public official documentation only when project artifacts are insufficient.
|
|
253
|
+
- Never expose secrets, private source, proprietary payloads, or repository-private data during retrieval.
|
|
254
|
+
- Never use production credentials, production databases/APIs/queues/storage, real user data, or destructive external operations.
|
|
255
|
+
- If an endpoint cannot be proven non-production, do not connect to it.
|
|
256
|
+
- Use contract-backed local fakes, mocks, fixtures, mock servers, containers, or project-owned isolated services.
|
|
257
|
+
- Record necessary unverified external behavior as a remaining risk or `TEST_BLOCKED` when it prevents the verdict.
|
|
262
258
|
|
|
263
|
-
##
|
|
259
|
+
## Final Verification
|
|
264
260
|
|
|
265
|
-
|
|
261
|
+
Before writing `TEST_PASS`:
|
|
266
262
|
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
|
|
263
|
+
1. Main-Agent review converged on the exact final state.
|
|
264
|
+
2. Independent review converged or was strictly and visibly downgraded under its capability contract.
|
|
265
|
+
3. Every required focused, module, contract/integration, applicable E2E, affected-module, and Regression Ring test passes reliably.
|
|
266
|
+
4. Any required project-wide regression passes.
|
|
267
|
+
5. No accepted correction, unresolved finding, verdict-capable scope edge, or required environment check remains.
|
|
268
|
+
6. No high-risk production repair awaits delta Code Review.
|
|
269
|
+
7. No mutation occurred after the final applicable review and execution.
|
|
272
270
|
|
|
273
|
-
|
|
271
|
+
## Terminal Results
|
|
274
272
|
|
|
275
|
-
|
|
273
|
+
Only these testing-domain outcomes are valid:
|
|
276
274
|
|
|
277
|
-
|
|
275
|
+
```text
|
|
276
|
+
TEST_PASS | TEST_NEEDS_FIXES | TEST_BLOCKED
|
|
277
|
+
```
|
|
278
278
|
|
|
279
|
-
|
|
279
|
+
- `TEST_PASS`: final reviewed state passes every required native test and no high-risk repair awaits review.
|
|
280
|
+
- `TEST_NEEDS_FIXES`: a known correction remains, a review/repair budget did not converge, or a completed high-risk repair requires delta Code Review.
|
|
281
|
+
- `TEST_BLOCKED`: expected truth, safe input, environment, execution reliability, external-target safety, or required review input prevents a safe verdict.
|
|
280
282
|
|
|
281
|
-
|
|
283
|
+
No conditional pass exists.
|
|
284
|
+
|
|
285
|
+
## Terminal Report and Result
|
|
286
|
+
|
|
287
|
+
Read `${SKILL_DIR}/references/test-report-template.md` before finalization. Write `test-report.md` and `test-result.json` together for every terminal outcome. The report must use the exact canonical `## Final Result` section and `- Result: <TEST_*>` marker from that template; do not rename it to `Terminal Result`, `Final Decision`, or another heading.
|
|
288
|
+
|
|
289
|
+
`test-result.json` is a terminal projection, not a checkpoint:
|
|
290
|
+
|
|
291
|
+
```json
|
|
292
|
+
{
|
|
293
|
+
"schema_version": 1,
|
|
294
|
+
"result": "TEST_PASS",
|
|
295
|
+
"report": "test-report.md",
|
|
296
|
+
"main_review_rounds": 3,
|
|
297
|
+
"independent_review": {
|
|
298
|
+
"status": "completed",
|
|
299
|
+
"responses": 2,
|
|
300
|
+
"downgrade_reason": null,
|
|
301
|
+
"final_state_rechecked": true
|
|
302
|
+
},
|
|
303
|
+
"repair_rounds": 1,
|
|
304
|
+
"production_changed": false,
|
|
305
|
+
"review_required": false,
|
|
306
|
+
"review_scope": null,
|
|
307
|
+
"unresolved_items": []
|
|
308
|
+
}
|
|
309
|
+
```
|
|
282
310
|
|
|
283
|
-
|
|
311
|
+
Allowed `independent_review.status` values are `completed`, `downgraded`, and `not_applicable`. `completed` requires one through five responses and `final_state_rechecked=true`; `downgraded` requires zero responses, a non-empty reason, and `final_state_rechecked=false`; `not_applicable` requires zero responses, no reason, and `final_state_rechecked=false`. `review_scope` is `delta` only when `review_required=true`; otherwise it is null. The report and JSON must agree on result, review requirement, and unresolved items. If consistency cannot be established, write a truthful `TEST_BLOCKED` result rather than fabricating pass.
|
|
284
312
|
|
|
285
|
-
**HANDOFF:** Return the
|
|
313
|
+
**HANDOFF:** Return the testing result plus `test-report.md` and `test-result.json` paths to the caller, then stop.
|