prizmkit 1.1.139 → 1.1.141
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bundled/VERSION.json +3 -3
- package/bundled/dev-pipeline/prizmkit_runtime/checkpoint_state.py +41 -121
- package/bundled/dev-pipeline/prizmkit_runtime/runner_models.py +0 -6
- package/bundled/dev-pipeline/prizmkit_runtime/runners.py +1 -2
- package/bundled/dev-pipeline/prizmkit_runtime/test_result.py +150 -0
- package/bundled/dev-pipeline/scripts/generate-bootstrap-prompt.py +21 -804
- package/bundled/dev-pipeline/scripts/generate-bugfix-prompt.py +4 -6
- package/bundled/dev-pipeline/scripts/generate-recovery-prompt.py +1 -1
- package/bundled/dev-pipeline/scripts/generate-refactor-prompt.py +4 -6
- package/bundled/dev-pipeline/scripts/prompt_framework.py +56 -55
- package/bundled/dev-pipeline/scripts/update-bug-status.py +1 -2
- package/bundled/dev-pipeline/scripts/update-checkpoint.py +14 -96
- package/bundled/dev-pipeline/scripts/update-feature-status.py +1 -2
- package/bundled/dev-pipeline/scripts/update-refactor-status.py +1 -2
- package/bundled/dev-pipeline/templates/bootstrap-prompt.md +3 -3
- package/bundled/dev-pipeline/templates/bootstrap-tier3.md +7 -18
- package/bundled/dev-pipeline/templates/bugfix-bootstrap-prompt.md +6 -6
- package/bundled/dev-pipeline/templates/refactor-bootstrap-prompt.md +5 -5
- package/bundled/dev-pipeline/templates/sections/bugfix-phase-commit-report.md +1 -1
- package/bundled/dev-pipeline/templates/sections/bugfix-phase-diagnose-plan.md +1 -1
- package/bundled/dev-pipeline/templates/sections/bugfix-phase-manual-verification.md +1 -1
- package/bundled/dev-pipeline/templates/sections/bugfix-phase-test.md +13 -10
- package/bundled/dev-pipeline/templates/sections/bugfix-reminders.md +1 -1
- package/bundled/dev-pipeline/templates/sections/checkpoint-system.md +14 -9
- package/bundled/dev-pipeline/templates/sections/headless-commit-authorization.md +1 -1
- package/bundled/dev-pipeline/templates/sections/phase-commit-full.md +1 -1
- package/bundled/dev-pipeline/templates/sections/phase-commit.md +1 -1
- package/bundled/dev-pipeline/templates/sections/phase-context-snapshot-agent-suffix.md +2 -2
- package/bundled/dev-pipeline/templates/sections/phase-context-snapshot-base.md +1 -1
- package/bundled/dev-pipeline/templates/sections/phase-context-snapshot-lite-suffix.md +1 -1
- package/bundled/dev-pipeline/templates/sections/phase-implement-lite.md +1 -1
- package/bundled/dev-pipeline/templates/sections/phase-prizmkit-test.md +14 -26
- package/bundled/dev-pipeline/templates/sections/phase-specify-plan-full.md +3 -3
- package/bundled/dev-pipeline/templates/sections/refactor-phase-test.md +13 -10
- package/bundled/dev-pipeline/templates/sections/refactor-reminders.md +1 -1
- package/bundled/dev-pipeline/tests/test_checkpoint_state.py +287 -384
- package/bundled/dev-pipeline/tests/test_generate_bootstrap_prompt.py +119 -684
- package/bundled/dev-pipeline/tests/test_generate_bugfix_prompt.py +17 -40
- package/bundled/dev-pipeline/tests/test_generate_refactor_prompt.py +20 -23
- package/bundled/dev-pipeline/tests/test_python_runner_parity.py +29 -32
- package/bundled/skills/_metadata.json +6 -6
- package/bundled/skills/prizmkit/SKILL.md +8 -13
- package/bundled/skills/prizmkit/references/workflow-state-protocol.md +61 -80
- package/bundled/skills/prizmkit-code-review/SKILL.md +3 -3
- package/bundled/skills/prizmkit-code-review/references/workflow-state-protocol.md +61 -80
- package/bundled/skills/prizmkit-committer/SKILL.md +3 -3
- package/bundled/skills/prizmkit-committer/references/workflow-state-protocol.md +61 -80
- package/bundled/skills/prizmkit-implement/references/workflow-state-protocol.md +61 -80
- package/bundled/skills/prizmkit-plan/references/workflow-state-protocol.md +61 -80
- package/bundled/skills/prizmkit-retrospective/SKILL.md +2 -2
- package/bundled/skills/prizmkit-retrospective/references/workflow-state-protocol.md +61 -80
- package/bundled/skills/prizmkit-test/SKILL.md +217 -187
- package/bundled/skills/prizmkit-test/references/external-contract-mock-guidance.md +119 -0
- package/bundled/skills/prizmkit-test/references/independent-test-review.md +269 -0
- package/bundled/skills/prizmkit-test/references/test-coverage-model.md +193 -0
- package/bundled/skills/prizmkit-test/references/test-report-template.md +164 -79
- package/bundled/skills/prizmkit-workflow/SKILL.md +15 -23
- package/bundled/skills/prizmkit-workflow/references/workflow-state-protocol.md +61 -80
- package/package.json +1 -1
- package/bundled/skills/prizmkit-test/assets/authoritative-records.schema.json +0 -476
- package/bundled/skills/prizmkit-test/assets/behavior-risk-matrix.schema.json +0 -116
- package/bundled/skills/prizmkit-test/assets/evidence-manifest.schema.json +0 -111
- package/bundled/skills/prizmkit-test/assets/evidence-package-template.json +0 -66
- package/bundled/skills/prizmkit-test/references/boundary-coverage-protocol.md +0 -104
- package/bundled/skills/prizmkit-test/references/contract-mock-protocol.md +0 -65
- package/bundled/skills/prizmkit-test/references/evidence-protocol.md +0 -208
- package/bundled/skills/prizmkit-test/references/evidence-request-protocol.md +0 -80
- package/bundled/skills/prizmkit-test/references/examples.md +0 -100
- package/bundled/skills/prizmkit-test/references/service-boundary-test-catalog.md +0 -228
- package/bundled/skills/prizmkit-test/references/test-generation-steps.md +0 -118
- package/bundled/skills/prizmkit-test/references/trusted-evidence-execution.md +0 -110
- package/bundled/skills/prizmkit-test/references/workflow-state-protocol.md +0 -181
- package/bundled/skills/prizmkit-test/scripts/build_test_evidence.py +0 -1851
- package/bundled/skills/prizmkit-test/scripts/validate_test_evidence.py +0 -1793
|
@@ -1,283 +1,313 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: "prizmkit-test"
|
|
3
|
-
description: "
|
|
3
|
+
description: "AI-led project-native testing for the complete affected business module and its Regression Ring. Adds comprehensive behavior, boundary, error, state, side-effect, permission, concurrency, idempotency, time, dependency, and consumer tests; repairs valid failures in a bounded loop; performs mandatory Main-Agent review plus one optional strict independent Test Reviewer; and returns TEST_PASS, TEST_NEEDS_FIXES, or TEST_BLOCKED with a human report and minimal terminal result. (project)"
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# PrizmKit Test
|
|
7
7
|
|
|
8
|
-
`/prizmkit-test`
|
|
8
|
+
`/prizmkit-test` is the independent testing stage for the current workspace. It understands the affected business behavior, fills project-native test gaps, executes the tests, repairs valid failures, reviews the final tests, and reports a truthful testing-domain result.
|
|
9
9
|
|
|
10
|
-
|
|
10
|
+
It does not build an evidence package, prove historical TDD order, validate a previous checkout, or manage pipeline/session state.
|
|
11
11
|
|
|
12
|
-
##
|
|
12
|
+
## Atomic Skill Boundary
|
|
13
13
|
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
Fix only the protocol mechanisms needed for safe, replayable evidence: project/evidence/request locators, schema-shaped requests, path confinement, real process execution, complete raw capture, runner-generated receipts, append-only history, and resume invalidation. Read `${SKILL_DIR}/references/trusted-evidence-execution.md` before inventory, execution, resume, or replay; it defines the trust boundary and builder commands. Read `${SKILL_DIR}/references/evidence-request-protocol.md` when authoring any request or authoritative JSON record; it defines request ownership, structured N/A, and the schema map.
|
|
17
|
-
|
|
18
|
-
## Canonical Evidence Builder Lifecycle
|
|
19
|
-
|
|
20
|
-
The Main Agent supplies semantic requests only; it does not author authoritative package records. Initialize and drive one evidence directory through the builder in this order:
|
|
14
|
+
The Main Agent owns this invocation from analysis through final report:
|
|
21
15
|
|
|
22
16
|
```text
|
|
23
|
-
|
|
17
|
+
resolve current change and requirement context
|
|
18
|
+
→ discover affected module and Regression Ring
|
|
19
|
+
→ model observable behavior and applicable risks
|
|
20
|
+
→ inspect and add project-native tests
|
|
21
|
+
→ execute and repair valid failures: maximum three rounds
|
|
22
|
+
→ mandatory Main-Agent test review: maximum ten completed rounds
|
|
23
|
+
→ optional independent Test Reviewer: one unit, maximum five responses
|
|
24
|
+
→ final native regression on the exact final state
|
|
25
|
+
→ write test-report.md and test-result.json
|
|
26
|
+
→ return TEST_PASS | TEST_NEEDS_FIXES | TEST_BLOCKED
|
|
24
27
|
```
|
|
25
28
|
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
- `init`: creates the package directories and immutable baseline identity.
|
|
29
|
-
- `capture-change`: captures the complete canonical change, including untracked, deleted, renamed, and copied files; never substitute plain `git diff`. `inventory` then requires its changed-file set to match the capture exactly and rejects stale captures or unaccounted live paths.
|
|
30
|
-
- `inventory`: hashes current source/tests/contracts/lockfiles, rejects category overlap, excludes runtime-only paths, and validates directory roots plus the complete capture-derived changed-file set.
|
|
31
|
-
- `prepare-tests`: discovers repository-native test files from the canonical capture and inventory, generates stable builder-owned test IDs, derives existing/added/modified status, and snapshots only added/modified live test sources.
|
|
32
|
-
- `execute`: runs native project commands and owns `executions.json`, `receipts/`, and `raw/`. Each request uses exactly one binding mode: `test_ids`, `test_paths`, or `auto_bind=true`. The builder infers recognized package-wide or explicit file/directory/glob command scope, blocks opaque commands or mismatched declarations, and persists verified canonical `test_ids` only.
|
|
33
|
-
- `finalize`: runs the pre-finalize semantic-record and required-layer checks, rechecks canonical capture freshness, blocks every selected unreliable execution, and owns manifest, stage hashes, verdict, generated-test snapshots, and evidence identity. If semantic records are incomplete or malformed, finalize fails before creating an immutable package; do not hand-edit a finalized package or start another package after a protocol budget is exhausted.
|
|
34
|
-
- `render-report`: derives `test-report.md`; it is never authoritative.
|
|
35
|
-
- `resume`: preserves prior receipts and refuses to rebind an identity-changing package; create a new package after scope/source identity changes.
|
|
29
|
+
After writing its result, this skill returns control to its caller. It must not:
|
|
36
30
|
|
|
37
|
-
|
|
31
|
+
- invoke implementation, Code Review, retrospective, or committer;
|
|
32
|
+
- update a workflow checkpoint or pipeline runtime state;
|
|
33
|
+
- classify an AI CLI session as success, crash, retry, or continuation;
|
|
34
|
+
- decide whether a workflow terminates or resumes;
|
|
35
|
+
- call a testing-domain non-pass result a runtime failure.
|
|
38
36
|
|
|
39
|
-
|
|
37
|
+
## Inputs and Outputs
|
|
40
38
|
|
|
39
|
+
| Parameter | Required | Description |
|
|
40
|
+
|---|---|---|
|
|
41
|
+
| `scope` | No | `full-project`, `module:<name>`, `feature:<slug>`, or `this-change`. Defaults to `this-change` when current change context exists, otherwise `full-project`. |
|
|
42
|
+
| `artifact_dir` | No | Active requirement directory containing available specification, plan, and prior review context. Reuse the caller-provided path. |
|
|
43
|
+
| `changed_files` | No | Explicit current changed paths. Highest-priority scope locator. |
|
|
44
|
+
| `diff_base` | No | Git comparison base when explicit changed paths are absent. |
|
|
45
|
+
| `test_commands` | No | Project-native commands when repository conventions are ambiguous. |
|
|
41
46
|
|
|
42
|
-
|
|
47
|
+
Write exactly two terminal artifacts in `artifact_dir` when it is available; otherwise use a clearly reported caller-visible artifact directory selected from the active request context:
|
|
43
48
|
|
|
44
49
|
```text
|
|
45
|
-
|
|
50
|
+
{artifact_dir}/
|
|
51
|
+
├── test-report.md
|
|
52
|
+
└── test-result.json
|
|
46
53
|
```
|
|
47
54
|
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
## Atomic Stage Boundary
|
|
55
|
+
Do not create `.prizmkit/test/evidence/`, manifests, hashes, attestations, package lifecycle records, or test-internal checkpoints.
|
|
51
56
|
|
|
52
|
-
|
|
57
|
+
## Context Loading and Scope Resolution
|
|
53
58
|
|
|
59
|
+
1. Reuse current requirement and workspace context already loaded by the Main Agent.
|
|
60
|
+
2. Load `root.prizm`, the relevant module indexes, and the applicable detail docs before modifying source or tests.
|
|
61
|
+
3. Inspect current staged, unstaged, untracked, deleted, and renamed paths. Changed lines locate impact but never define completeness.
|
|
62
|
+
4. Read the available specification, acceptance criteria, plan, and prior review only as needed to establish expected behavior.
|
|
63
|
+
5. Inspect manifests, runner configuration, CI conventions, existing test assertions, fixtures, fakes, mocks, contracts, and coverage support.
|
|
64
|
+
6. If necessary scope cannot be determined safely, report the precise missing input and return `TEST_BLOCKED`.
|
|
54
65
|
|
|
55
|
-
|
|
56
|
-
- runtime configuration, schema, migration, generated-runtime, lockfile, or dependency changes;
|
|
57
|
-
- public interfaces and data contracts;
|
|
58
|
-
- frontend, backend, library, CLI, adapter, and multi-module changes;
|
|
59
|
-
- legacy modules whose observable test coverage is incomplete;
|
|
60
|
-
- explicit test, boundary, evidence, confidence, or audit requests.
|
|
66
|
+
Do not infer the active requirement solely from the name of an old artifact directory. Do not use a historical worktree, second checkout, mutation proof, baseline-failure proof, or test overlay.
|
|
61
67
|
|
|
62
|
-
|
|
68
|
+
## Coverage Model
|
|
63
69
|
|
|
64
|
-
|
|
70
|
+
Read `${SKILL_DIR}/references/test-coverage-model.md` before designing or reviewing tests.
|
|
65
71
|
|
|
66
|
-
|
|
67
|
-
2. Never use production credentials, production databases, production APIs, production queues/storage, or destructive operations against real data.
|
|
68
|
-
3. Preserve complete command/environment/output values as requested. Mark the package `sensitivity=project-controlled`; the project owns access control, retention, and upload policy.
|
|
69
|
-
4. Treat code-level dependencies as mock-first. Real deployed test-environment validation is a separate authorized activity.
|
|
70
|
-
5. Never delete existing tests.
|
|
71
|
-
6. Evidence-protocol recovery is separately bounded: allow at most two package initializations and two semantic preflight failures for one canonical change identity. After the budget is exhausted, return `TEST_BLOCKED`; do not create another versioned package or hand-edit finalized evidence.
|
|
72
|
-
7. For `TEST_FAIL`, classify and persist an evidence-backed `repair_scope` of `test-infrastructure`, `production`, `runtime`, `schema`, `dependency`, `public-interface`, or `unknown`. Test-infrastructure scope routes to implement then test; production-family scope routes to implement, code-review, then test. If evidence cannot support a safe classification, return `TEST_BLOCKED`.
|
|
72
|
+
### Affected Business Module
|
|
73
73
|
|
|
74
|
-
|
|
74
|
+
Prefer an explicit project module. If none exists, derive one cohesive boundary from files that jointly implement the same observable responsibility. Cover every discoverable observable behavior of that module, including relevant legacy behavior whose tests are incomplete.
|
|
75
75
|
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
| Parameter | Required | Description |
|
|
79
|
-
|-----------|----------|-------------|
|
|
80
|
-
| `scope` | No | `full-project`, `module:<name>`, `feature:<slug>`, or `this-change`. Defaults to `this-change` with a valid artifact directory, otherwise `full-project`. |
|
|
81
|
-
| `artifact_dir` | No | Change artifact containing `spec.md` + `plan.md`; used to resolve acceptance conditions and changed scope. |
|
|
82
|
-
| `changed_files` | No | Explicit changed files; highest-priority scope input. |
|
|
83
|
-
| `diff_base` | No | Git baseline used when `changed_files` is absent. |
|
|
84
|
-
| `evidence_target` | No | Existing evidence ID/directory to validate and resume. |
|
|
85
|
-
| `execution_budget` | No | Bounded time/attempt budget. Incomplete necessary work is `TEST_BLOCKED`, never truncated to pass. |
|
|
86
|
-
| `test_commands` | No | Explicit project-native commands when discovery is ambiguous. |
|
|
76
|
+
### Regression Ring
|
|
87
77
|
|
|
88
|
-
|
|
78
|
+
Expand only along concrete coupling evidence:
|
|
89
79
|
|
|
90
|
-
|
|
80
|
+
- direct callers;
|
|
81
|
+
- consumers of values, errors, events, files, or ordering;
|
|
82
|
+
- shared schemas, types, protocols, generated assets, and adapters;
|
|
83
|
+
- persistence, cache, lock, queue, transaction, and other shared state dependencies.
|
|
91
84
|
|
|
92
|
-
|
|
85
|
+
Do not perform an unconditional repository-wide sweep.
|
|
93
86
|
|
|
94
|
-
|
|
87
|
+
### Behavior and Risks
|
|
95
88
|
|
|
96
|
-
-
|
|
97
|
-
- Load `${SKILL_DIR}/assets/evidence-manifest.schema.json` when creating or validating `manifest.json`.
|
|
98
|
-
- Load `${SKILL_DIR}/assets/behavior-risk-matrix.schema.json` during `CONTRACT_MODEL` or matrix validation.
|
|
99
|
-
- Load `${SKILL_DIR}/assets/authoritative-records.schema.json` when creating or validating any other authoritative request/record.
|
|
100
|
-
- Load `${SKILL_DIR}/assets/evidence-package-template.json` when initializing or resuming an evidence directory.
|
|
101
|
-
- Read `${SKILL_DIR}/references/contract-mock-protocol.md` only when dependency, contract, or external-boundary risks require doubles.
|
|
102
|
-
- Read `${SKILL_DIR}/references/test-generation-steps.md` during `INFRA_READY`, `TEST_BUILD`, or `EXECUTE_PROVE`.
|
|
103
|
-
- Read `${SKILL_DIR}/references/service-boundary-test-catalog.md` only when domain boundary signals need risk discovery.
|
|
104
|
-
- Read `${SKILL_DIR}/references/boundary-coverage-protocol.md` when defining module roots, exclusions, Regression Ring, or behavior-risk completeness.
|
|
89
|
+
Model outside-in from business capability or acceptance criterion to public entry point, rules, invariants, state, side effects, low-level logic, dependencies, and consumers. For each behavior, assess applicable:
|
|
105
90
|
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
### 1. CHANGE_CLASSIFY
|
|
111
|
-
|
|
112
|
-
Classify the scope as `behavior` or `lightweight` and write `change-classification.json`.
|
|
91
|
+
```text
|
|
92
|
+
functional | boundary | error | state | side-effect | permission
|
|
93
|
+
concurrency | idempotency | time | dependency | consumer
|
|
94
|
+
```
|
|
113
95
|
|
|
114
|
-
|
|
115
|
-
- `lightweight`: exclusively docs/comments/formatting with deterministic evidence that runtime behavior cannot change.
|
|
96
|
+
A dimension may be omitted only with a concise behavior-specific reason. Coverage percentages are diagnostic signals, never a completion threshold.
|
|
116
97
|
|
|
117
|
-
|
|
98
|
+
### Test Layers
|
|
118
99
|
|
|
119
|
-
|
|
100
|
+
Use the lowest layer that proves a property without losing relevant composition, then add higher layers for composition risk:
|
|
120
101
|
|
|
121
|
-
|
|
102
|
+
```text
|
|
103
|
+
focused/unit
|
|
104
|
+
→ module/component
|
|
105
|
+
→ contract/integration
|
|
106
|
+
→ code-level E2E when applicable
|
|
107
|
+
→ complete affected-module regression
|
|
108
|
+
→ Regression Ring verification
|
|
109
|
+
```
|
|
122
110
|
|
|
123
|
-
-
|
|
124
|
-
- affected module boundary (`explicit` or `cohesion-derived`);
|
|
125
|
-
- Primary Scope containing all observable behavior of that module;
|
|
126
|
-
- Regression Ring containing direct callers, consumers, shared contracts, and state dependencies, each mapped to planned tests;
|
|
127
|
-
- contract, lockfile, test, and module-boundary discovery evidence;
|
|
128
|
-
- Unresolved Edges for dynamic/unprovable coupling.
|
|
111
|
+
Do not mechanically test every private function. Add direct focused tests for critical low-level logic when complexity, boundaries, risk, or failure localization justify them; otherwise prove it through public behavior.
|
|
129
112
|
|
|
130
|
-
|
|
113
|
+
## Test Construction
|
|
131
114
|
|
|
132
|
-
|
|
115
|
+
Read `${SKILL_DIR}/references/test-coverage-model.md` and, when external or cross-module contracts apply, `${SKILL_DIR}/references/external-contract-mock-guidance.md`.
|
|
133
116
|
|
|
134
|
-
|
|
117
|
+
Expected behavior follows this precedence:
|
|
135
118
|
|
|
136
|
-
1.
|
|
137
|
-
2. machine-readable
|
|
138
|
-
3. acceptance
|
|
119
|
+
1. confirmed specification;
|
|
120
|
+
2. machine-readable contract;
|
|
121
|
+
3. acceptance criteria;
|
|
139
122
|
4. trusted existing tests;
|
|
140
|
-
5. callers
|
|
123
|
+
5. callers and consumers;
|
|
141
124
|
6. current implementation.
|
|
142
125
|
|
|
143
|
-
|
|
126
|
+
Conflicting higher-precedence truth that cannot be resolved is `TEST_BLOCKED`; do not preserve a possible implementation defect as a characterization test.
|
|
144
127
|
|
|
145
|
-
|
|
128
|
+
Tests must:
|
|
146
129
|
|
|
147
|
-
|
|
130
|
+
- assert observable outputs, errors, state transitions, side effects, ordering, counts, and absence of partial effects;
|
|
131
|
+
- isolate only dependencies irrelevant to the property being proved;
|
|
132
|
+
- follow project-native naming, location, framework, fixture, and helper conventions;
|
|
133
|
+
- control clocks, randomness, concurrency, synchronization, and fixtures deterministically;
|
|
134
|
+
- verify failure cleanup and useful diagnostics;
|
|
135
|
+
- avoid vacuous assertions and assertions that merely duplicate implementation steps;
|
|
136
|
+
- never delete or weaken a valid existing test just to obtain green output.
|
|
148
137
|
|
|
149
|
-
|
|
138
|
+
Reuse an adequate existing framework. Add only the smallest necessary project-native infrastructure when a required property cannot otherwise be expressed.
|
|
150
139
|
|
|
151
|
-
|
|
140
|
+
Minimal behavior-preserving testability seams are allowed, including dependency injection through an existing abstraction, pure-function extraction, an internal adapter, or controllable clock/randomness/state. Classify any production change under the repair boundary below.
|
|
152
141
|
|
|
153
|
-
|
|
154
|
-
2. module/component;
|
|
155
|
-
3. contract/integration;
|
|
156
|
-
4. complete affected-module regression;
|
|
157
|
-
5. Regression Ring.
|
|
142
|
+
## Native Execution and Failure Repair
|
|
158
143
|
|
|
159
|
-
|
|
144
|
+
The target project owns its runner, command, working directory, timeout, concurrency, retry behavior already encoded by the project, coverage tooling, and isolated integration environment. Do not impose an ecosystem-specific command policy.
|
|
160
145
|
|
|
161
|
-
|
|
146
|
+
During construction and repair, run the smallest relevant tests first. Before a pass decision, run all required layers, the complete affected-module regression, the Regression Ring, and project-wide regression when concrete coupling or project convention requires it.
|
|
162
147
|
|
|
163
|
-
|
|
148
|
+
A nonzero execution is development feedback, not an immediate workflow stop. Classify it:
|
|
164
149
|
|
|
165
|
-
|
|
150
|
+
| Classification | Action |
|
|
151
|
+
|---|---|
|
|
152
|
+
| `test-defect` | Repair the test, fixture, mock, import, syntax, or test infrastructure. |
|
|
153
|
+
| `local-production-defect` | Repair the proven internal implementation defect when public and cross-module contracts remain unchanged. |
|
|
154
|
+
| `high-risk-production-defect` | Repair only when safe, then require delta Code Review and return `TEST_NEEDS_FIXES`. |
|
|
155
|
+
| `environment-unavailable` | Return `TEST_BLOCKED` with the unavailable safe prerequisite. |
|
|
156
|
+
| `truth-unresolved` | Return `TEST_BLOCKED`; never guess the assertion. |
|
|
157
|
+
| `flaky-or-unreliable` | Diagnose within budget; otherwise return `TEST_BLOCKED`. |
|
|
166
158
|
|
|
167
|
-
|
|
159
|
+
Use at most three execution-failure repair rounds. One round is reproduce → establish truth → repair → focused verification → affected-module and Regression Ring verification. Initial test construction is not a repair round. After the third unsuccessful round:
|
|
168
160
|
|
|
169
|
-
|
|
161
|
+
- known remaining correction → `TEST_NEEDS_FIXES`;
|
|
162
|
+
- unresolved truth, reliability, safety, or environment → `TEST_BLOCKED`.
|
|
170
163
|
|
|
171
|
-
|
|
164
|
+
Never retry until green or weaken scope/assertions to manufacture a pass.
|
|
172
165
|
|
|
173
|
-
|
|
166
|
+
## Production Repair Boundary
|
|
174
167
|
|
|
175
|
-
|
|
168
|
+
A repair may close inside this skill only when evidence establishes that it is limited to private/internal implementation, a behavior-preserving algorithm correction, test infrastructure, a behavior-preserving testability seam, or local error handling with unchanged public obligations.
|
|
176
169
|
|
|
177
|
-
|
|
170
|
+
Set `review_required=true` and `review_scope=delta` when a repair affects or may affect:
|
|
178
171
|
|
|
179
|
-
|
|
172
|
+
- public API or observable public behavior;
|
|
173
|
+
- schema, migration, serialization, or generated protocol shape;
|
|
174
|
+
- dependency or lockfile;
|
|
175
|
+
- authorization, identity, secrets, permissions, or tenant isolation;
|
|
176
|
+
- persistence, transaction, data integrity, or concurrency semantics;
|
|
177
|
+
- cross-module contracts or consumer obligations;
|
|
178
|
+
- compatibility guarantees;
|
|
179
|
+
- any change whose locality cannot be established confidently.
|
|
180
180
|
|
|
181
|
-
|
|
182
|
-
- Bind each applicable behavior risk to planned tests and selected, reliable execution receipts. A passing current-state test is valid evidence even when the historical implementation would also have passed it.
|
|
183
|
-
- Preserve complete runner receipts, raw outputs, request bindings, and replay links. The builder owns execution provenance; the Main Agent does not author receipt records.
|
|
184
|
-
- Required test failures produce `TEST_FAIL`; any selected unreliable execution, missing mapping, unavailable execution, unresolved risk, or budget truncation produces `TEST_BLOCKED`. `TEST_PASS` requires every selected execution to be reliable with exit code zero and every planned test to have a selected reliable zero-exit receipt.
|
|
181
|
+
Record completed repairs and test results, return `TEST_NEEDS_FIXES`, and stop. Do not invoke Code Review or prescribe caller routing from this skill.
|
|
185
182
|
|
|
186
|
-
|
|
183
|
+
## Mandatory Main-Agent Test Review
|
|
187
184
|
|
|
188
|
-
|
|
185
|
+
The Main Agent must personally review the complete current production-and-test state after construction and required execution. This responsibility may not be delegated.
|
|
189
186
|
|
|
190
|
-
|
|
187
|
+
Use at most ten completed rounds. Track report facts:
|
|
191
188
|
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
- `verdict.json`
|
|
200
|
-
- `source-change.patch`
|
|
201
|
-
- complete `raw/` stdout/stderr
|
|
202
|
-
- `generated-tests/` snapshots
|
|
203
|
-
- `contracts/` snapshots
|
|
204
|
-
- derived `test-report.md`
|
|
205
|
-
|
|
206
|
-
Hash baseline, diff, inventoried source/tests/contracts/lockfiles, environment, plan, and every evidence file. Evidence files remain individually content-addressed through `manifest.json`. Set `environment_claim=mocked-code-level-only`.
|
|
189
|
+
```yaml
|
|
190
|
+
main_review_rounds: 0
|
|
191
|
+
accepted_findings: 0
|
|
192
|
+
fixed_findings: 0
|
|
193
|
+
rejected_findings: 0
|
|
194
|
+
unresolved_findings: 0
|
|
195
|
+
```
|
|
207
196
|
|
|
208
|
-
|
|
197
|
+
Every complete round checks:
|
|
209
198
|
|
|
210
|
-
|
|
199
|
+
- missing behavior, acceptance criteria, Regression Ring edges, and applicable risk dimensions;
|
|
200
|
+
- critical low-level boundary logic;
|
|
201
|
+
- incorrect truth precedence or expected behavior;
|
|
202
|
+
- weak, vacuous, implementation-only, nondeterministic, or flaky assertions;
|
|
203
|
+
- over-mocking, invented contracts, and missing negative side-effect assertions;
|
|
204
|
+
- unjustified layer omissions or tests absent from actual native execution;
|
|
205
|
+
- production repairs outside this skill's safe boundary.
|
|
211
206
|
|
|
212
|
-
|
|
207
|
+
For each candidate finding record exactly one decision:
|
|
213
208
|
|
|
214
|
-
```
|
|
215
|
-
|
|
216
|
-
--evidence-dir .prizmkit/test/evidence/<evidence-id> \
|
|
217
|
-
--project-root <target-project-root>
|
|
209
|
+
```text
|
|
210
|
+
accepted | rejected | unresolved
|
|
218
211
|
```
|
|
219
212
|
|
|
220
|
-
|
|
213
|
+
Round behavior:
|
|
221
214
|
|
|
222
|
-
|
|
215
|
+
```text
|
|
216
|
+
accepted = 0 and unresolved = 0
|
|
217
|
+
→ Main-Agent review converged
|
|
223
218
|
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
--attest
|
|
229
|
-
```
|
|
219
|
+
accepted > 0 and rounds remain
|
|
220
|
+
→ repair directly
|
|
221
|
+
→ targeted native verification
|
|
222
|
+
→ next complete review round
|
|
230
223
|
|
|
231
|
-
|
|
224
|
+
known correction remains after round ten
|
|
225
|
+
→ TEST_NEEDS_FIXES
|
|
232
226
|
|
|
233
|
-
|
|
227
|
+
truth, safety, input, or environment remains unresolved
|
|
228
|
+
→ TEST_BLOCKED
|
|
229
|
+
```
|
|
234
230
|
|
|
235
|
-
|
|
231
|
+
All evidence-based rejections with no unresolved item are normal convergence. Any mutation invalidates prior review and execution of that content.
|
|
236
232
|
|
|
237
|
-
|
|
238
|
-
2. recompute code/test/contract/dependency/environment/plan input hashes;
|
|
239
|
-
3. preserve prior executions and raw output immutably;
|
|
240
|
-
4. invalidate the first changed stage and every downstream result;
|
|
241
|
-
5. resume from the last valid predecessor.
|
|
233
|
+
## Optional Independent Test Reviewer
|
|
242
234
|
|
|
243
|
-
|
|
235
|
+
Only after Main-Agent review converges, read `${SKILL_DIR}/references/independent-test-review.md` and follow its complete contract.
|
|
244
236
|
|
|
245
|
-
|
|
237
|
+
- Create at most one Reviewer for this invocation.
|
|
238
|
+
- Permit at most five responses.
|
|
239
|
+
- Use native continuation of the exact same Reviewer after an accepted correction.
|
|
240
|
+
- The Reviewer is optional and available only when every structural capability in the reference is proven.
|
|
241
|
+
- If capability is unavailable or unproven, create no weaker replacement; record a visible strict downgrade and preserve the converged mandatory Main-Agent review.
|
|
242
|
+
- The Main Agent retains adjudication, mutation, execution, final result, and reporting authority.
|
|
243
|
+
- If response five still contains an accepted correction, return `TEST_NEEDS_FIXES`.
|
|
244
|
+
- If response five still contains unresolved truth, input, or safety, return `TEST_BLOCKED`.
|
|
245
|
+
- Never claim `TEST_PASS` for a final mutation that was not independently rechecked when independent review had already begun, unless the reference's strict downgrade rule establishes the permitted Main-Agent fallback.
|
|
246
246
|
|
|
247
|
-
|
|
247
|
+
## External Services and Safety
|
|
248
248
|
|
|
249
|
-
-
|
|
250
|
-
- `TEST_FAIL`: a valid reliable test reproduces an implementation or resolved-contract failure. The authoritative verdict record must include `repair_scope` and enough evidence for the next stage to route safely.
|
|
251
|
-
- `TEST_BLOCKED`: unknown scope, conflicting truth, unavailable/unreliable/flaky necessary execution, incomplete evidence, unmapped tests, budget truncation, or deterministic validation failure.
|
|
249
|
+
Read `${SKILL_DIR}/references/external-contract-mock-guidance.md` whenever an external service, persistence boundary, queue, filesystem, generated client, or cross-module endpoint affects the tested property.
|
|
252
250
|
|
|
253
|
-
|
|
251
|
+
- Prefer project-owned schemas, generated types, SDK interfaces, fixtures, and documentation.
|
|
252
|
+
- Use approved retrieval of public official documentation only when project artifacts are insufficient.
|
|
253
|
+
- Never expose secrets, private source, proprietary payloads, or repository-private data during retrieval.
|
|
254
|
+
- Never use production credentials, production databases/APIs/queues/storage, real user data, or destructive external operations.
|
|
255
|
+
- If an endpoint cannot be proven non-production, do not connect to it.
|
|
256
|
+
- Use contract-backed local fakes, mocks, fixtures, mock servers, containers, or project-owned isolated services.
|
|
257
|
+
- Record necessary unverified external behavior as a remaining risk or `TEST_BLOCKED` when it prevents the verdict.
|
|
254
258
|
|
|
255
|
-
##
|
|
259
|
+
## Final Verification
|
|
256
260
|
|
|
257
|
-
Before writing
|
|
261
|
+
Before writing `TEST_PASS`:
|
|
258
262
|
|
|
259
|
-
|
|
263
|
+
1. Main-Agent review converged on the exact final state.
|
|
264
|
+
2. Independent review converged or was strictly and visibly downgraded under its capability contract.
|
|
265
|
+
3. Every required focused, module, contract/integration, applicable E2E, affected-module, and Regression Ring test passes reliably.
|
|
266
|
+
4. Any required project-wide regression passes.
|
|
267
|
+
5. No accepted correction, unresolved finding, verdict-capable scope edge, or required environment check remains.
|
|
268
|
+
6. No high-risk production repair awaits delta Code Review.
|
|
269
|
+
7. No mutation occurred after the final applicable review and execution.
|
|
260
270
|
|
|
261
|
-
##
|
|
271
|
+
## Terminal Results
|
|
262
272
|
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
- testing-domain verdict;
|
|
266
|
-
- evidence directory and ID;
|
|
267
|
-
- validator result and key execution pointers;
|
|
268
|
-
- sensitivity and mocked-versus-real warning;
|
|
269
|
-
- authoritative evidence pointers for external automation and interactive consumers.
|
|
270
|
-
|
|
271
|
-
Do not embed an independent AI reviewer, decide overall code quality, claim broad Spec compliance, repair a validly detected business defect, authorize commit/release, or perform deployed test/production validation. The derived report is a view; `verdict.json`, validation records, and the evidence manifest remain authoritative.
|
|
273
|
+
Only these testing-domain outcomes are valid:
|
|
272
274
|
|
|
273
|
-
|
|
275
|
+
```text
|
|
276
|
+
TEST_PASS | TEST_NEEDS_FIXES | TEST_BLOCKED
|
|
277
|
+
```
|
|
274
278
|
|
|
275
|
-
|
|
279
|
+
- `TEST_PASS`: final reviewed state passes every required native test and no high-risk repair awaits review.
|
|
280
|
+
- `TEST_NEEDS_FIXES`: a known correction remains, a review/repair budget did not converge, or a completed high-risk repair requires delta Code Review.
|
|
281
|
+
- `TEST_BLOCKED`: expected truth, safe input, environment, execution reliability, external-target safety, or required review input prevents a safe verdict.
|
|
276
282
|
|
|
277
|
-
|
|
283
|
+
No conditional pass exists.
|
|
278
284
|
|
|
279
|
-
##
|
|
285
|
+
## Terminal Report and Result
|
|
286
|
+
|
|
287
|
+
Read `${SKILL_DIR}/references/test-report-template.md` before finalization. Write `test-report.md` and `test-result.json` together for every terminal outcome.
|
|
288
|
+
|
|
289
|
+
`test-result.json` is a terminal projection, not a checkpoint:
|
|
290
|
+
|
|
291
|
+
```json
|
|
292
|
+
{
|
|
293
|
+
"schema_version": 1,
|
|
294
|
+
"result": "TEST_PASS",
|
|
295
|
+
"report": "test-report.md",
|
|
296
|
+
"main_review_rounds": 3,
|
|
297
|
+
"independent_review": {
|
|
298
|
+
"status": "completed",
|
|
299
|
+
"responses": 2,
|
|
300
|
+
"downgrade_reason": null,
|
|
301
|
+
"final_state_rechecked": true
|
|
302
|
+
},
|
|
303
|
+
"repair_rounds": 1,
|
|
304
|
+
"production_changed": false,
|
|
305
|
+
"review_required": false,
|
|
306
|
+
"review_scope": null,
|
|
307
|
+
"unresolved_items": []
|
|
308
|
+
}
|
|
309
|
+
```
|
|
280
310
|
|
|
281
|
-
|
|
311
|
+
Allowed `independent_review.status` values are `completed`, `downgraded`, and `not_applicable`. `completed` requires one through five responses and `final_state_rechecked=true`; `downgraded` requires zero responses, a non-empty reason, and `final_state_rechecked=false`; `not_applicable` requires zero responses, no reason, and `final_state_rechecked=false`. `review_scope` is `delta` only when `review_required=true`; otherwise it is null. The report and JSON must agree on result, review requirement, and unresolved items. If consistency cannot be established, write a truthful `TEST_BLOCKED` result rather than fabricating pass.
|
|
282
312
|
|
|
283
|
-
**HANDOFF:** Return the
|
|
313
|
+
**HANDOFF:** Return the testing result plus `test-report.md` and `test-result.json` paths to the caller, then stop.
|