prizmkit 1.1.140 → 1.1.142

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (83) hide show
  1. package/bundled/VERSION.json +3 -3
  2. package/bundled/dev-pipeline/README.md +1 -1
  3. package/bundled/dev-pipeline/prizmkit_runtime/checkpoint_state.py +165 -125
  4. package/bundled/dev-pipeline/prizmkit_runtime/runner_classification.py +13 -1
  5. package/bundled/dev-pipeline/prizmkit_runtime/runner_models.py +0 -6
  6. package/bundled/dev-pipeline/prizmkit_runtime/runners.py +5 -3
  7. package/bundled/dev-pipeline/prizmkit_runtime/test_result.py +160 -0
  8. package/bundled/dev-pipeline/scripts/check-session-status.py +11 -1
  9. package/bundled/dev-pipeline/scripts/generate-bootstrap-prompt.py +23 -806
  10. package/bundled/dev-pipeline/scripts/generate-bugfix-prompt.py +6 -8
  11. package/bundled/dev-pipeline/scripts/generate-recovery-prompt.py +1 -1
  12. package/bundled/dev-pipeline/scripts/generate-refactor-prompt.py +6 -8
  13. package/bundled/dev-pipeline/scripts/prompt_framework.py +68 -56
  14. package/bundled/dev-pipeline/scripts/update-bug-status.py +21 -2
  15. package/bundled/dev-pipeline/scripts/update-checkpoint.py +52 -131
  16. package/bundled/dev-pipeline/scripts/update-feature-status.py +21 -2
  17. package/bundled/dev-pipeline/scripts/update-refactor-status.py +21 -2
  18. package/bundled/dev-pipeline/templates/bootstrap-prompt.md +3 -3
  19. package/bundled/dev-pipeline/templates/bootstrap-tier3.md +8 -19
  20. package/bundled/dev-pipeline/templates/bugfix-bootstrap-prompt.md +6 -6
  21. package/bundled/dev-pipeline/templates/refactor-bootstrap-prompt.md +5 -5
  22. package/bundled/dev-pipeline/templates/sections/bugfix-phase-commit-report.md +1 -1
  23. package/bundled/dev-pipeline/templates/sections/bugfix-phase-diagnose-plan.md +1 -1
  24. package/bundled/dev-pipeline/templates/sections/bugfix-phase-manual-verification.md +1 -1
  25. package/bundled/dev-pipeline/templates/sections/bugfix-phase-review.md +1 -1
  26. package/bundled/dev-pipeline/templates/sections/bugfix-phase-test.md +13 -10
  27. package/bundled/dev-pipeline/templates/sections/bugfix-reminders.md +1 -1
  28. package/bundled/dev-pipeline/templates/sections/checkpoint-system.md +18 -9
  29. package/bundled/dev-pipeline/templates/sections/headless-commit-authorization.md +1 -1
  30. package/bundled/dev-pipeline/templates/sections/phase-commit-full.md +1 -1
  31. package/bundled/dev-pipeline/templates/sections/phase-commit.md +1 -1
  32. package/bundled/dev-pipeline/templates/sections/phase-context-snapshot-agent-suffix.md +2 -2
  33. package/bundled/dev-pipeline/templates/sections/phase-context-snapshot-base.md +1 -1
  34. package/bundled/dev-pipeline/templates/sections/phase-context-snapshot-lite-suffix.md +1 -1
  35. package/bundled/dev-pipeline/templates/sections/phase-implement-lite.md +1 -1
  36. package/bundled/dev-pipeline/templates/sections/phase-prizmkit-test.md +14 -26
  37. package/bundled/dev-pipeline/templates/sections/phase-review-full.md +2 -2
  38. package/bundled/dev-pipeline/templates/sections/phase-specify-plan-full.md +3 -3
  39. package/bundled/dev-pipeline/templates/sections/refactor-phase-review.md +1 -1
  40. package/bundled/dev-pipeline/templates/sections/refactor-phase-test.md +13 -10
  41. package/bundled/dev-pipeline/templates/sections/refactor-reminders.md +1 -1
  42. package/bundled/dev-pipeline/templates/session-status-schema.json +1 -1
  43. package/bundled/dev-pipeline/tests/test_checkpoint_state.py +494 -413
  44. package/bundled/dev-pipeline/tests/test_generate_bootstrap_prompt.py +144 -684
  45. package/bundled/dev-pipeline/tests/test_generate_bugfix_prompt.py +20 -41
  46. package/bundled/dev-pipeline/tests/test_generate_refactor_prompt.py +24 -24
  47. package/bundled/dev-pipeline/tests/test_python_runner_parity.py +176 -36
  48. package/bundled/skills/_metadata.json +6 -6
  49. package/bundled/skills/prizmkit/SKILL.md +13 -18
  50. package/bundled/skills/prizmkit/references/workflow-state-protocol.md +73 -80
  51. package/bundled/skills/prizmkit-code-review/SKILL.md +7 -7
  52. package/bundled/skills/prizmkit-code-review/references/workflow-state-protocol.md +73 -80
  53. package/bundled/skills/prizmkit-committer/SKILL.md +8 -7
  54. package/bundled/skills/prizmkit-committer/references/workflow-state-protocol.md +73 -80
  55. package/bundled/skills/prizmkit-implement/SKILL.md +3 -2
  56. package/bundled/skills/prizmkit-implement/references/workflow-state-protocol.md +73 -80
  57. package/bundled/skills/prizmkit-plan/SKILL.md +2 -1
  58. package/bundled/skills/prizmkit-plan/references/workflow-state-protocol.md +73 -80
  59. package/bundled/skills/prizmkit-retrospective/SKILL.md +8 -6
  60. package/bundled/skills/prizmkit-retrospective/references/workflow-state-protocol.md +73 -80
  61. package/bundled/skills/prizmkit-test/SKILL.md +217 -189
  62. package/bundled/skills/prizmkit-test/references/external-contract-mock-guidance.md +119 -0
  63. package/bundled/skills/prizmkit-test/references/independent-test-review.md +269 -0
  64. package/bundled/skills/prizmkit-test/references/test-coverage-model.md +193 -0
  65. package/bundled/skills/prizmkit-test/references/test-report-template.md +164 -79
  66. package/bundled/skills/prizmkit-workflow/SKILL.md +28 -36
  67. package/bundled/skills/prizmkit-workflow/references/workflow-state-protocol.md +73 -80
  68. package/package.json +1 -1
  69. package/bundled/skills/prizmkit-test/assets/authoritative-records.schema.json +0 -477
  70. package/bundled/skills/prizmkit-test/assets/behavior-risk-matrix.schema.json +0 -116
  71. package/bundled/skills/prizmkit-test/assets/evidence-manifest.schema.json +0 -111
  72. package/bundled/skills/prizmkit-test/assets/evidence-package-template.json +0 -66
  73. package/bundled/skills/prizmkit-test/references/boundary-coverage-protocol.md +0 -104
  74. package/bundled/skills/prizmkit-test/references/contract-mock-protocol.md +0 -65
  75. package/bundled/skills/prizmkit-test/references/evidence-protocol.md +0 -208
  76. package/bundled/skills/prizmkit-test/references/evidence-request-protocol.md +0 -80
  77. package/bundled/skills/prizmkit-test/references/examples.md +0 -100
  78. package/bundled/skills/prizmkit-test/references/service-boundary-test-catalog.md +0 -228
  79. package/bundled/skills/prizmkit-test/references/test-generation-steps.md +0 -118
  80. package/bundled/skills/prizmkit-test/references/trusted-evidence-execution.md +0 -107
  81. package/bundled/skills/prizmkit-test/references/workflow-state-protocol.md +0 -181
  82. package/bundled/skills/prizmkit-test/scripts/build_test_evidence.py +0 -1930
  83. package/bundled/skills/prizmkit-test/scripts/validate_test_evidence.py +0 -1847
@@ -1,111 +0,0 @@
1
- {
2
- "$schema": "https://json-schema.org/draft/2020-12/schema",
3
- "$id": "https://prizmkit.dev/schemas/test-evidence-manifest.schema.json",
4
- "title": "PrizmKit Test Evidence Manifest",
5
- "type": "object",
6
- "additionalProperties": false,
7
- "required": [
8
- "protocol_version",
9
- "evidence_id",
10
- "evidence_id_inputs",
11
- "sensitivity",
12
- "environment_claim",
13
- "change_class",
14
- "baseline_commit",
15
- "working_diff_sha256",
16
- "target_hashes",
17
- "stages",
18
- "files",
19
- "final_verdict"
20
- ],
21
- "properties": {
22
- "protocol_version": { "const": "1.0" },
23
- "evidence_id": { "type": "string", "pattern": "^[a-f0-9]{64}$" },
24
- "evidence_id_inputs": {
25
- "type": "object",
26
- "additionalProperties": false,
27
- "required": ["baseline_commit", "working_diff_sha256", "scope_sha256"],
28
- "properties": {
29
- "baseline_commit": { "type": "string", "minLength": 1 },
30
- "working_diff_sha256": { "$ref": "#/$defs/sha256" },
31
- "scope_sha256": { "$ref": "#/$defs/sha256" }
32
- }
33
- },
34
- "sensitivity": { "const": "project-controlled" },
35
- "environment_claim": { "const": "mocked-code-level-only" },
36
- "change_class": { "enum": ["behavior", "lightweight"] },
37
- "baseline_commit": { "type": "string", "minLength": 1 },
38
- "working_diff_sha256": { "$ref": "#/$defs/sha256" },
39
- "target_hashes": {
40
- "type": "object",
41
- "additionalProperties": false,
42
- "required": ["source", "tests", "contracts", "lockfiles", "environment", "plan"],
43
- "properties": {
44
- "source": { "$ref": "#/$defs/sha256" },
45
- "tests": { "$ref": "#/$defs/sha256" },
46
- "contracts": { "$ref": "#/$defs/sha256" },
47
- "lockfiles": { "$ref": "#/$defs/sha256" },
48
- "environment": { "$ref": "#/$defs/sha256" },
49
- "plan": { "$ref": "#/$defs/sha256" }
50
- }
51
- },
52
- "stages": {
53
- "type": "array",
54
- "minItems": 9,
55
- "items": { "$ref": "#/$defs/stage" }
56
- },
57
- "files": {
58
- "type": "array",
59
- "minItems": 1,
60
- "items": { "$ref": "#/$defs/file" }
61
- },
62
- "final_verdict": { "enum": ["TEST_PASS", "TEST_FAIL", "TEST_BLOCKED"] },
63
- "compatibility": { "const": "legacy-test-report-interface-not-supported" }
64
- },
65
- "$defs": {
66
- "sha256": { "type": "string", "pattern": "^[a-f0-9]{64}$" },
67
- "stage": {
68
- "type": "object",
69
- "additionalProperties": false,
70
- "required": ["name", "status", "outputs"],
71
- "properties": {
72
- "name": {
73
- "enum": [
74
- "CHANGE_CLASSIFY",
75
- "SCOPE_DISCOVER",
76
- "CONTRACT_MODEL",
77
- "TEST_PLAN",
78
- "INFRA_READY",
79
- "TEST_BUILD",
80
- "EXECUTE_PROVE",
81
- "EVIDENCE_PACKAGE",
82
- "EVIDENCE_VALIDATE"
83
- ]
84
- },
85
- "status": { "enum": ["complete", "pending", "not_applicable"] },
86
- "outputs": { "type": "array", "items": { "type": "string", "minLength": 1 } },
87
- "not_applicable": {
88
- "type": ["object", "null"],
89
- "properties": {
90
- "rationale": { "type": "string", "minLength": 8 },
91
- "evidence": { "type": "array", "minItems": 1, "items": { "type": "object" } },
92
- "considered_signals": { "type": "array", "minItems": 1, "items": { "type": "string", "minLength": 1 } },
93
- "conflicts": { "type": "array", "items": { "type": "object" } }
94
- },
95
- "required": ["rationale", "evidence", "considered_signals", "conflicts"],
96
- "additionalProperties": false
97
- }
98
- }
99
- },
100
- "file": {
101
- "type": "object",
102
- "additionalProperties": false,
103
- "required": ["path", "sha256", "produced_by"],
104
- "properties": {
105
- "path": { "type": "string", "minLength": 1 },
106
- "sha256": { "$ref": "#/$defs/sha256" },
107
- "produced_by": { "type": "string", "minLength": 1 }
108
- }
109
- }
110
- }
111
- }
@@ -1,66 +0,0 @@
1
- {
2
- "protocol_version": "1.0",
3
- "lifecycle_commands": [
4
- "init",
5
- "capture-change",
6
- "inventory",
7
- "prepare-tests",
8
- "execute",
9
- "finalize",
10
- "render-report",
11
- "validate --attest",
12
- "resume"
13
- ],
14
- "generated_tests_policy": "Only added or modified repository-native tests are snapshotted; existing tests have no generated snapshot.",
15
- "records": {
16
- "change_classification": "change-classification.json",
17
- "scope": "scope.json",
18
- "target_inventory": "target-inventory.json",
19
- "environment": "environment.json",
20
- "behavior_risk_matrix": "behavior-risk-matrix.json",
21
- "test_plan": "test-plan.json",
22
- "test_preparation": "test-preparation.json",
23
- "infrastructure_changes": "infrastructure-changes.json",
24
- "executions": "executions.json",
25
- "verdict": "verdict.json",
26
- "validation": "validation.json",
27
- "source_patch": "source-change.patch",
28
- "change_capture": "change-capture.json",
29
- "lifecycle": "lifecycle.json",
30
- "derived_report": "test-report.md"
31
- },
32
- "directories": {
33
- "requests": "requests/",
34
- "receipts": "receipts/",
35
- "raw_outputs": "raw/",
36
- "generated_test_snapshots": "generated-tests/",
37
- "contract_snapshots": "contracts/"
38
- },
39
- "stage_statuses": [
40
- { "name": "CHANGE_CLASSIFY", "status": "pending", "outputs": [] },
41
- { "name": "SCOPE_DISCOVER", "status": "pending", "outputs": [] },
42
- { "name": "CONTRACT_MODEL", "status": "pending", "outputs": [] },
43
- { "name": "TEST_PLAN", "status": "pending", "outputs": [] },
44
- { "name": "INFRA_READY", "status": "pending", "outputs": [] },
45
- { "name": "TEST_BUILD", "status": "pending", "outputs": [] },
46
- { "name": "EXECUTE_PROVE", "status": "pending", "outputs": [] },
47
- { "name": "EVIDENCE_PACKAGE", "status": "pending", "outputs": [] },
48
- { "name": "EVIDENCE_VALIDATE", "status": "pending", "outputs": [] }
49
- ],
50
- "resume": {
51
- "last_valid_stage": null,
52
- "current_target_hashes": {
53
- "source": "<sha256>",
54
- "tests": "<sha256>",
55
- "contracts": "<sha256>",
56
- "lockfiles": "<sha256>",
57
- "environment": "<sha256>",
58
- "plan": "<sha256>"
59
- },
60
- "preserve_prior_executions": true,
61
- "invalidate_downstream_on_input_change": true,
62
- "semantic_review_required": true
63
- },
64
- "required_notice": "Evidence contains complete project-controlled command, environment, and output values. The project owns access control, retention, and upload policy. Production credentials, production services, and destructive real-data operations are prohibited.",
65
- "compatibility": "legacy-test-report-interface-not-supported"
66
- }
@@ -1,104 +0,0 @@
1
- # Behavior-Risk Completeness Protocol
2
-
3
- This protocol replaces interface-count and line-coverage completion with observable behavior/risk evidence. The authoritative matrix is `behavior-risk-matrix.json`, validated against `${SKILL_DIR}/assets/behavior-risk-matrix.schema.json`.
4
-
5
- ## Affected Module Boundary
6
-
7
- Use an explicit project module when available. Otherwise derive a cohesive boundary from files that jointly implement one observable responsibility. Record the boundary decision, every declared module root, and every discovered file. Every changed file must appear in inventory or an evidence-backed exclusion.
8
-
9
- Inventory discovery is a guard against obvious omissions, not a deterministic proof of semantic completeness. Static patterns cannot fully resolve reflection, registration, generated code, runtime configuration, or business concepts spread across unexpected files; preserve those as model judgment or unresolved edges.
10
-
11
- - **Primary Scope:** all observable behavior of the affected module.
12
- - **Regression Ring:** direct callers, consumers, shared contracts, and state dependencies.
13
- - **Unresolved Edges:** dynamic or unprovable coupling, each marked verdict-capable or informational.
14
-
15
- Changed lines locate the module but do not define completeness. A verdict-capable unresolved edge prevents `TEST_PASS`.
16
-
17
- ## Behavior-Risk Matrix
18
-
19
- For each public observable behavior, record:
20
-
21
- | Dimension | Required Content |
22
- |-----------|------------------|
23
- | Public behavior | User/caller-observable capability or guarantee |
24
- | Contract source | Specification, machine-readable contract, acceptance condition, trusted test, caller, or implementation |
25
- | Preconditions | Required identity, state, configuration, or dependency state |
26
- | Inputs | Valid classes, empty/null/type/format variants |
27
- | Boundaries | Contract-defined min/max/exact transition values |
28
- | Outputs | Values, errors, response shape, generated identifiers |
29
- | Side effects | Writes, calls, events, files, messages, or absence of partial effects |
30
- | State transitions | Before/after state and idempotent/retry semantics |
31
- | Error behavior | Validation, authorization, dependency, timeout, and malformed-response outcomes |
32
- | Risks | Functional, boundary, permission, concurrency, idempotency, time, dependency, consumer |
33
-
34
- Every risk cell is:
35
-
36
- - `applicable`: mapped to one or more actual test IDs and execution IDs;
37
- - `not_applicable`: a structured decision containing behavior-specific rationale, typed evidence, considered signals, and explanations for every detected conflict;
38
- - `unresolved`: missing truth, testability, execution, or evidence.
39
-
40
- The model decides applicability from project semantics. The validator checks the structure and obvious signal conflicts only: identity/role/tenant suggests permission risk; locks/shared mutable state/workers suggest concurrency; retry/dedup/idempotency keys suggest idempotency; timeout/TTL/expiry suggests time; clients/contracts/network/storage suggest dependency; callers/consumers/shared contracts suggest consumer risk. A detected signal does not force applicability when the structured conflict explanation and evidence establish why it is irrelevant.
41
-
42
- “Covered by happy path,” “probably fine,” or bare “N/A” is invalid rationale.
43
-
44
- ## Truth Resolution
45
-
46
- Use this precedence and record the consulted sources:
47
-
48
- 1. specification;
49
- 2. machine-readable contract;
50
- 3. acceptance condition;
51
- 4. trusted existing test;
52
- 5. caller/consumer;
53
- 6. current implementation.
54
-
55
- Do not freeze a possible current implementation bug into a characterization test. Conflicting higher-precedence sources that cannot be resolved produce `TEST_BLOCKED`.
56
-
57
- ## Applicable Risk Guidance
58
-
59
- ### Functional and Boundary
60
-
61
- Cover typical/minimal/complex valid inputs; null/empty/wrong type; numeric/string/collection limits; malformed formats; output/error shape; every contract-relevant branch. Do not mechanically test private branches with no observable effect.
62
-
63
- ### Permission
64
-
65
- Apply for identity, role, tenant, ownership, policy, entitlement, secret, or protected-resource behavior. Include missing/invalid identity, wrong owner/tenant, insufficient grant, and default-deny behavior.
66
-
67
- ### Concurrency
68
-
69
- Apply when shared mutable state, locks, optimistic versions, duplicate workers/calls, race-prone caches, or transaction ordering can change observable results. Use deterministic coordination rather than timing sleeps.
70
-
71
- ### Idempotency
72
-
73
- Apply to retries, deduplication keys, create/update/delete repetition, webhook/job replay, token rotation, and state transitions. Assert both result and side-effect count.
74
-
75
- ### Time
76
-
77
- Apply to expiry, schedules, windows, date ranges, TTL, time zones, and clock-sensitive signatures. Control the clock and test exact boundary plus before/after.
78
-
79
- ### Dependency
80
-
81
- Apply to databases, filesystems, queues, providers, clocks/randomness, cross-module adapters, and network clients. Use contract-driven doubles and assert failure mapping, no partial side effects, retry bounds, and cleanup.
82
-
83
- ### Consumer
84
-
85
- Apply when callers depend on return shape, errors, side effects, ordering, generated assets, shared types/contracts, state, or invocation conventions. Regression Ring executions prove these assumptions.
86
-
87
- ## Completion Rule
88
-
89
- The affected module is complete only when:
90
-
91
- 1. every observable behavior is represented;
92
- 2. every applicable risk has tests and reliable execution evidence;
93
- 3. every N/A cell has a concrete rationale;
94
- 4. every necessary behavior is bound to a selected, reliable current-workspace execution;
95
- 5. complete affected-module regression passes;
96
- 6. Regression Ring verification passes;
97
- 7. no verdict-capable unresolved edge remains;
98
- 8. cleanup and deterministic package validation succeed.
99
-
100
- Line, branch, and function coverage are diagnostic signals for possible omitted behavior. They never substitute for matrix evidence and no mechanical threshold is required.
101
-
102
- ## Derived Markdown
103
-
104
- `test-report.md` may render a compact matrix summary, but it is not authoritative. The JSON matrix and hashed test/execution records control the verdict. Editing the report or any linked artifact without regenerating hashes must fail deterministic validation.
@@ -1,65 +0,0 @@
1
- # Contract-Driven Mock Protocol
2
-
3
- Use this protocol for external databases, queues, filesystems, SaaS APIs, browser-facing backends, generated clients, and cross-module endpoints. Code-level evidence is mock-first; linking a real test database or deployed environment is outside `/prizmkit-test`.
4
-
5
- ## Contract Source Precedence
6
-
7
- Use the highest available source:
8
-
9
- 1. Shared machine-readable contract used by producer and consumer: OpenAPI, JSON Schema, Protocol Buffers, GraphQL schema, database schema, AsyncAPI, shared language types, or generated client model.
10
- 2. Versioned contract fixture already stored by the target project.
11
- 3. Locally derived contract fixture built from a named source such as public adapter types, caller expectations, migration schema, provider documentation snapshot, or captured non-production example already authorized for the project.
12
-
13
- Never invent request/response fields independently when a shared source exists. Never derive expected behavior only from the changed implementation when a higher-precedence source disagrees.
14
-
15
- ## Required Contract Snapshot
16
-
17
- Copy the exact contract input into `contracts/` and record:
18
-
19
- - contract ID and version when available;
20
- - source path or derivation source;
21
- - SHA-256;
22
- - producer and consumer/module names;
23
- - derivation command or steps;
24
- - unsupported or ambiguous fields;
25
- - whether it represents an external provider, local module, or persistence boundary.
26
-
27
- If derivation is not reproducible or sources conflict, add an Unresolved Edge and issue `TEST_BLOCKED` when it can affect the verdict.
28
-
29
- ## Building Doubles
30
-
31
- Generate or configure fakes, mocks, fixtures, and mock servers from the contract snapshot. Verify both request and response directions where applicable:
32
-
33
- - required/optional fields and nullability;
34
- - field names, types, enums, formats, and bounds;
35
- - status/error variants;
36
- - stateful sequence or transition rules;
37
- - pagination/cursor semantics;
38
- - idempotency, time, and concurrency behavior;
39
- - malformed, timeout, and dependency-failure variants.
40
-
41
- A mock-success-only double is insufficient. Record the matrix behavior/risk cells each double supports and the tests/executions that exercised them.
42
-
43
- ## Databases and Stateful Services
44
-
45
- Prefer an in-memory fake or isolated disposable service only when it preserves the contract being tested. Do not point code-level tests at production or a user-linked deployed test database. For database adapters, derive the fixture from schema/migrations/shared models and test transaction, constraint, rollback, and error behavior relevant to the affected module.
46
-
47
- Any isolated resource must have a unique identifier, bounded lifetime, and verified cleanup. Cleanup failure produces `TEST_BLOCKED`.
48
-
49
- ## Real-Environment Separation
50
-
51
- Every package and report must state:
52
-
53
- - `environment_claim=mocked-code-level-only`
54
- - Mocked code-level evidence does not verify a real deployed environment.
55
- - Real test-environment, production, credential, network policy, data migration, and operational validation require a separate authorized process.
56
-
57
- Reject a package that claims deployment/environment verification from contract mocks, local fakes, containers, or isolated services.
58
-
59
- ## Prohibitions
60
-
61
- - No production credentials, databases, queues, APIs, object storage, or real user data.
62
- - No destructive operation against external data.
63
- - No independently invented incompatible mock contract.
64
- - No silent contract drift.
65
- - No automatic upload of complete raw evidence; the project owns access, retention, and upload policy.
@@ -1,208 +0,0 @@
1
- # Test Evidence Protocol
2
-
3
- This reference is normative for behavior-changing `/prizmkit-test` runs. Structured JSON is authoritative; `test-report.md` is a derived view. The protocol validates current-workspace test evidence and does not perform historical baseline or mutation testing.
4
-
5
- ## Evidence Target and Identity
6
-
7
- Write packages under `.prizmkit/test/evidence/<evidence-id>/`. Compute `evidence-id` as lowercase SHA-256 of canonical, key-sorted, compact JSON containing exactly:
8
-
9
- - `baseline_commit`
10
- - `working_diff_sha256`
11
- - `scope_sha256`
12
-
13
- Store those values as `evidence_id_inputs` in `manifest.json`. Record separate aggregate hashes for inventoried source, tests, contracts, lockfiles, full environment values, and plan inputs. Hash every evidence file except `manifest.json` in the manifest; do not create a self-referential manifest hash.
14
-
15
- Use two separate integrity domains:
16
-
17
- - **Evidence package:** hash each evidence record through `manifest.json`; any report, receipt, raw output, request, or evidence mutation remains detectable even though evidence output is outside source identity.
18
-
19
- Runtime-only churn must not invalidate a source evidence package. Drift in an inventoried source, test, contract, or lockfile must still fail live inventory validation. Never replace a completed package's stable snapshot identity with a later hash of the mutable parent workspace.
20
-
21
- Hashes and validator receipts provide byte-level binding and drift detection, not cryptographic non-repudiation. A caller with the same OS permissions can replace the runner, requests, project tree, or package before validation. Preserve runner/request hashes and use replay to improve auditability without claiming resistance to a malicious same-privilege producer.
22
-
23
- `source-change.patch` preserves the complete target diff. Snapshot generated/changed tests under `generated-tests/`, contracts under `contracts/`, and raw stdout/stderr under `raw/`. Never replace old execution/output records.
24
-
25
- ## Bounded Protocol Recovery
26
-
27
- Evidence-protocol repair is separate from production/test repair. The builder permits only a bounded recovery budget for semantic-record preflight and package initialization. After the budget is exhausted, return `TEST_BLOCKED`; do not create another versioned evidence package or hand-edit authoritative records. A payload-finalized package remains mutable only until validator attestation; after attestation it is sealed and immutable. `Evidence validation: PASS` means only that the package is structurally valid; the testing verdict must be read separately from `Testing verdict: TEST_PASS | TEST_FAIL | TEST_BLOCKED`.
28
-
29
-
30
- The evidence package is an audit of repository behavior, not a request to invent a parallel test suite. Discover and execute the smallest relevant native tests before adding anything. Add, modify, or delete tests only in the target repository. The builder's `prepare-tests` command derives stable test IDs and change status from canonical Git capture, and snapshots only added or modified repository-native tests. `generated-tests/` is an evidence archive, never an execution workspace; it may be empty when existing tests cover the behavior.
31
-
32
- The canonical package lifecycle is `init → capture-change → inventory → prepare-tests → execute → finalize → validate --attest → resume`. Builder commands own `source-change.patch`, native test preparation, receipts, raw outputs, manifest, verdict, and initial report. The validator alone owns `validation.json` and refreshes the report during attestation. Inventory must consume a fresh capture and exactly preserve its complete changed-file set. Canonical behavior packages require a schema-valid `test-preparation.json` bound byte-for-byte to the builder-generated `test-plan.json` and its hashed semantic request. Evidence identity is derived from the resolved baseline, canonical patch hash, and scope hash; identity-changing resume creates a new package instead of rewriting the old directory.
33
-
34
-
35
- Run exactly these states in order:
36
-
37
- 1. `CHANGE_CLASSIFY`
38
- 2. `SCOPE_DISCOVER`
39
- 3. `CONTRACT_MODEL`
40
- 4. `TEST_PLAN`
41
- 5. `INFRA_READY`
42
- 6. `TEST_BUILD`
43
- 7. `EXECUTE_PROVE`
44
- 8. `EVIDENCE_PACKAGE`
45
- 9. `EVIDENCE_VALIDATE`
46
-
47
- Each manifest stage records `name`, `status`, and output paths. A stage may be `not_applicable` only with a concrete rationale. A later state cannot complete until every applicable predecessor output exists and validates.
48
-
49
- ### CHANGE_CLASSIFY
50
-
51
- Classify `behavior` when executable behavior can change through code, runtime configuration, schemas, migrations, generated runtime assets, lockfiles, or dependency changes. Classify `lightweight` only when the scope is exclusively documentation, comments, or formatting with no generated/runtime behavior effect.
52
-
53
- For lightweight changes, record deterministic checks such as patch classification, parser/format validation, and generated-output consistency. Mark behavior-only stages not applicable with rationale. `TEST_PASS` then means only that the lightweight claim was proven; never describe it as complete runtime behavior evidence.
54
-
55
- Unknown or mixed classification enters the full protocol; unresolved classification becomes `TEST_BLOCKED`.
56
-
57
- ### SCOPE_DISCOVER
58
-
59
- Record:
60
-
61
- - Affected module: explicit module boundary or cohesion-derived files that jointly express one observable responsibility.
62
- - Primary Scope: every public behavior of the affected module, not only changed lines.
63
- - Regression Ring: directly affected callers, consumers, shared contracts, and state dependencies.
64
- - Unresolved Edges: dynamic loading, reflection, generated clients, runtime registration, remote contracts, or coupling that cannot be proven statically.
65
-
66
- Record changed files, declared module roots, contract/lockfile/test discovery, evidence-backed exclusions, and Regression Ring mappings in both scope and inventory records. Every changed file must be inventoried or explicitly excluded. The deterministic validator checks these declared relationships and obvious omissions, but inventory cannot prove that the model discovered every semantic behavior or dynamic edge.
67
-
68
- Every edge records whether it can change the verdict. A verdict-capable unresolved edge prevents `TEST_PASS`.
69
-
70
- ### CONTRACT_MODEL
71
-
72
- Resolve expected truth in this order:
73
-
74
- 1. specifications;
75
- 2. machine-readable contracts;
76
- 3. acceptance conditions;
77
- 4. existing trusted tests;
78
- 5. callers/consumers;
79
- 6. current implementation.
80
-
81
- Record conflicts rather than silently characterizing current behavior. If higher-precedence sources conflict and cannot be resolved, issue `TEST_BLOCKED`.
82
-
83
- Populate `behavior-risk-matrix.json` using `assets/behavior-risk-matrix.schema.json`. Enumerate observable behavior, preconditions, input classes and boundary values, outputs, side effects, state transitions, errors, and every functional, boundary, permission, concurrency, idempotency, time, dependency, and consumer risk. Every risk is applicable, not applicable with concrete rationale, or unresolved.
84
-
85
- Coverage percentages are diagnostic omission signals. They can prompt another matrix review but cannot prove or replace behavior-risk completeness.
86
-
87
- ### TEST_PLAN
88
-
89
- Assess all test layers:
90
-
91
- 1. `focused`
92
- 2. `module-component`
93
- 3. `contract-integration`
94
- 4. `affected-module-regression`
95
- 5. `regression-ring`
96
-
97
- Choose unit, module/component, contract, integration, and code-level E2E tests from actual risk. Do not require every test type mechanically. Every omitted layer needs a concrete not-applicable rationale. If a necessary layer cannot be executed within the configured budget, persist progress and issue `TEST_BLOCKED` until complete.
98
-
99
- ### INFRA_READY
100
-
101
- Discover manifests, existing tests, commands, runner configuration, fixtures, and conventions before changing infrastructure. Reuse an adequate native runner. Install only necessary dependencies after registry/version verification. Create only required configuration, fixtures, fakes, contract-backed mock servers, or isolated services. Never delete existing tests.
102
-
103
- Record every infrastructure change, command, verification, and cleanup responsibility in `infrastructure-changes.json`. Production credentials, production databases, production APIs, and destructive real-data operations are prohibited.
104
-
105
- ### TEST_BUILD
106
-
107
- Fill the whole affected module's observable gaps: public behavior, contracts, boundaries, invariants, failure paths, state transitions, and coupling risks. Do not stop after the changed happy path and do not mechanically exercise unrelated private lines.
108
-
109
- When code has no controllable seam, the test work may make the minimum behavior-preserving production refactor needed for dependency injection, pure-function extraction, adapters, or controllable state. Record the patch and behavior-preservation check. If a public production contract would need to change, stop with `TEST_BLOCKED`.
110
-
111
- A valid test that exposes a business/contract defect is evidence for `TEST_FAIL`. Do not repair the business defect inside this skill.
112
-
113
- ### EXECUTE_PROVE
114
-
115
- Run progressively:
116
-
117
- 1. focused tests;
118
- 2. module/component tests;
119
- 3. contract/integration tests;
120
- 4. complete affected-module regression;
121
- 5. Regression Ring verification.
122
-
123
- Only omit plan-marked non-required layers. Every N/A is structured with rationale, typed evidence, considered signals, and explanations for detected conflicts. The model retains semantic judgment; validation checks structure and obvious signal conflicts, not hidden intent. Append only runner-generated execution receipts containing:
124
-
125
- - runner-generated receipt format, runner hash, request path/hash, and unique execution identifier;
126
- - unique selected execution identifier;
127
- - layer and exact argv;
128
- - confined working directory;
129
- - complete effective environment key/value map used by the command;
130
- - tool/runtime probe results;
131
- - integer exit code (booleans are invalid);
132
- - paths and hashes for complete stdout and stderr;
133
- - reliability/flakiness classification.
134
-
135
- Do not overwrite failed or superseded attempts. Do not retry until green. Use a bounded documented recovery attempt; required flakiness, unavailable infrastructure, unreliable execution, or cleanup failure is `TEST_BLOCKED`.
136
-
137
- For each necessary behavior, validate the current implementation directly in the project root:
138
-
139
- 1. Execute the planned project-native tests for the behavior and its required layer.
140
- 2. Bind the behavior risk to selected, reliable execution receipts and preserve complete raw output.
141
- 3. Treat a current-state pass as valid evidence; this protocol does not claim historical baseline failure or mutation sensitivity.
142
-
143
- TDD history, baseline-failure proof, and mutation testing are outside this skill. Missing behavior mappings, unreliable execution, unavailable infrastructure, or budget truncation prevents `TEST_PASS`.
144
-
145
- ### EVIDENCE_PACKAGE
146
-
147
- Create the complete package from `assets/evidence-package-template.json`, then hash its records. Set:
148
-
149
- - `sensitivity=project-controlled`
150
- - `environment_claim=mocked-code-level-only`
151
- - `compatibility=legacy-test-report-interface-not-supported`
152
-
153
- Preserve complete commands, environment values, stdout, stderr, patches, hashes, and history without automatic redaction. The project owns access control, retention, and upload policy. This does not permit production secrets or services.
154
-
155
- Generate `test-report.md` solely from structured records. Regeneration must produce the same evidence pointers and verdict. The report is not independently editable authority.
156
-
157
- Prepare `verdict.json` and the derived report, then let the validator finalize the non-circular attestation. Callers must not predeclare validator success as evidence.
158
-
159
- ### EVIDENCE_VALIDATE
160
-
161
- Run:
162
-
163
- ```bash
164
- python3 ${SKILL_DIR}/scripts/validate_test_evidence.py \
165
- --evidence-dir .prizmkit/test/evidence/<evidence-id> \
166
- --project-root <target-project-root> \
167
- --attest
168
- ```
169
-
170
- `--attest` first validates the mutable `payload-finalized` package while excluding only validator-owned attestation/view records, atomically writes `validation.json`, marks validation complete, refreshes final file hashes, seals the lifecycle, then immediately performs a complete second validation. Subsequent checks omit `--attest` and verify the stored attestation. This avoids trusting a caller-authored `passed` marker or creating a self-referential manifest. The attestation proves package integrity and protocol consistency only; it is not proof against a same-permission producer.
171
-
172
- The validator checks all shipped JSON schemas, evidence-directory identity, live target-project inventories and aggregate hashes, changed-file/module-root/exclusion/Regression Ring cross-links, patch/diff binding, stage ordering and output ownership, generated-test snapshot existence, planned test/inventory/execution linkage, module/matrix completeness, successful selected runner receipts with request/raw-output binding, structured N/A and obvious signal conflicts, production/unknown external-target safety, strict verdict semantics, and honest mocked-versus-real claims. `TEST_PASS` requires deterministic validator success.
173
-
174
- When feasible, replay recorded requests through `build_test_evidence.py execute --replay-receipt ...`. Replay performs a new real execution and emits a new linked receipt; it does not pretend the attestation establishes what historically ran.
175
-
176
- ## Resume and Invalidation
177
-
178
- On invocation for an existing evidence target:
179
-
180
- 1. Validate manifest identity and every recorded file hash.
181
- 2. Recompute source, tests, contracts, lockfiles, environment, and plan target hashes.
182
- 3. Find the first stage whose input hash or output is invalid.
183
- 4. Preserve all prior executions and raw outputs immutably.
184
- 5. Invalidate that stage and every downstream stage; do not reuse their verdict claims.
185
- 6. Resume from the first invalid stage and write new execution identifiers.
186
-
187
- Stages are idempotent with respect to the same canonical inputs. Never silently truncate a large module. Persist matrix progress and return `TEST_BLOCKED` until every necessary behavior and Regression Ring cell is complete.
188
-
189
- ## Terminal Semantics
190
-
191
- - `TEST_PASS`: all required behavior/risk cells and Regression Ring checks are covered, all required executions are reliable and successful, no verdict-capable unresolved edge remains, code-level requests remain replayable, and deterministic validation passes. This is protocol-validated code-level evidence, not a mathematical proof that no business behavior was omitted.
192
- - `TEST_FAIL`: valid reliable evidence reproduces an implementation or resolved-contract failure. The package must remain complete enough to prove the failure.
193
- - `TEST_BLOCKED`: scope/truth uncertainty, unavailable execution, necessary flakiness, incomplete evidence, failed cleanup, budget truncation, unproven behavior, conflicting truth, or deterministic validation failure.
194
-
195
- No conditional pass exists.
196
-
197
- ## Responsibility Boundary
198
-
199
- Return only the testing-domain verdict and evidence pointers. Do not:
200
-
201
- - embed an independent AI reviewer or adversarial PASS judge;
202
- - decide overall code quality;
203
- - claim broad Spec compliance beyond tested contracts;
204
- - repair a business defect exposed by a valid test;
205
- - authorize commit, release, or deployment;
206
- - claim mocked code-level evidence validates a deployed test or production environment.
207
-
208
- The legacy report/verdict interface is intentionally unsupported. Do not migrate Pipeline or other Skill consumers and do not add a compatibility layer in this change.
@@ -1,80 +0,0 @@
1
- # Evidence Request Protocol
2
-
3
- Load this reference when writing an inventory, execution, resume, or replay request. Validate every authoritative record against `${SKILL_DIR}/assets/authoritative-records.schema.json`; the validator loads the named `$defs` for each file.
4
-
5
- ## Canonical Lifecycle Ownership
6
-
7
- Start every new package with `init`, then run builder-owned `capture-change` before semantic inventory. `capture-change` is the only producer of `source-change.patch`; it records tracked, untracked, deleted, renamed, and copied paths against the resolved baseline. The Main Agent must not create or edit the patch.
8
-
9
- Use existing project-native tests first. The Main Agent supplies only semantic test paths, behavior mappings, selectors, runner hints, and layer decisions. Run `prepare-tests` after inventory: it derives stable file/test-unit IDs and existing/added/modified status from canonical capture, snapshots added/modified repository tests, and writes normalized `test-plan.json`. The Main Agent must not hand-author IDs, status, snapshot paths, hashes, or receipt fields. Inventory `changed_files` must exactly match the fresh canonical capture, and every live captured path must be inventoried or explicitly excluded. Each execution request must use exactly one binding mode: `test_ids`, `test_paths`, or `auto_bind: true`. The builder recomputes command scope; explicit declarations must exactly match it, opaque commands are blocked, recognized package-wide commands cover all prepared tests, and verified requests persist canonical `test_ids` only. Every changed native test must be bound.
10
-
11
- `finalize` creates the manifest, stage dependency hashes, verdict, evidence identity, and deterministic report inputs. `render-report` only renders the derived Markdown view. The validator alone creates `validation.json` during `--attest`; never hand-author a validation attestation or rebind a finalized package after identity inputs change.
12
-
13
-
14
- The model writes project-semantic requests after inspecting manifests, runner configuration, contracts, test conventions, changed files, and module boundaries. The bundled builder owns process execution, raw output, receipts, current-workspace execution and invalidation calculations.
15
-
16
- Never hand-author `executions.json` receipt fields. Never copy a command's claimed output into `raw/`. Invoke the builder so provenance is linked to the actual process.
17
-
18
- ## Producer Boundary and Recovery
19
-
20
- The model supplies schema-shaped semantic requests and mutable semantic records only before payload finalization. The builder performs structural and derived-field preflight before creating a payload-finalized package. Validator attestation then seals it. Once sealed, the returned `evidence_dir`, `manifest_path`, `verdict_path`, and `validation_path` are the only valid handoff paths; do not continue using the old staging directory. A validator pass is not a testing pass: read the testing verdict separately.
21
-
22
-
23
- Place a request under `requests/` with:
24
-
25
- - `request_version: "1.0"`;
26
- - `categories`: project-relative glob arrays for source, tests, contracts, and lockfiles;
27
- - nonempty `changed_files` and declared `module_roots`;
28
- - structured `exclusions`, each with path, rationale, and discovery evidence;
29
- - `discovery_evidence` for manifests, runner configuration, filesystem discovery, contracts, lockfiles, and boundary analysis;
30
- - nonempty `plan_inputs` carrying the semantic planning inputs whose hash controls resume.
31
-
32
- An empty inventory is not evidence of an empty module. Ensure every changed file exists and is either inventoried or explicitly excluded with evidence. The builder independently enumerates every actual file under each model-declared module root; every enumerated file must appear in one inventory category or a structured exclusion. This does not decide whether the model selected the right semantic module boundary, but it prevents silent omission after that boundary is declared.
33
-
34
- ## Execution Request
35
-
36
- Use an argv array so the runner does not interpret a shell string. Required fields are:
37
-
38
- - `request_version`, `purpose`, `command`, `cwd`;
39
- - environment additions (the runner starts from a minimal runtime environment and the receipt records every effective value);
40
- - model-selected `tool_version_commands`;
41
- - selected `layer` and exactly one canonical binding (`test_ids`, `test_paths`, or `auto_bind: true`);
42
- - structured `external_targets`.
43
-
44
- Optional `timeout_seconds`, `attempt_policy`, and `concurrency` remain model-chosen. The builder applies a bounded default and maximum timeout; it does not silently retry or claim concurrency that it did not execute.
45
-
46
- The request path must be beneath the evidence directory. `cwd` resolves beneath the selected execution root. Missing commands produce an actual exit 127 receipt; missing cwd or escaped paths block before execution.
47
-
48
- ## Execution Evidence
49
-
50
- Execute each required native test layer through `build_test_evidence.py execute` in the target project root. The request supplies the project-native argv, cwd, environment additions, tool probes, layer, test binding, timeout, and external-target classification. The builder records the actual process result, complete environment, raw output, hashes, and runner-generated receipt. No historical checkout, mutation patch, test overlay, or separate worktree is created.
51
-
52
- Source drift between capture and validation is detected by independently recomputing the complete canonical Git change (including tracked, untracked, deleted, renamed, and copied paths while excluding reserved runtime roots) and comparing both patch hash and changed-file entries.
53
-
54
- ## Structured N/A
55
-
56
- Use the same structure for omitted layers, not-applicable risk cells, and stage N/A:
57
-
58
- - `rationale`: behavior-specific reasoning;
59
- - `evidence`: one or more typed discovery records;
60
- - `considered_signals`: signals examined before deciding N/A;
61
- - `conflicts`: every detected contradictory signal plus an explanation.
62
-
63
- The model retains semantic judgment. The deterministic validator checks shape and obvious signal conflicts only. For example, auth/tenant/role signals conflict with permission N/A; lock/shared-state/worker signals conflict with concurrency N/A; retry/idempotency/dedup signals conflict with idempotency N/A; timeout/TTL/expiry signals conflict with time N/A; clients/contracts/network signals conflict with dependency N/A; callers/consumers/shared-contract mappings conflict with consumer N/A. A conflict can be accepted only when explicitly recorded and explained.
64
-
65
- ## Record-to-Schema Map
66
-
67
- | Authoritative record | Schema definition |
68
- |---|---|
69
- | `change-classification.json` | `changeClassification` |
70
- | `scope.json` | `scope` |
71
- | `target-inventory.json` | `targetInventory` |
72
- | `test-plan.json` | `testPlan` |
73
- | `requests/*.execution.json` | `executionRequest` |
74
- | receipt entries in `executions.json` | `executionReceipt` / `executionLog` |
75
- | `infrastructure-changes.json` | `infrastructureChanges` |
76
- | `verdict.json` | `verdict` |
77
- | `validation.json` | `validation` |
78
- | `lightweight-verification.json` | `lightweightVerification` |
79
-
80
- `manifest.json` and `behavior-risk-matrix.json` retain their dedicated schemas. All schemas reject unknown fields where the protocol owns the record.