gemstack-ai 1.1.2 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/skills/gemstack-plan/SKILL.md +2 -1
- package/.agents/skills/gemstack-qa/SKILL.md +3 -0
- package/.agents/skills/gemstack-ship/SKILL.md +5 -1
- package/.agents/skills/gemstack-spec/SKILL.md +3 -2
- package/.agents/skills/gemstack-tasks/SKILL.md +4 -3
- package/.gemstack/state.json +7 -9
- package/CHANGELOG.md +32 -0
- package/README.md +13 -0
- package/RELEASE_NOTES.md +24 -0
- package/docs/architecture-consistency.md +14 -2
- package/docs/spec-driven-development.md +26 -0
- package/{gemstack-ai-1.1.2.tgz → gemstack-ai-1.2.0.tgz} +0 -0
- package/handoff.md +28 -15
- package/package.json +2 -2
- package/specs/007-mechanical-test-matrix-closure-evidence/.gemstack.json +5 -0
- package/specs/007-mechanical-test-matrix-closure-evidence/closure.json +59 -0
- package/specs/007-mechanical-test-matrix-closure-evidence/plan.md +484 -0
- package/specs/007-mechanical-test-matrix-closure-evidence/spec.md +597 -0
- package/specs/007-mechanical-test-matrix-closure-evidence/tasks.md +536 -0
- package/specs/templates/plan.md +30 -0
- package/specs/templates/spec.md +18 -0
- package/specs/templates/tasks.md +9 -0
- package/src/cli.js +6 -0
- package/src/commands/collect.js +340 -0
- package/src/commands/ship.js +79 -0
- package/src/commands/verify.js +128 -6
- package/src/lib/closure-context.js +444 -0
- package/src/lib/runner-adapters.js +347 -0
- package/src/lib/test-matrix.js +187 -0
|
@@ -0,0 +1,597 @@
|
|
|
1
|
+
# Especificación de Funcionalidad: Mechanical Test Matrix & Closure Evidence (Upgrade B)
|
|
2
|
+
|
|
3
|
+
**Feature Branch**: `007-mechanical-test-matrix-closure-evidence`
|
|
4
|
+
**Feature Directory**: `specs/007-mechanical-test-matrix-closure-evidence/`
|
|
5
|
+
**Lifecycle Status**: `SPEC_COMPLETE`
|
|
6
|
+
**Stop Reason**: `SPEC_COMPLETE_AWAITING_REVIEW`
|
|
7
|
+
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## 1. Primary Objective & Problem Statement
|
|
11
|
+
|
|
12
|
+
Gemstack v1.1.2 operates as a deterministic architecture consistency engine (`FrozenContractRegistry`, cross-phase inheritance, mutation detection, finding fingerprints, sidecar persistence). However, architectural consistency alone does not guarantee mechanical closure truth:
|
|
13
|
+
1. An AI agent can report "25/25 tests passing" when only 11 or 0 tests were actually executed.
|
|
14
|
+
2. AI agents frequently fabricate phantom tests, execute tautological assertions, count tests without asserting their status, or report total counts that diverge mathematically from physical test runner outputs.
|
|
15
|
+
3. Test suite boundaries blur between canonical acceptance requirements (P1), edge-case internal tests, and supporting/sanity tests.
|
|
16
|
+
4. Premature closure (`SHIPPED`) is claimed without verifiable, mechanical evidence linking tasks, physical tests, execution runs, and fresh repository state.
|
|
17
|
+
|
|
18
|
+
**Upgrade B** promotes Gemstack from an *Architecture Consistency Engine* to an **Architecture Consistency and Mechanical Implementation Closure Engine**. It eliminates false closure by establishing a deterministic, verifiable chain of custody:
|
|
19
|
+
```text
|
|
20
|
+
CANONICAL TEST DECLARATION (spec.md)
|
|
21
|
+
→ TEST RUNNER & SUITE MAPPING (plan.md)
|
|
22
|
+
→ TASK EXECUTION & TEST BINDING (tasks.md)
|
|
23
|
+
→ NATIVE RUNNER EXECUTION (node:test / npm test)
|
|
24
|
+
→ STRUCTURED CLOSURE MANIFEST (closure.json)
|
|
25
|
+
→ MECHANICAL CLOSURE GATE (gemstack verify / gemstack ship)
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
---
|
|
29
|
+
|
|
30
|
+
## 2. Invariants & Bootstrap Contracts
|
|
31
|
+
|
|
32
|
+
This feature dogfoods Gemstack contracts and establishes bootstrap architecture invariants:
|
|
33
|
+
|
|
34
|
+
```gemstack-contracts
|
|
35
|
+
[
|
|
36
|
+
{
|
|
37
|
+
"id": "zero-dependency-core",
|
|
38
|
+
"type": "BOOLEAN_INVARIANT",
|
|
39
|
+
"value": true,
|
|
40
|
+
"description": "Upgrade B implementation must introduce zero external production npm dependencies, using Node.js built-ins exclusively."
|
|
41
|
+
},
|
|
42
|
+
{
|
|
43
|
+
"id": "upgrade-b-taxonomy-types",
|
|
44
|
+
"type": "ENUM_SET",
|
|
45
|
+
"values": [
|
|
46
|
+
"CANONICAL_TEST",
|
|
47
|
+
"PHYSICAL_TEST",
|
|
48
|
+
"SUPPORTING_TEST",
|
|
49
|
+
"DECLARED_TEST",
|
|
50
|
+
"EXECUTED_TEST",
|
|
51
|
+
"PASSED_TEST",
|
|
52
|
+
"FAILED_TEST",
|
|
53
|
+
"SKIPPED_TEST",
|
|
54
|
+
"PHANTOM_TEST",
|
|
55
|
+
"ORPHAN_TEST"
|
|
56
|
+
],
|
|
57
|
+
"description": "Canonical test taxonomy and state classifications in Upgrade B."
|
|
58
|
+
},
|
|
59
|
+
{
|
|
60
|
+
"id": "upgrade-b-closure-gate-levels",
|
|
61
|
+
"type": "ENUM_SET",
|
|
62
|
+
"values": [
|
|
63
|
+
"REQUIRED",
|
|
64
|
+
"SUPPLEMENTAL"
|
|
65
|
+
],
|
|
66
|
+
"description": "Strictness levels for mechanical closure gates."
|
|
67
|
+
},
|
|
68
|
+
{
|
|
69
|
+
"id": "upgrade-b-closure-status-values",
|
|
70
|
+
"type": "ENUM_SET",
|
|
71
|
+
"values": [
|
|
72
|
+
"NOT_EVALUATED",
|
|
73
|
+
"BLOCKED",
|
|
74
|
+
"VERIFIED",
|
|
75
|
+
"VERIFIED_WITH_EXCEPTIONS",
|
|
76
|
+
"STALE"
|
|
77
|
+
],
|
|
78
|
+
"description": "Formal mechanical closure states."
|
|
79
|
+
},
|
|
80
|
+
{
|
|
81
|
+
"id": "mechanical-reconciliation-exact",
|
|
82
|
+
"type": "BOOLEAN_INVARIANT",
|
|
83
|
+
"value": true,
|
|
84
|
+
"description": "Total physical executed tests must exactly equal canonical executed tests plus supporting executed tests."
|
|
85
|
+
},
|
|
86
|
+
{
|
|
87
|
+
"id": "closure-manifest-is-derived-only",
|
|
88
|
+
"type": "BOOLEAN_INVARIANT",
|
|
89
|
+
"value": true,
|
|
90
|
+
"description": "closure.json is a derived mechanical evidence snapshot generated by collection, never hand-edited."
|
|
91
|
+
},
|
|
92
|
+
{
|
|
93
|
+
"id": "verify-is-read-only",
|
|
94
|
+
"type": "BOOLEAN_INVARIANT",
|
|
95
|
+
"value": true,
|
|
96
|
+
"description": "gemstack verify is strictly read-only: it validates evidence and never collects, mutates, or regenerates closure.json."
|
|
97
|
+
},
|
|
98
|
+
{
|
|
99
|
+
"id": "git-is-optional-for-closure",
|
|
100
|
+
"type": "BOOLEAN_INVARIANT",
|
|
101
|
+
"value": true,
|
|
102
|
+
"description": "Git metadata is optional enrichment; closure context resolution does not require git repository existence."
|
|
103
|
+
},
|
|
104
|
+
{
|
|
105
|
+
"id": "no-generic-test-runner",
|
|
106
|
+
"type": "BOOLEAN_INVARIANT",
|
|
107
|
+
"value": true,
|
|
108
|
+
"description": "Gemstack does not reimplement a test runner; it interfaces with native runners via standardized adapters."
|
|
109
|
+
},
|
|
110
|
+
{
|
|
111
|
+
"id": "legacy-specs-preserve-compatibility",
|
|
112
|
+
"type": "BOOLEAN_INVARIANT",
|
|
113
|
+
"value": true,
|
|
114
|
+
"description": "Specs without gemstack-test-matrix blocks operate in progressive legacy mode without blocking or breaking backward compatibility."
|
|
115
|
+
}
|
|
116
|
+
]
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
---
|
|
120
|
+
|
|
121
|
+
## 3. Canonical Test Taxonomy & Terminology
|
|
122
|
+
|
|
123
|
+
Upgrade B establishes precise, unambiguous definitions for test inventory and execution:
|
|
124
|
+
|
|
125
|
+
| Term | Definition |
|
|
126
|
+
| :--- | :--- |
|
|
127
|
+
| **`CANONICAL_TEST`** | A formal acceptance test specification defined in `spec.md` with a unique ID (`TEST-CLOSURE-xxx`), layer, category, description, and pass criteria. It represents a non-negotiable contract requirement. |
|
|
128
|
+
| **`PHYSICAL_TEST`** | An actual executable test function in code (e.g. `it(...)`, `test(...)`) identifiable by runner, file path, line number, or canonical annotation. |
|
|
129
|
+
| **`SUPPORTING_TEST`** | A valid physical test in the codebase (e.g., edge case, benchmark, regression helper) that executes during test runs but is not designated as a primary canonical acceptance test. |
|
|
130
|
+
| **`DECLARED_TEST`** | A test ID explicitly listed in specification or plan tables as intended for implementation. |
|
|
131
|
+
| **`EXECUTED_TEST`** | A physical test that was actually invoked and executed by the native test runner during the recorded verification run. |
|
|
132
|
+
| **`PASSED_TEST`** | An executed test that completed with zero assertion failures and zero uncaught exceptions. |
|
|
133
|
+
| **`FAILED_TEST`** | An executed test that threw an assertion error, unhandled rejection, or runner failure. |
|
|
134
|
+
| **`SKIPPED_TEST`** | A test ignored or skipped by runner directives (e.g. `test.skip`). Skipped canonical tests are treated as unfulfilled requirements. |
|
|
135
|
+
| **`PHANTOM_TEST`** | An ID claimed in structured closure evidence as executed or passed, but absent from authoritative runner execution logs. Strictly non-waivable. |
|
|
136
|
+
| **`ORPHAN_TEST`** | A physical test attempting to fulfill a canonical test ID that does not exist in the frozen specification test matrix. |
|
|
137
|
+
|
|
138
|
+
---
|
|
139
|
+
|
|
140
|
+
## 4. Structured Test Matrix Format (`gemstack-test-matrix`)
|
|
141
|
+
|
|
142
|
+
Canonical tests are declared inside `spec.md` using column-0 fenced blocks with tag `gemstack-test-matrix`, formatted as strict JSON:
|
|
143
|
+
|
|
144
|
+
```gemstack-test-matrix
|
|
145
|
+
[
|
|
146
|
+
{
|
|
147
|
+
"id": "TEST-CLOSURE-A01",
|
|
148
|
+
"category": "PARSING",
|
|
149
|
+
"layer": "UNIT",
|
|
150
|
+
"description": "Parses a valid gemstack-test-matrix block from spec.md into structured CanonicalTest objects.",
|
|
151
|
+
"pass_criteria": "Returns array of CanonicalTest objects matching all schema fields.",
|
|
152
|
+
"gate": "REQUIRED"
|
|
153
|
+
},
|
|
154
|
+
{
|
|
155
|
+
"id": "TEST-CLOSURE-A02",
|
|
156
|
+
"category": "PARSING",
|
|
157
|
+
"layer": "UNIT",
|
|
158
|
+
"description": "Emits TEST_MATRIX_PARSE_ERROR on malformed JSON or TEST_MATRIX_DUPLICATE_ID on duplicate IDs.",
|
|
159
|
+
"pass_criteria": "Throws or reports specific Upgrade B error code without using Upgrade A contract error codes.",
|
|
160
|
+
"gate": "REQUIRED"
|
|
161
|
+
},
|
|
162
|
+
{
|
|
163
|
+
"id": "TEST-CLOSURE-A03",
|
|
164
|
+
"category": "PARSING",
|
|
165
|
+
"layer": "UNIT",
|
|
166
|
+
"description": "Ignores non-column-0 or un-tagged fenced blocks cleanly.",
|
|
167
|
+
"pass_criteria": "Returns empty matrix or only parses strict column-0 blocks without throwing.",
|
|
168
|
+
"gate": "REQUIRED"
|
|
169
|
+
},
|
|
170
|
+
{
|
|
171
|
+
"id": "TEST-CLOSURE-B01",
|
|
172
|
+
"category": "SIGNATURE",
|
|
173
|
+
"layer": "UNIT",
|
|
174
|
+
"description": "Computes deterministic SHA-256 acceptanceSignature digest over canonical test matrix.",
|
|
175
|
+
"pass_criteria": "Produces identical 64-char lowercase hex digest regardless of key order in input JSON objects.",
|
|
176
|
+
"gate": "REQUIRED"
|
|
177
|
+
},
|
|
178
|
+
{
|
|
179
|
+
"id": "TEST-CLOSURE-B02",
|
|
180
|
+
"category": "SIGNATURE",
|
|
181
|
+
"layer": "UNIT",
|
|
182
|
+
"description": "Emits ACCEPTANCE_SIGNATURE_MISMATCH when structured matrix disagrees with expected signature on approved artifact.",
|
|
183
|
+
"pass_criteria": "Raises ACCEPTANCE_SIGNATURE_MISMATCH finding distinct from FROZEN_ARTIFACT_CHANGED.",
|
|
184
|
+
"gate": "REQUIRED"
|
|
185
|
+
},
|
|
186
|
+
{
|
|
187
|
+
"id": "TEST-CLOSURE-C01",
|
|
188
|
+
"category": "RECONCILIATION",
|
|
189
|
+
"layer": "UNIT",
|
|
190
|
+
"description": "Verifies exact arithmetic: executed canonical + executed supporting = total physical executed.",
|
|
191
|
+
"pass_criteria": "Returns pass: true when counts match exactly; returns CLOSURE_RECONCILIATION_FAILURE when counts differ.",
|
|
192
|
+
"gate": "REQUIRED"
|
|
193
|
+
},
|
|
194
|
+
{
|
|
195
|
+
"id": "TEST-CLOSURE-C02",
|
|
196
|
+
"category": "RECONCILIATION",
|
|
197
|
+
"layer": "UNIT",
|
|
198
|
+
"description": "Identifies missing or unexecuted canonical tests when required canonical test IDs are not executed.",
|
|
199
|
+
"pass_criteria": "Emits REQUIRED_TEST_NOT_EXECUTED or REQUIRED_TEST_MISSING and sets status BLOCKED.",
|
|
200
|
+
"gate": "REQUIRED"
|
|
201
|
+
},
|
|
202
|
+
{
|
|
203
|
+
"id": "TEST-CLOSURE-C03",
|
|
204
|
+
"category": "RECONCILIATION",
|
|
205
|
+
"layer": "UNIT",
|
|
206
|
+
"description": "Detects phantom tests (claimed tests absent from runner execution traces).",
|
|
207
|
+
"pass_criteria": "Emits non-waivable PHANTOM_TEST finding and halts verification.",
|
|
208
|
+
"gate": "REQUIRED"
|
|
209
|
+
},
|
|
210
|
+
{
|
|
211
|
+
"id": "TEST-CLOSURE-C04",
|
|
212
|
+
"category": "RECONCILIATION",
|
|
213
|
+
"layer": "UNIT",
|
|
214
|
+
"description": "Detects orphan tests (physical tests claiming non-existent canonical IDs).",
|
|
215
|
+
"pass_criteria": "Emits ORPHAN_TEST finding identifying the unregistered canonical ID.",
|
|
216
|
+
"gate": "REQUIRED"
|
|
217
|
+
},
|
|
218
|
+
{
|
|
219
|
+
"id": "TEST-CLOSURE-D01",
|
|
220
|
+
"category": "RUNNER_ADAPTER",
|
|
221
|
+
"layer": "INTEGRATION",
|
|
222
|
+
"description": "Executes node:test via child_process without shell:true and extracts structured results.",
|
|
223
|
+
"pass_criteria": "Successfully parses passed, failed, and skipped test counts from execution output.",
|
|
224
|
+
"gate": "REQUIRED"
|
|
225
|
+
},
|
|
226
|
+
{
|
|
227
|
+
"id": "TEST-CLOSURE-D02",
|
|
228
|
+
"category": "RUNNER_ADAPTER",
|
|
229
|
+
"layer": "INTEGRATION",
|
|
230
|
+
"description": "Extracts canonical test IDs embedded in test title strings or metadata.",
|
|
231
|
+
"pass_criteria": "Correctly associates test outcome with canonical ID TEST-CLOSURE-xxx.",
|
|
232
|
+
"gate": "REQUIRED"
|
|
233
|
+
},
|
|
234
|
+
{
|
|
235
|
+
"id": "TEST-CLOSURE-D03",
|
|
236
|
+
"category": "RUNNER_ADAPTER",
|
|
237
|
+
"layer": "UNIT",
|
|
238
|
+
"description": "Handles runner execution failure or non-zero exit code gracefully.",
|
|
239
|
+
"pass_criteria": "Reports runner failure without crashing Gemstack CLI process.",
|
|
240
|
+
"gate": "REQUIRED"
|
|
241
|
+
},
|
|
242
|
+
{
|
|
243
|
+
"id": "TEST-CLOSURE-E01",
|
|
244
|
+
"category": "TRACEABILITY",
|
|
245
|
+
"layer": "UNIT",
|
|
246
|
+
"description": "Validates unidirectional task binding (TASK -> TEST IDs) and enforces validation_required field.",
|
|
247
|
+
"pass_criteria": "Raises TASK_VALIDATION_MISSING if validation_required: true lacks valid test bindings.",
|
|
248
|
+
"gate": "REQUIRED"
|
|
249
|
+
},
|
|
250
|
+
{
|
|
251
|
+
"id": "TEST-CLOSURE-E02",
|
|
252
|
+
"category": "TRACEABILITY",
|
|
253
|
+
"layer": "UNIT",
|
|
254
|
+
"description": "Confirms every required canonical test is bound to at least one implementation task.",
|
|
255
|
+
"pass_criteria": "Raises UNMAPPED_CANONICAL_TEST finding if a canonical test has no associated task.",
|
|
256
|
+
"gate": "REQUIRED"
|
|
257
|
+
},
|
|
258
|
+
{
|
|
259
|
+
"id": "TEST-CLOSURE-F01",
|
|
260
|
+
"category": "MANIFEST",
|
|
261
|
+
"layer": "UNIT",
|
|
262
|
+
"description": "Generates valid closure.json matching schema with complete execution and context metadata.",
|
|
263
|
+
"pass_criteria": "Serialized JSON matches closure manifest specification with nullable Git fields for non-Git setups.",
|
|
264
|
+
"gate": "REQUIRED"
|
|
265
|
+
},
|
|
266
|
+
{
|
|
267
|
+
"id": "TEST-CLOSURE-F02",
|
|
268
|
+
"category": "MANIFEST",
|
|
269
|
+
"layer": "UNIT",
|
|
270
|
+
"description": "Computes deterministic structured closureContextHash across clean Git, dirty Git, and non-Git setups.",
|
|
271
|
+
"pass_criteria": "Produces stable SHA-256 digest from canonical JSON object containing repository, phase, signature, and test files context.",
|
|
272
|
+
"gate": "REQUIRED"
|
|
273
|
+
},
|
|
274
|
+
{
|
|
275
|
+
"id": "TEST-CLOSURE-F03",
|
|
276
|
+
"category": "MANIFEST",
|
|
277
|
+
"layer": "UNIT",
|
|
278
|
+
"description": "Verifies read-only detection of stale evidence when relevant bound files change after manifest generation.",
|
|
279
|
+
"pass_criteria": "gemstack verify marks status STALE and emits CLOSURE_EVIDENCE_STALE without modifying closure.json.",
|
|
280
|
+
"gate": "REQUIRED"
|
|
281
|
+
},
|
|
282
|
+
{
|
|
283
|
+
"id": "TEST-CLOSURE-G01",
|
|
284
|
+
"category": "GATES",
|
|
285
|
+
"layer": "INTEGRATION",
|
|
286
|
+
"description": "Blocks transition to SHIPPED when any REQUIRED closure gate fails or evidence is STALE.",
|
|
287
|
+
"pass_criteria": "Ship command halts with blocker error and does not mutate lifecycle to SHIPPED.",
|
|
288
|
+
"gate": "REQUIRED"
|
|
289
|
+
},
|
|
290
|
+
{
|
|
291
|
+
"id": "TEST-CLOSURE-G02",
|
|
292
|
+
"category": "GATES",
|
|
293
|
+
"layer": "INTEGRATION",
|
|
294
|
+
"description": "Allows transition to SHIPPED when all REQUIRED gates pass (status VERIFIED or policy-permitted VERIFIED_WITH_EXCEPTIONS).",
|
|
295
|
+
"pass_criteria": "Ship command proceeds successfully when verified evidence is fresh and valid.",
|
|
296
|
+
"gate": "REQUIRED"
|
|
297
|
+
},
|
|
298
|
+
{
|
|
299
|
+
"id": "TEST-CLOSURE-H01",
|
|
300
|
+
"category": "LEGACY",
|
|
301
|
+
"layer": "UNIT",
|
|
302
|
+
"description": "Preserves backward compatibility for specs without gemstack-test-matrix blocks.",
|
|
303
|
+
"pass_criteria": "Verification passes in legacy mode with informational notice and zero errors.",
|
|
304
|
+
"gate": "REQUIRED"
|
|
305
|
+
}
|
|
306
|
+
]
|
|
307
|
+
```
|
|
308
|
+
|
|
309
|
+
### Schema Attributes for Test Matrix Entries
|
|
310
|
+
- `id` (*string*, required): Unique canonical identifier matching regex `^TEST-[A-Z0-9]+-[A-Z0-9]+$`.
|
|
311
|
+
- `category` (*string*, required): Functional group matching specification sections.
|
|
312
|
+
- `layer` (*enum*, required): `"UNIT"` | `"INTEGRATION"` | `"E2E"` | `"CLI"`.
|
|
313
|
+
- `description` (*string*, required): Human-readable scope and intent of the test.
|
|
314
|
+
- `pass_criteria` (*string*, required): Explicit, deterministic conditions under which the test passes.
|
|
315
|
+
- `gate` (*enum*, required): `"REQUIRED"` (blocks shipping) | `"SUPPLEMENTAL"` (advisory/informational).
|
|
316
|
+
|
|
317
|
+
### Test Matrix Error Taxonomy (Upgrade B)
|
|
318
|
+
Upgrade B does not reuse Upgrade A `CONTRACT_PARSE_ERROR`. It defines a dedicated parsing error family:
|
|
319
|
+
- `TEST_MATRIX_PARSE_ERROR`: Malformed JSON or malformed column-0 fenced block.
|
|
320
|
+
- `TEST_MATRIX_DUPLICATE_ID`: Duplicate canonical test ID declared within the matrix.
|
|
321
|
+
- `TEST_MATRIX_INVALID_SHAPE`: Missing required attributes, invalid enum values, or regex mismatch.
|
|
322
|
+
|
|
323
|
+
### Cryptographic Content Digest (`acceptanceSignature`)
|
|
324
|
+
To prevent unauthorized additions, deletions, or modifications to canonical test requirements across phases:
|
|
325
|
+
$$\text{acceptanceSignature} = \text{SHA-256}(\text{normalized_canonical_test_matrix_json})$$
|
|
326
|
+
The `acceptanceSignature` is a deterministic SHA-256 content digest (not an asymmetric digital signature) calculated over the LF-normalized, key-sorted canonical array and is locked into phase hashes and `.gemstack/state.json` when `spec.md` is frozen.
|
|
327
|
+
|
|
328
|
+
---
|
|
329
|
+
|
|
330
|
+
## 5. Matrix Distribution Across Phases (Traceability & Separation of Concerns)
|
|
331
|
+
|
|
332
|
+
| Phase Artifact | Role & Responsibility | Matrix Element |
|
|
333
|
+
| :--- | :--- | :--- |
|
|
334
|
+
| **`spec.md`** | **Declares** what must be accepted. Owns canonical test IDs, criteria, categories, gates, and the `acceptanceSignature`. | Canonical Test Matrix (`gemstack-test-matrix`) |
|
|
335
|
+
| **`plan.md`** | **Maps** canonical tests to technical implementation strategies, runner engines, physical file targets, and directory structures. | Test Execution Plan & Suite Mappings |
|
|
336
|
+
| **`tasks.md`** | **Binds** specific actionable implementation tasks to canonical test IDs (`validation_required: true`). | Task ↔ Test Traceability Mapping |
|
|
337
|
+
| **`src/` & `tests/`** | **Implements** physical test code asserting the criteria. | Physical tests annotated or mapped to test IDs |
|
|
338
|
+
| **`closure.json`** | **Records** mechanical proof of execution, reconciliation, and pass/fail counts. | Generated Closure Manifest |
|
|
339
|
+
|
|
340
|
+
### Task Binding Direction & `validation_required` Semantics
|
|
341
|
+
- The canonical binding is strictly unidirectional: `TASK → TEST IDs`.
|
|
342
|
+
- Reverse mapping (`TEST → TASKS`) is derived mechanically by the verification engine and must never be stored manually.
|
|
343
|
+
- Every implementation task must declare `validation_required: true | false`.
|
|
344
|
+
- If `validation_required: true` and no valid canonical or supporting test binding is present, `gemstack verify` emits `TASK_VALIDATION_MISSING` as a blocker.
|
|
345
|
+
- Non-functional tasks (e.g. documentation, scaffolding) may declare `validation_required: false`.
|
|
346
|
+
|
|
347
|
+
---
|
|
348
|
+
|
|
349
|
+
## 6. Mechanical Reconciliation & Set Semantics
|
|
350
|
+
|
|
351
|
+
Gemstack strictly enforces exact arithmetic reconciliation and set equality:
|
|
352
|
+
|
|
353
|
+
$$\text{Total Physical Tests Executed} = \text{Canonical Tests Executed} + \text{Supporting Tests Executed}$$
|
|
354
|
+
|
|
355
|
+
### Set Semantics for Closure:
|
|
356
|
+
1. **Exact Set Equality**:
|
|
357
|
+
$$\text{EXPECTED\_REQUIRED\_CANONICAL\_IDS} == \text{EXECUTED\_REQUIRED\_CANONICAL\_IDS}$$
|
|
358
|
+
- A required canonical test ID that exists in code but was not executed is `REQUIRED_TEST_NOT_EXECUTED`.
|
|
359
|
+
- A required canonical test ID with no physical test implementation is `REQUIRED_TEST_MISSING`.
|
|
360
|
+
2. **Zero Failures**:
|
|
361
|
+
$$\text{Failed Tests} = 0 \quad \land \quad \text{Skipped Required Tests} = 0$$
|
|
362
|
+
3. **No Phantoms** (`NO_PHANTOM_TESTS`):
|
|
363
|
+
Every reported canonical test must map 1:1 to an actual physical test with a verified execution event in native runner output.
|
|
364
|
+
4. **No Orphans** (`NO_ORPHAN_CANONICAL_BINDINGS`):
|
|
365
|
+
Every physical test claiming a canonical test ID must reference an ID present in the frozen `gemstack-test-matrix`. Physical tests without canonical IDs are classified as `SUPPORTING_TEST`.
|
|
366
|
+
|
|
367
|
+
If any equation or set condition is violated, `gemstack verify` immediately emits `CLOSURE_RECONCILIATION_FAILURE` or specific taxonomy findings.
|
|
368
|
+
|
|
369
|
+
Additionally:
|
|
370
|
+
1. **Set Inclusion**:
|
|
371
|
+
$$\text{Declared Required Canonical IDs} \subseteq \text{Executed Passed Canonical IDs}$$
|
|
372
|
+
2. **Zero Failures**:
|
|
373
|
+
$$\text{Failed Tests} = 0 \quad \land \quad \text{Skipped Required Tests} = 0$$
|
|
374
|
+
3. **No Phantoms**:
|
|
375
|
+
Every reported canonical test must map 1:1 to an actual physical test with a verified execution event in the runner output.
|
|
376
|
+
4. **No Orphans**:
|
|
377
|
+
Every physical test referencing a canonical test ID must reference an ID present in the frozen `gemstack-test-matrix`.
|
|
378
|
+
|
|
379
|
+
If any formula check fails or counts diverge, `gemstack verify` immediately halts with a `CLOSURE_RECONCILIATION_FAILURE` finding.
|
|
380
|
+
|
|
381
|
+
---
|
|
382
|
+
|
|
383
|
+
## 7. Structured Closure Manifest (`closure.json`)
|
|
384
|
+
|
|
385
|
+
The closure manifest is an evidence snapshot stored at `specs/<feature>/closure.json`. It is a derived artifact produced strictly by the evidence collection step and never hand-authored.
|
|
386
|
+
|
|
387
|
+
### Schema Direction:
|
|
388
|
+
```json
|
|
389
|
+
{
|
|
390
|
+
"$schema": "https://gemstack.dev/schemas/closure-v1.json",
|
|
391
|
+
"version": "1.0",
|
|
392
|
+
"feature_directory": "specs/007-mechanical-test-matrix-closure-evidence/",
|
|
393
|
+
"generated_at": "2026-09-11T12:00:00.000Z",
|
|
394
|
+
"status": "VERIFIED",
|
|
395
|
+
"closure_context": {
|
|
396
|
+
"closure_context_hash": "a1b2c3d4e5f67890abcdef1234567890abcdef1234567890abcdef1234567890",
|
|
397
|
+
"repository_type": "git",
|
|
398
|
+
"git_commit": "26db5c93c39aa2b5a1df97906eb46bc37894c214",
|
|
399
|
+
"working_tree_clean": true,
|
|
400
|
+
"relevant_files_digest": "d1e2f3a4b5c6..."
|
|
401
|
+
},
|
|
402
|
+
"acceptance_signature": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
|
|
403
|
+
"canonical_summary": {
|
|
404
|
+
"required_total": 20,
|
|
405
|
+
"required_passed": 20,
|
|
406
|
+
"supplemental_total": 0,
|
|
407
|
+
"supplemental_passed": 0
|
|
408
|
+
},
|
|
409
|
+
"physical_summary": {
|
|
410
|
+
"supporting_total": 5,
|
|
411
|
+
"supporting_passed": 5,
|
|
412
|
+
"total_executed": 25,
|
|
413
|
+
"total_passed": 25,
|
|
414
|
+
"total_failed": 0,
|
|
415
|
+
"total_skipped": 0
|
|
416
|
+
},
|
|
417
|
+
"reconciliation": {
|
|
418
|
+
"math_valid": true,
|
|
419
|
+
"phantoms_detected": 0,
|
|
420
|
+
"orphans_detected": 0,
|
|
421
|
+
"missing_canonical_ids": []
|
|
422
|
+
},
|
|
423
|
+
"task_traceability_summary": {
|
|
424
|
+
"tasks_total": 17,
|
|
425
|
+
"tasks_with_validation": 15,
|
|
426
|
+
"tasks_documentation_only": 2,
|
|
427
|
+
"unmapped_canonical_tests": []
|
|
428
|
+
},
|
|
429
|
+
"required_gates": {
|
|
430
|
+
"ARCHITECTURE_CONSISTENCY_PASS": "PASS",
|
|
431
|
+
"CANONICAL_MATRIX_VALID": "PASS",
|
|
432
|
+
"ALL_REQUIRED_CANONICAL_MAPPED": "PASS",
|
|
433
|
+
"ALL_REQUIRED_CANONICAL_EXECUTED": "PASS",
|
|
434
|
+
"ALL_REQUIRED_CANONICAL_PASSED": "PASS",
|
|
435
|
+
"NO_PHANTOM_TESTS": "PASS",
|
|
436
|
+
"NO_ORPHAN_CANONICAL_BINDINGS": "PASS",
|
|
437
|
+
"TASK_TRACEABILITY_COMPLETE": "PASS",
|
|
438
|
+
"EXACT_RECONCILIATION_PASS": "PASS",
|
|
439
|
+
"EVIDENCE_FRESHNESS_VERIFIED": "PASS",
|
|
440
|
+
"CLOSURE_CONTEXT_RESOLVED": "PASS",
|
|
441
|
+
"REQUIRED_PROJECT_GATES_PASS": "PASS"
|
|
442
|
+
},
|
|
443
|
+
"supplemental_gates": {},
|
|
444
|
+
"exceptions": [],
|
|
445
|
+
"evidence_sources": [
|
|
446
|
+
{
|
|
447
|
+
"command": "node --test tests/**/*.test.js",
|
|
448
|
+
"runner": "node:test",
|
|
449
|
+
"exit_code": 0,
|
|
450
|
+
"duration_ms": 420,
|
|
451
|
+
"environment": {
|
|
452
|
+
"node_version": "v20.18.0",
|
|
453
|
+
"os": "win32",
|
|
454
|
+
"arch": "x64"
|
|
455
|
+
}
|
|
456
|
+
}
|
|
457
|
+
],
|
|
458
|
+
"blockers": [],
|
|
459
|
+
"warnings": []
|
|
460
|
+
}
|
|
461
|
+
```
|
|
462
|
+
|
|
463
|
+
### Optional Git Metadata & Worktree Cleanness:
|
|
464
|
+
- Git is NOT a mandatory closure authority. If `.git` does not exist, `git_commit: null` and `working_tree_clean: null`.
|
|
465
|
+
- When Git is present, `working_tree_clean: true | false` is recorded as diagnostic metadata.
|
|
466
|
+
- A dirty working tree is NOT an inherent blocker. The closure context simply binds deterministically to the dirty state that was physically tested.
|
|
467
|
+
|
|
468
|
+
---
|
|
469
|
+
|
|
470
|
+
## 8. Deterministic Closure Context & Freshness (`closureContextHash`)
|
|
471
|
+
|
|
472
|
+
Evidence is only valid for the exact project state against which it was collected. To prevent whole-repository scanning, the closure context is formed from a deterministic structured object:
|
|
473
|
+
|
|
474
|
+
```json
|
|
475
|
+
{
|
|
476
|
+
"repositoryContext": "<HEAD_SHA_or_null>",
|
|
477
|
+
"worktreeClean": true,
|
|
478
|
+
"phaseHashes": {
|
|
479
|
+
"spec": "<sha256>",
|
|
480
|
+
"plan": "<sha256>",
|
|
481
|
+
"tasks": "<sha256>"
|
|
482
|
+
},
|
|
483
|
+
"acceptanceSignature": "<sha256>",
|
|
484
|
+
"testFilesHash": "<sha256_of_executed_test_files>",
|
|
485
|
+
"implementationContextHash": "<sha256_of_bound_implementation_files>"
|
|
486
|
+
}
|
|
487
|
+
```
|
|
488
|
+
|
|
489
|
+
$$\text{closureContextHash} = \text{SHA-256}(\text{canonical_json_serialization}(\text{contextObject}))$$
|
|
490
|
+
|
|
491
|
+
### Relevant File Scope Determination:
|
|
492
|
+
- Phase artifacts (`spec.md`, `plan.md`, `tasks.md`).
|
|
493
|
+
- Test matrix `acceptanceSignature`.
|
|
494
|
+
- Declared physical test files identified by runner execution.
|
|
495
|
+
- Implementation-relevant files explicitly derived from `tasks.md` task-to-file bindings (or minimal safe mappings defined in `plan.md`).
|
|
496
|
+
- Entire repository recursive scanning is strictly avoided.
|
|
497
|
+
|
|
498
|
+
### Read-Only Freshness Validation (`gemstack verify`):
|
|
499
|
+
- `gemstack verify` recomputes the current `closureContextHash` in-memory without modifying any files.
|
|
500
|
+
- If the computed hash differs from `closure.json`, the status is `STALE` and verification emits `CLOSURE_EVIDENCE_STALE`.
|
|
501
|
+
- `gemstack verify` NEVER mutates or regenerates `closure.json` to make stale evidence fresh.
|
|
502
|
+
|
|
503
|
+
---
|
|
504
|
+
|
|
505
|
+
## 9. Test Runner Adapter & Execution Architecture
|
|
506
|
+
|
|
507
|
+
1. **Native Runner Integration**: Gemstack interfaces with native runners via standardized adapters (dogfooding built-in `node:test` via `node --test`).
|
|
508
|
+
2. **Zero Dependencies**: Subprocess execution uses Node.js standard library `node:child_process` (`spawn` or `execFile` with explicit argument arrays, preventing shell quoting bugs and glob expansion discrepancies).
|
|
509
|
+
3. **MVP Scope**:
|
|
510
|
+
- `node:test`: Fully supported first-class adapter capturing structured execution results.
|
|
511
|
+
- Generic command exit-code gate: Fully supported for standard build/lint/typecheck commands.
|
|
512
|
+
- Vitest / Playwright: Forward-compatible adapter contracts defined without requiring immediate introspection implementation.
|
|
513
|
+
4. **Trace Extraction**: Structured native output formats (e.g. `--test-reporter=tap` or `--test-reporter=spec`) are parsed for test names and canonical IDs embedded in titles (`[TEST-CLOSURE-xxx]`).
|
|
514
|
+
|
|
515
|
+
---
|
|
516
|
+
|
|
517
|
+
## 10. Operational Command Separation: Collect vs. Verify vs. Ship
|
|
518
|
+
|
|
519
|
+
- **Evidence Collection (`COLLECT`)**: Runs configured test commands, captures runner execution traces, reconciles counts, and generates/regenerates `closure.json`. This responsibility belongs to the pre-ship preparation workflow (e.g. `gemstack ship --collect` or review/qa skills). A new command `gemstack close` is NOT needed at this stage because collection maps cleanly to existing lifecycle flows.
|
|
520
|
+
- **Verification (`VERIFY`)**: `gemstack verify` is strictly read-only (`VERIFY = VALIDATE`). It reads `closure.json`, validates gates, checks freshness against current `closureContextHash`, and reports findings without modifying disk state.
|
|
521
|
+
- **Shipping (`SHIP`)**: Validates that evidence status is `VERIFIED` (or policy-permitted `VERIFIED_WITH_EXCEPTIONS`) and that evidence is fresh, before performing human-approved phase transition.
|
|
522
|
+
|
|
523
|
+
---
|
|
524
|
+
|
|
525
|
+
## 11. Closure Gates & State Transitions
|
|
526
|
+
|
|
527
|
+
### Formal Closure States:
|
|
528
|
+
- `NOT_EVALUATED`: No closure evidence collected yet.
|
|
529
|
+
- `BLOCKED`: One or more REQUIRED gates failed, or evidence is inconsistent/missing.
|
|
530
|
+
- `VERIFIED`: All REQUIRED gates passed with zero exceptions.
|
|
531
|
+
- `VERIFIED_WITH_EXCEPTIONS`: All REQUIRED gates passed, with approved non-blocker exceptions explicitly waived with human approval.
|
|
532
|
+
- `STALE`: Bound source or test files changed after evidence collection.
|
|
533
|
+
|
|
534
|
+
### Required Mechanical Gates (`REQUIRED`):
|
|
535
|
+
1. `ARCHITECTURE_CONSISTENCY_PASS`: Zero open blockers from Upgrade A consistency engine.
|
|
536
|
+
2. `CANONICAL_MATRIX_VALID`: `gemstack-test-matrix` schema, IDs, and syntax valid.
|
|
537
|
+
3. `ALL_REQUIRED_CANONICAL_MAPPED`: Every required test bound to an implementation task.
|
|
538
|
+
4. `ALL_REQUIRED_CANONICAL_EXECUTED`: Exact set equality between expected and executed required IDs.
|
|
539
|
+
5. `ALL_REQUIRED_CANONICAL_PASSED`: All executed required tests passed.
|
|
540
|
+
6. `NO_PHANTOM_TESTS`: Zero claimed tests absent from physical runner output.
|
|
541
|
+
7. `NO_ORPHAN_CANONICAL_BINDINGS`: Zero physical tests claiming undeclared canonical IDs.
|
|
542
|
+
8. `TASK_TRACEABILITY_COMPLETE`: All tasks with `validation_required: true` have valid test bindings.
|
|
543
|
+
9. `EXACT_RECONCILIATION_PASS`: Mathematical count equation balances perfectly.
|
|
544
|
+
10. `EVIDENCE_FRESHNESS_VERIFIED`: `closureContextHash` matches current working tree state.
|
|
545
|
+
11. `CLOSURE_CONTEXT_RESOLVED`: Context hash successfully computed across git or non-git environment.
|
|
546
|
+
12. `REQUIRED_PROJECT_GATES_PASS`: All explicitly configured project commands (e.g. build, lint) exited with code 0.
|
|
547
|
+
|
|
548
|
+
### Required Project Gates Canonical Sources:
|
|
549
|
+
Project-specific required commands must be sourced from trusted configuration:
|
|
550
|
+
- Structured feature metadata (`specs/<feature>/.gemstack.json`).
|
|
551
|
+
- Gemstack configuration file (`.gemstackrc` or `gemstack.config.json`).
|
|
552
|
+
- Explicit trusted `package.json` scripts referenced in `plan.md`.
|
|
553
|
+
|
|
554
|
+
### Accepted Exceptions & Non-Waivable Classes:
|
|
555
|
+
The following blocker classes are strictly **NON-WAIVABLE**:
|
|
556
|
+
- `TEST_MATRIX_PARSE_ERROR`
|
|
557
|
+
- `TEST_MATRIX_DUPLICATE_ID`
|
|
558
|
+
- `TEST_MATRIX_INVALID_SHAPE`
|
|
559
|
+
- `ACCEPTANCE_SIGNATURE_MISMATCH`
|
|
560
|
+
- `PHANTOM_TEST` (fictitious evidence is never acceptable)
|
|
561
|
+
- `CLOSURE_EVIDENCE_STALE`
|
|
562
|
+
- `CLOSURE_EVIDENCE_INCONSISTENT`
|
|
563
|
+
|
|
564
|
+
Potentially waivable only via explicit human approval bound to `contextHash`:
|
|
565
|
+
- Specific `REQUIRED_GATE_FAILED` or `REQUIRED_TEST_SKIPPED` under formal waiver.
|
|
566
|
+
- Waiving an exception does NOT convert a failure to PASS; it marks the feature `VERIFIED_WITH_EXCEPTIONS`.
|
|
567
|
+
|
|
568
|
+
---
|
|
569
|
+
|
|
570
|
+
## 12. Complete Canonical Error and Finding Codes (Upgrade B)
|
|
571
|
+
|
|
572
|
+
Upgrade B defines a dedicated, non-redundant error taxonomy sharing Upgrade A fingerprint primitives:
|
|
573
|
+
- `TEST_MATRIX_PARSE_ERROR`
|
|
574
|
+
- `TEST_MATRIX_DUPLICATE_ID`
|
|
575
|
+
- `TEST_MATRIX_INVALID_SHAPE`
|
|
576
|
+
- `ACCEPTANCE_SIGNATURE_MISMATCH` (matrix signature inconsistent with approved artifact)
|
|
577
|
+
- `REQUIRED_TEST_MISSING` (canonical requirement has no physical implementation)
|
|
578
|
+
- `REQUIRED_TEST_NOT_EXECUTED` (physical test exists but was not invoked in runner run)
|
|
579
|
+
- `REQUIRED_TEST_FAILED` (executed canonical test threw an assertion or error)
|
|
580
|
+
- `REQUIRED_TEST_SKIPPED` (required test bypassed by runner skip directive)
|
|
581
|
+
- `PHANTOM_TEST` (claimed test not found in physical runner output)
|
|
582
|
+
- `ORPHAN_TEST` (physical test claims an undeclared canonical ID)
|
|
583
|
+
- `DUPLICATE_TEST_BINDING` (multiple distinct tasks claiming conflicting ownership)
|
|
584
|
+
- `TASK_VALIDATION_MISSING` (task has `validation_required: true` without test bindings)
|
|
585
|
+
- `UNMAPPED_CANONICAL_TEST` (canonical test has no task bound to it)
|
|
586
|
+
- `CLOSURE_RECONCILIATION_FAILURE` (test count arithmetic equation fails to balance)
|
|
587
|
+
- `REQUIRED_GATE_MISSING` (configured required gate command not found)
|
|
588
|
+
- `REQUIRED_GATE_FAILED` (configured required gate command exited non-zero)
|
|
589
|
+
- `CLOSURE_EVIDENCE_STALE` (code changed after closure manifest was generated)
|
|
590
|
+
- `CLOSURE_EVIDENCE_INCONSISTENT` (closure manifest contents contradict runner evidence)
|
|
591
|
+
|
|
592
|
+
---
|
|
593
|
+
|
|
594
|
+
## 13. Legacy Mode & Progressive Adoption
|
|
595
|
+
|
|
596
|
+
- Specs created under earlier versions (001–006) without a `gemstack-test-matrix` block operate in **Legacy Progressive Mode**.
|
|
597
|
+
- In Legacy Mode, `gemstack verify` issues an informational notice (`LEGACY_SPEC_NO_TEST_MATRIX`) and does not enforce canonical ID reconciliation or block shipping, preserving 100% backward compatibility.
|