valent-pipeline 0.19.25 → 0.19.26
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/package.json
CHANGED
|
@@ -39,7 +39,7 @@
|
|
|
39
39
|
"items": { "type": "string" },
|
|
40
40
|
"description": "Distinctive tokens that MUST appear verbatim in the implementing test code (Step 9b)."
|
|
41
41
|
},
|
|
42
|
-
"required": { "type": "boolean", "description": "Defaults true
|
|
42
|
+
"required": { "type": "boolean", "description": "Defaults true (behavioral case: must FAIL at the red gate, PASS at green). false = exempt from the red-required set (gate warns instead of failing) and skip auto-waived — use for a regression-guard asserting an already-satisfied infrastructure invariant the story must not regress (already green at red-time; would pass on an empty checkout). See steps/qa-a/write-acceptance-tests.md; CRITIC's acceptance audit flags behavioral cases mislabeled required:false." },
|
|
43
43
|
"ac": {
|
|
44
44
|
"type": "array",
|
|
45
45
|
"minItems": 1,
|
|
@@ -21,6 +21,13 @@ For EACH test case in qa-test-spec.md:
|
|
|
21
21
|
2. Are the assertions as strong as the spec requires? (No weakening — present-but-inert tokens, shape-only checks, status-only error checks.)
|
|
22
22
|
3. Is database state verification present where the spec requires it?
|
|
23
23
|
|
|
24
|
+
## Regression-guard honesty (`required: false` cases)
|
|
25
|
+
|
|
26
|
+
For EACH acceptance case marked `required: false` in `qa-test-spec.manifest.json`:
|
|
27
|
+
1. Does it genuinely assert only an **already-satisfied infrastructure invariant** (a config/harness/convention established by a prior story/epic that this story must not regress)? Or is it a **behavioral** case for THIS story's new work, mislabeled non-required to dodge the red gate?
|
|
28
|
+
2. The test of intent: would an empty checkout of this story's branch (no implementation) make the case pass? If yes, it is a legitimate invariant-guard. If it should only pass *after* this story's code lands, it is behavioral and must be `required: true`.
|
|
29
|
+
3. A behavioral case marked `required: false` is a spec cheat — it lets unbuilt behavior ship without ever proving red→green. Flag it HIGH: the red gate is blind to it.
|
|
30
|
+
|
|
24
31
|
## Output
|
|
25
32
|
|
|
26
33
|
Record every finding in the Pass 3 table. Assign preliminary severity.
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
The acceptance test IS the spec and (after the green gate) the proof. You author it **before any implementation exists**, and the pipeline enforces the red-green mechanism around it:
|
|
8
8
|
|
|
9
|
-
- The **RED gate** runs your suite pre-implementation and every required acceptance case must RUN and FAIL. A case that passes against an empty repo is a spec bug (gameable assertion or wrong expectation) and bounces straight back to you.
|
|
9
|
+
- The **RED gate** runs your suite pre-implementation and every *required* acceptance case must RUN and FAIL. A case that passes against an empty repo is a spec bug (gameable assertion or wrong expectation) and bounces straight back to you — **unless** it only asserts an already-satisfied infrastructure invariant, which you mark `required: false` (Step C). Those are regression-guards: exempt from red, still asserted at green.
|
|
10
10
|
- Your test files are then **frozen by hash**. Dev agents cannot edit them — any change auto-REJECTs to you for arbitration. The implementation must come to the tests; the tests never go to the implementation.
|
|
11
11
|
- The **GREEN gate** re-runs the suite post-implementation; red evidence + green evidence + the diff SHA become the story's proof object (`proof.json`).
|
|
12
12
|
|
|
@@ -29,15 +29,24 @@ The Live Smoke Test tables (`api` profile) and UI Integration Smoke rows (`ui` p
|
|
|
29
29
|
|
|
30
30
|
Run the suite once via the configured command. Verify every new test **executes and fails for the behavioral reason you expect** (assertion/404/missing element). Fix syntax/import/collection errors now — `missing-at-red` failures at the gate waste a rework cycle. On greenfield story 1 where the framework cannot run at all yet, note it: the red gate records `mode: weak` and the green gate compensates.
|
|
31
31
|
|
|
32
|
-
## Step C: Manifest entries
|
|
32
|
+
## Step C: Manifest entries — classify each case (behavioral vs invariant-guard)
|
|
33
33
|
|
|
34
|
-
In `qa-test-spec.manifest.json` (Step 10), each acceptance case carries
|
|
34
|
+
In `qa-test-spec.manifest.json` (Step 10), each acceptance case carries a `required` flag. **It is not boilerplate — it encodes whether the case can honestly go red:**
|
|
35
|
+
|
|
36
|
+
- **Behavioral case** (`required: true`, the default) — asserts the NEW behavior this story builds. It MUST fail pre-implementation (an empty/old checkout cannot satisfy it) and pass post-implementation. This is the red→green case.
|
|
37
|
+
- **Invariant-guard case** (`required: false`) — asserts a cross-cutting infrastructure invariant **already established by a prior story/epic** that this story must not regress: e.g. `vitest pool: 'forks'`, a `tests/setup.ts` mock harness, an ESLint rule's presence, ESM `.js` import conventions, an existing directory contract. It is **already green at red-time** (the infra exists on the branch), so forcing it into the red-required set makes red dishonest — the gate rejects it as `premature-pass` and blocks the story with zero implementation done. Mark it `required: false`: still executed and asserted at the GREEN gate (the guard still fires on a regression), but EXEMPT from the red-required set. **Do not delete or weaken the guard — just mark it non-required so red is honest.**
|
|
35
38
|
|
|
36
39
|
```json
|
|
37
40
|
{ "id": "TC-3.4", "kind": "acceptance", "testFile": "tests/acceptance/{story_id}/cards.spec.ts",
|
|
38
41
|
"tier": "e2e", "priority": "P0", "assertionTargets": ["422", "cards.title"], "required": true, "ac": ["AC-3"] }
|
|
42
|
+
{ "id": "TC-6.1", "kind": "acceptance", "testFile": "tests/acceptance/{story_id}/determinism.spec.ts",
|
|
43
|
+
"tier": "integration", "priority": "P1", "assertionTargets": ["forks", "isolate"], "required": false, "ac": ["AC-6"] }
|
|
39
44
|
```
|
|
40
45
|
|
|
46
|
+
The second case is a regression-guard — it asserts a pre-existing determinism/isolation invariant, so it is `required: false`.
|
|
47
|
+
|
|
48
|
+
**Litmus test:** if an empty checkout of this story's branch (no implementation) would make the case PASS, it is an invariant-guard → `required: false`. Everything testing what *this* story adds stays `required: true`. Misclassifying a behavioral case as a guard to dodge red is a spec cheat — CRITIC's acceptance audit checks for it, so the only thing you buy by cheating is a rejection.
|
|
49
|
+
|
|
41
50
|
plus the manifest-level `"files": [...]` list naming EVERY file you wrote (tests + helpers). Cases the dev should implement as unit/integration tests keep `kind: "unit-spec"` (or omit `kind`) — those remain specs, and the dev writes them.
|
|
42
51
|
|
|
43
52
|
## Step D: What remains prose
|