valent-pipeline 0.19.59 → 0.19.61
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/package.json
CHANGED
|
@@ -89,6 +89,10 @@ For each NFR-sensitive path: `[NFR-PERF]` response time + load patterns; `[NFR-S
|
|
|
89
89
|
|
|
90
90
|
**The validating test a WS-driven sanction names MUST be a true end-to-end reachability case — never a service-layer test that bypasses the live event path.** When the UXA spec carries a Step 6b reachability contract for a WS-driven screen, author a REQUIRED `e2e` acceptance case that **drives the real triggering event over the actual connection** (register/login → issue the WS command → assert the screen leaves loading and renders), then sanction the matching visual checkpoint to it. A test that calls the service/handler layer directly (e.g. asserts `POST /api/careers` or the service method) can pass green while the **live WS command path is structurally broken** — a missing initial-state push, or a bootstrap command that violates a pre-event invariant (writing an FK row keyed on an id that does not exist until the event lands). That gap stays invisible until the visual gate a full attempt later. The reachability case is what forces the live path; the service-layer test is necessary but NOT sufficient to back a sanction.
|
|
91
91
|
|
|
92
|
+
**`na` is for genuinely browser-UNDRIVABLE events — NOT a fallback for a screen the MCP merely failed to reach.** Eligible: a WS event NO browser action can trigger on a correctly-built app — a server-INITIATED push (a phase that advances on a timer, a cross-session/another-player event, a connection severed mid-session). NOT eligible: a WS event a USER ACTION triggers — clicking "Start Career" → `career.create` → the in-career screen mounts — is browser-drivable and MUST be verified for real (a real screenshot), never sanctioned. If the MCP "can't drive" such a user-triggered flow, the cause is almost always the APP, not the gate — an unstyled / zero-layout / zero-CSS shell where clicks and fills don't land on the handlers (class-presence passing is not pixel-rendering). The fix is to make the app render and be drivable (a styling/build story), then verify for real — not to `na` the checkpoint away and permanently forfeit its pixel verification. When in doubt between "undrivable" and "app is broken", assume app-is-broken and do NOT sanction.
|
|
93
|
+
|
|
94
|
+
**Sanction a shared prerequisite as a SET — never a subset.** When several checkpoints depend on the SAME genuinely-undrivable WS prerequisite (e.g. every in-career VV checkpoint needs the same `career.created` push), sanction them ALL `N/A (sanction)`, each naming its validating e2e case — or sanction NONE. A partial sanction (some `na`, some left BLOCKED under the identical prerequisite) is itself the defect: the gate runs, the un-sanctioned siblings come back BLOCKED, and JUDGE rejects on them while the `na` ones pass. The Step 6b reachability contract (UXA) enumerates exactly which VV-IDs a prerequisite blocks — sanction precisely that set.
|
|
95
|
+
|
|
92
96
|
## Step 9b: Test Quality Bar — make every case implementable AND falsifiable
|
|
93
97
|
|
|
94
98
|
The dev agents implement exactly what you specify, and CRITIC rejects tests that are weak, unfalsifiable, or bound to the wrong artifact. Spec the bar IN so it is built right the first time (each rule below traces to a real rework cycle):
|
|
@@ -49,8 +49,10 @@ For EACH WS-driven screen, document its **reachability contract**:
|
|
|
49
49
|
| **Initial-state-push payload** | The exact state fields the event MUST push so the screen's loading state resolves to active — name them (e.g. `school_name`, `season`, `current_phase`); "sends state" is not enough |
|
|
50
50
|
| **Pre-event session state** | What exists BEFORE the event (e.g. session keyed on a player/user id, no record yet) — flags bootstrap commands that must NOT assume post-event invariants (FK rows, ids that do not exist until the event lands) |
|
|
51
51
|
| **Unreachable-if** | The concrete failure: which screen hangs in which state if the push (or the bootstrap command) is missing/broken |
|
|
52
|
+
| **Trigger kind** | USER-ACTION (a click/submit sends the WS command, e.g. "Start Career" → `career.create`) or SERVER-INITIATED (a timed phase advance, a cross-session/another-player event, a mid-session disconnect). Decides visual coverage below |
|
|
53
|
+
| **Blocked visual checkpoints** | The exact VV-IDs whose setup needs this prerequisite — so QA-A sanctions precisely that SET, never a subset (Step 9) |
|
|
52
54
|
|
|
53
|
-
Each contract becomes a REQUIRED end-to-end reachability acceptance case in QA-A (drive the real event → assert the screen renders),
|
|
55
|
+
Each contract becomes a REQUIRED end-to-end reachability acceptance case in QA-A (drive the real event → assert the screen renders). **Visual coverage depends on Trigger kind, NOT on "it's WS":** a USER-ACTION-triggered screen IS browser-drivable (the MCP performs the click/fill that sends the command) and MUST be visually verified FOR REAL — do NOT sanction it `N/A`; if the MCP can't drive it the cause is usually an unstyled/zero-layout app, which is a styling/build fix, not a sanction. Only a SERVER-INITIATED event the browser genuinely cannot trigger is `N/A`-eligible (QA-A Step 9), and then the whole blocked-VV set is sanctioned together, each naming its e2e case. A screen reachable via a plain HTTP load only is out of scope — this is for event-driven reachability.
|
|
54
56
|
|
|
55
57
|
## Step 7: Accessibility Checklist (WCAG AA)
|
|
56
58
|
|
|
@@ -12,7 +12,7 @@ import { join } from 'path';
|
|
|
12
12
|
import { randomBytes } from 'crypto';
|
|
13
13
|
import { crosscheckVerdict, formatCrosscheck } from '../lib/crosscheck.js';
|
|
14
14
|
import { extractFencedBlock } from '../lib/handoff.js';
|
|
15
|
-
import { evidenceDir, latestPins, resolveRoot, utcCompact, latestAssert, authenticateBaseline } from '../lib/evidence.js';
|
|
15
|
+
import { evidenceDir, latestPins, resolveRoot, utcCompact, latestAssert, assertPassedCountsForLabel, authenticateBaseline } from '../lib/evidence.js';
|
|
16
16
|
|
|
17
17
|
const GATE_ARTIFACTS = {
|
|
18
18
|
judge: 'judge-decision.md',
|
|
@@ -67,6 +67,11 @@ export async function crosscheckCmd(options) {
|
|
|
67
67
|
// verify — the bare "newest assert" let a later red/acceptance assert shadow the qa-b green one.
|
|
68
68
|
const expect = options.expect === 'red' ? 'red' : 'green';
|
|
69
69
|
const got = latestAssert(evDir, { label: options.label || null, expect });
|
|
70
|
+
// Multi-suite label-multiplexing (2026-06-25): passed-counts of every green assert UNDER THE GATE'S
|
|
71
|
+
// LABEL, so the crosscheck can tell a real sibling-suite count (a run JUDGE legitimately counted)
|
|
72
|
+
// from a fabricated testsPassed. Gated on a label being bound — the unbound/legacy recount stays
|
|
73
|
+
// strict (it must not tolerate a count from an unrelated label). Single-run stories are unaffected.
|
|
74
|
+
const siblingPassedCounts = options.label ? assertPassedCountsForLabel(evDir, { label: options.label, expect }) : [];
|
|
70
75
|
// Baseline authentication feeds the preExistingExcluded rule (review pass-3 #5).
|
|
71
76
|
let baselineAuth = null;
|
|
72
77
|
const baselineCandidates = [join(evDir, 'baseline.json'), join(root, '.valent-pipeline', 'evidence-baseline.json')];
|
|
@@ -89,6 +94,7 @@ export async function crosscheckCmd(options) {
|
|
|
89
94
|
baselineAuth,
|
|
90
95
|
assertRequired: options.requireAssert === true,
|
|
91
96
|
expectedLabel: options.label || null,
|
|
97
|
+
siblingPassedCounts,
|
|
92
98
|
});
|
|
93
99
|
if (malformedInputs.length) {
|
|
94
100
|
result.mismatches.push(...malformedInputs);
|
package/src/lib/crosscheck.js
CHANGED
|
@@ -35,7 +35,7 @@ const OPEN_STATUSES = new Set(['open', 'fix-in-progress']);
|
|
|
35
35
|
* @returns {{ok:boolean, skipped:boolean, mismatches:Array<{field:string, claimed:*, actual:*}>,
|
|
36
36
|
* warnings:string[], recounted:object}}
|
|
37
37
|
*/
|
|
38
|
-
export function crosscheckVerdict({ reported = null, bugsBlock = null, assertVerdict = null, pin = null, diskVerdict = null, baselineAuth = null, assertRequired = false, expectedLabel = null }) {
|
|
38
|
+
export function crosscheckVerdict({ reported = null, bugsBlock = null, assertVerdict = null, pin = null, diskVerdict = null, baselineAuth = null, assertRequired = false, expectedLabel = null, siblingPassedCounts = [] }) {
|
|
39
39
|
const mismatches = [];
|
|
40
40
|
const warnings = [];
|
|
41
41
|
const recounted = {};
|
|
@@ -87,6 +87,18 @@ export function crosscheckVerdict({ reported = null, bugsBlock = null, assertVer
|
|
|
87
87
|
warnings.push(`${field}: self-reported ${reported[field]} but artifacts show ${actual} — under-report (safe direction: under-stating passes cannot sneak a ship), reconciled to the recount`);
|
|
88
88
|
return;
|
|
89
89
|
}
|
|
90
|
+
// Multi-suite label-multiplexing (2026-06-25): when several DISTINCT green suites run under ONE
|
|
91
|
+
// label at one gate (e.g. acceptance + server-project, case sets the record-time auto-namespace
|
|
92
|
+
// did not split), the label-bound recount picks the LATEST assert, which may not be the suite
|
|
93
|
+
// JUDGE counted. A testsPassed OVER-claim that EXACTLY equals some OTHER green same-label
|
|
94
|
+
// assert's passed count is a real machine recount of a sibling run JUDGE counted — not padded
|
|
95
|
+
// green — so reconcile with a warning. A value matching NO real same-label assert stays a hard
|
|
96
|
+
// mismatch (the fabrication case the crosscheck exists to catch). Strictly safer than binding
|
|
97
|
+
// the recount to the run JUDGE *cites* (which would let the gate choose its own recount source).
|
|
98
|
+
if (field === 'testsPassed' && numeric && reported[field] > actual && Array.isArray(siblingPassedCounts) && siblingPassedCounts.includes(reported[field])) {
|
|
99
|
+
warnings.push(`testsPassed: self-reported ${reported[field]} but the label-bound assert shows ${actual} — matches a sibling green run under the same label (multi-suite label-multiplexing); reconciled to a real recount, not a fabrication`);
|
|
100
|
+
return;
|
|
101
|
+
}
|
|
90
102
|
mismatches.push({ field, claimed: reported[field], actual });
|
|
91
103
|
};
|
|
92
104
|
|
package/src/lib/evidence.js
CHANGED
|
Binary file
|