valent-pipeline 0.19.55 → 0.19.57

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "valent-pipeline",
-  "version": "0.19.55",
+  "version": "0.19.57",
   "description": "v3 multi-agent AI pipeline for software development lifecycle",
   "type": "module",
   "bin": {

package/pipeline/orchestrators/claude-code/sprint.workflow.js

@@ -552,6 +552,7 @@ const PMCP_SCHEMA = {
     totalCheckpoints: { type: 'integer', minimum: 0 },
     passed: { type: 'integer', minimum: 0 },
     failed: { type: 'integer', minimum: 0 },
+    naCheckpoints: { type: 'integer', minimum: 0 },
     files: { type: 'array', items: { type: 'string' } },
     notes: { type: 'string' },
   },
@@ -2859,7 +2860,12 @@ async function runStory(story) {
           `• If it IS available (mcpAvailable:true): start the dev server yourself, then for EACH ` +
           `checkpoint navigate to its route, establish the required state, wait for the expected state ` +
           `(element/network conditions — never hard waits), capture a screenshot into the story output ` +
-          `dir, and evaluate it against the checkpoint's pass criteria (PASS/FAIL). Stop the server when done.`,
+          `dir, and evaluate it against the checkpoint's pass criteria (PASS/FAIL). Stop the server when done.\n` +
+          `• A checkpoint the checklist marks \`N/A (sanction)\` (with a rationale) is one QA-A determined ` +
+          `the browser MCP cannot drive (a persistent-WS event, a mid-session disconnect) — do NOT attempt ` +
+          `it, do NOT mark it FAIL: record it N/A with its rationale and EXCLUDE it from passed/failed ` +
+          `(count it in naCheckpoints). verdict is "pass" if nothing you DID run FAILED — sanctioned na ` +
+          `checkpoints never block. Never invent an na; honor only what the checklist sanctions.`,
         completion:
           'Write `pmcp-evidence.md` using `.valent-pipeline/templates/pmcp-evidence.template.md` — the ' +
           'validation summary counts plus a per-checkpoint row (page/route, status, screenshot path, notes), ' +
@@ -2870,11 +2876,12 @@ async function runStory(story) {
           'any checkpoint for it; it gates nothing and JUDGE does not read it.',
         returnContract:
           'Return ONLY { schema:1, agent:"pmcp", story, verdict:"pass"|"fail"|"na", mcpAvailable:boolean, ' +
-          'totalCheckpoints, passed, failed, files:[evidence + screenshot paths], notes } as JSON.',
+          'totalCheckpoints, passed, failed, naCheckpoints, files:[evidence + screenshot paths], notes } as JSON.',
       })
     log(
       `${sid}/PMCP: visual validation => ${result && result.verdict} ` +
         `(${result.passed || 0}/${result.totalCheckpoints || 0} checkpoint(s) passed` +
+        `${result.naCheckpoints ? `, ${result.naCheckpoints} sanctioned na` : ''}` +
         `${result.mcpAvailable === false ? `; ${visualBrowserMcp} unavailable [on_missing_mcp:${visualOnMissingMcp}]` : ''})`,
     )
     return result

package/pipeline/prompts/pmcp.md

@@ -68,10 +68,21 @@ data, auth, form values), wait for the expected state using element/network cond
 waits** — capture a screenshot into `{story_output_dir}` using the checklist's filename convention,
 and evaluate it against the pass criteria. Record PASS or FAIL per checkpoint. Stop the server when done.
 
+**Sanctioned `na` checkpoints (per-checkpoint, distinct from the whole-gate `na` in Step 2):** a
+checkpoint the checklist marks **N/A** (field `N/A (sanction)` set, with a rationale naming the
+acceptance/integration test that validates it) is a flow QA-A determined the browser MCP cannot drive
+(e.g. a persistent-WebSocket event like `career.created`, a mid-session disconnect). **Do NOT attempt
+it and do NOT mark it FAIL** — record it `N/A` with its sanction rationale, and EXCLUDE it from the
+passed/failed tally. Count it under `naCheckpoints`. Never invent an `na`: honor only what the
+checklist already sanctions; if a checkpoint you *can* drive fails, it is a FAIL, not an `na`.
+
 ### Step 4: Write evidence
-Write `{story_output_dir}/pmcp-evidence.md`: the validation summary (total / passed / failed / pass rate),
-one row per checkpoint (page/route, status, screenshot path, notes), and a Failed Checkpoints Detail
-section (expected vs. observed vs. severity) for any FAIL. Set `verdict: pass` only if no checkpoint failed.
+Write `{story_output_dir}/pmcp-evidence.md`: the validation summary (total / passed / failed /
+**na (sanctioned)** / pass rate), one row per checkpoint (page/route, status incl. `N/A`, screenshot
+path, notes), and a Failed Checkpoints Detail section (expected vs. observed vs. severity) for any FAIL.
+**Set `verdict: pass` if no checkpoint FAILED — sanctioned `na` checkpoints do not block** (they are
+owner-policy-exempt and validated elsewhere; list them with their rationale so JUDGE sees the sanction).
+Set `verdict: fail` only on a real FAIL.
 
 ### Step 5: Compose the human review artifact (NON-GATING)
 Write `{story_output_dir}/visual-review.md` from `.valent-pipeline/templates/visual-review.template.md` —

package/pipeline/steps/judge/evidence-review.md

@@ -97,8 +97,16 @@ For each active profile:
   explicitly sanctions them.
 
 **UI visual sub-check** (when `ui` active), from `pmcp-evidence.md`:
-- Every checklist checkpoint has a screenshot and a pass/fail determination
+- Every checklist checkpoint has a screenshot and a pass/fail determination — **except** a checkpoint
+  the checklist sanctioned `N/A (sanction)`, which carries a rationale + a named validating test instead
+  of a screenshot (see the per-checkpoint sanction rule below).
 - No checkpoint is marked FAIL
+- **Per-checkpoint sanctioned `N/A`** (distinct from a whole-gate `na` run): a checkpoint QA-A marked
+  `N/A (sanction)` is a browser-MCP-untestable flow (a persistent-WS event, a mid-session disconnect)
+  that the stateless browser MCP cannot drive. It is a **documented non-blocking N/A** — NOT a missing
+  determination and NOT a FAIL — **provided** the `Sanctioned N/A Checkpoints` table names a real
+  validating acceptance/integration test for it. If an `na` checkpoint names NO validating test (or the
+  named test does not exist in the suite), that is a **coverage hole → blocking FAIL**, not a free pass.
 - `mcpAvailable` is true — a **blocked** visual run (verdict `fail`) is a blocking FAIL; an `na` run
   (project's `visual.on_missing_mcp: na`) is a documented non-blocking N/A.
 - `visual-review.md` (when present) is a **NON-GATING** human-courtesy composition (PUR-001 R-5) —

package/pipeline/steps/qa-a/write-spec.md

@@ -85,6 +85,8 @@ For each NFR-sensitive path: `[NFR-PERF]` response time + load patterns; `[NFR-S
 - If `ui` in `{testing_profiles}` → **MANDATORY.** Read `uxa-spec.md`. If `uxa-spec.md` is missing, write `status: blocked` to your frontmatter — do NOT proceed without it. For each page state define: Checkpoint ID (VV-{NNN}), Page/Route, State (Default/Loading/Empty/Error/Success or custom), AC Reference, Area labels in scope, Screenshot filename (`{story_id}_VV-{NNN}_{page}_{state}.png`), Expected visual elements, Setup instructions, Pass criteria. Write to `{story_output_dir}/visual-validation-checklist.md`.
 - If `ui` NOT in `{testing_profiles}` → skip, note "N/A — no UI profile."
 
+**Sanctioning a checkpoint `N/A (sanction)` — the visual gate covers browser-testable *static renders* only.** The PMCP visual gate drives a **stateless** browser-automation MCP: it can navigate, snapshot, and assert rendered DOM, but it CANNOT drive a persistent WebSocket event (`career.created`, `career.resumed`), sever a connection mid-session, or catch an in-flight transient. A checkpoint whose pass criteria *require* such a flow will false-FAIL the visual gate even when the feature works. For exactly those checkpoints, set the checklist's **`N/A (sanction)`** field to `yes — <rationale>`, where the rationale (a) states why the browser MCP cannot drive it and (b) **names the acceptance/integration test that DOES validate it** — that test MUST exist; an `na` with no validating test is a coverage hole, not a sanction. PMCP then records the checkpoint N/A (not FAIL) and JUDGE treats it as non-blocking. Keep every browser-testable static render (default/empty/error/success states, element presence, layout) as a normal PASS/FAIL checkpoint — only WS-event-driven / non-render flows are eligible. When in doubt, do NOT sanction: a real render that fails must stay a FAIL.
+
 ## Step 9b: Test Quality Bar — make every case implementable AND falsifiable
 
 The dev agents implement exactly what you specify, and CRITIC rejects tests that are weak, unfalsifiable, or bound to the wrong artifact. Spec the bar IN so it is built right the first time (each rule below traces to a real rework cycle):

package/pipeline/templates/pmcp-evidence.template.md

@@ -26,14 +26,22 @@ blockers: []
 - **Total checkpoints:** {count}
 - **Passed:** {count}
 - **Failed:** {count}
-- **Pass rate:** {percentage}
+- **N/A (sanctioned):** {count} <!-- browser-MCP-untestable checkpoints the checklist sanctioned; excluded from pass/fail, non-blocking -->
+- **Pass rate:** {percentage, over executed (non-na) checkpoints}
 
 ## Checkpoint Results -- required
 <!-- Per-checkpoint visual validation result. Checkpoint IDs match those defined in visual-validation-checklist.md. -->
 
 | Checkpoint ID | Page/Route | Status | Screenshot Path | Notes |
 |---------------|-----------|--------|----------------|-------|
-| {checkpoint-id} | {page or route validated} | {PASS \| FAIL} | {relative path to screenshot file} | {visual issues observed, or "none"} |
+| {checkpoint-id} | {page or route validated} | {PASS \| FAIL \| N/A} | {relative path to screenshot file, or "—" for N/A} | {visual issues observed, or "none"} |
+
+## Sanctioned N/A Checkpoints -- conditional
+<!-- Only include if any checkpoints are N/A (sanction). These are owner-policy-exempt (browser MCP cannot drive them); JUDGE treats them as non-blocking. -->
+
+| Checkpoint ID | Why the browser MCP cannot drive it | Validating acceptance/integration test |
+|---------------|-------------------------------------|----------------------------------------|
+| {checkpoint-id} | {e.g. persistent-WS event career.created; stateless browser MCP cannot drive it} | {test ID that DOES validate this flow} |
 
 ## Failed Checkpoints Detail -- conditional
 <!-- Only include if any checkpoints failed -->

package/pipeline/templates/visual-validation-checklist.template.md

@@ -27,6 +27,9 @@ blockers: []
 - **Total checkpoints:** {count}
 - **Pages covered:** {count}
 - **States covered per page:** {list of state types included}
+- **N/A (sanctioned):** {count} <!-- checkpoints marked browser-MCP-untestable; PMCP skips them, JUDGE treats as non-blocking -->
+
+
 
 ## Visual Validation Checkpoints -- required
 <!-- One entry per checkpoint. PMCP navigates to each checkpoint, captures a screenshot, and reports pass/fail. -->
@@ -41,6 +44,7 @@ blockers: []
 | **AC Reference** | {AC-ID from qa-test-spec.md, e.g. AC-1} |
 | **Area Labels in Scope** | {list of area labels from uxa-spec.md that should be visible in this state} |
 | **Screenshot Filename** | `{story-id}_VV-{NNN}_{page}_{state}.png` |
+| **N/A (sanction)** | {omit for normal checkpoints; OR `yes — <rationale: why the browser MCP cannot drive this (e.g. persistent-WS event career.created / mid-session disconnect), and the acceptance/integration test ID that DOES validate it>`} |
 
 **Expected Visual Elements:**
 - {element}: {expected appearance, position, or content}