baldart 3.17.0 → 3.18.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +93 -0
- package/README.md +21 -12
- package/VERSION +1 -1
- package/bin/baldart.js +34 -3
- package/framework/.claude/agents/REGISTRY.md +1 -0
- package/framework/.claude/agents/visual-fidelity-verifier.md +319 -0
- package/framework/.claude/commands/design-review.md +2 -0
- package/framework/.claude/skills/e2e-review/SKILL.md +552 -0
- package/framework/.claude/skills/new/SKILL.md +99 -85
- package/framework/.claude/skills/playwright-skill/SKILL.md +29 -0
- package/framework/.claude/skills/prd/assets/card-template.yml +7 -0
- package/framework/.claude/skills/webapp-testing/SKILL.md +23 -0
- package/framework/AGENTS.md +1 -0
- package/framework/docs/PROJECT-CONFIGURATION.md +20 -1
- package/framework/templates/baldart.config.template.yml +38 -0
- package/package.json +1 -1
- package/src/commands/configure.js +58 -0
- package/src/commands/status.js +36 -4
- package/src/commands/update.js +33 -14
|
@@ -0,0 +1,552 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: e2e-review
|
|
3
|
+
description: >
|
|
4
|
+
Deterministic, BLOCKING end-to-end review run after a feature is implemented.
|
|
5
|
+
Combines functional E2E (Playwright spec written by `coder`, executed via
|
|
6
|
+
playwright-skill) with visual fidelity diff (`visual-fidelity-verifier`
|
|
7
|
+
multimodal agent), aggregates findings under a strict severity gate, runs a
|
|
8
|
+
bounded self-heal loop, and either passes / blocks / accepts an explicit
|
|
9
|
+
user override. Invocable manually via /e2e-review CARD-ID, and auto-called
|
|
10
|
+
by /new Phase 2.6. Triggers on: /e2e-review, "verifica end-to-end", "review
|
|
11
|
+
schermate", "controlla i mockup", "e2e review".
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
> **YOLO MODE**: When invoked by `/new` (programmatic mode), every spawned
|
|
15
|
+
> agent MUST use `mode: "bypassPermissions"`. When invoked manually, the user
|
|
16
|
+
> approves prompts.
|
|
17
|
+
|
|
18
|
+
## Project Context
|
|
19
|
+
|
|
20
|
+
**Reads from `baldart.config.yml`:**
|
|
21
|
+
`paths.backlog_dir`, `paths.app_dir`, `paths.components_primitives`,
|
|
22
|
+
`paths.components_root`, `paths.e2e_tests_dir`, `paths.design_system`,
|
|
23
|
+
`paths.ui_guidelines`, `features.has_e2e_review`,
|
|
24
|
+
`features.has_design_system`, `features.has_backlog`,
|
|
25
|
+
`features.e2e_review.fidelity_tolerance`,
|
|
26
|
+
`features.e2e_review.max_self_heal_iterations`,
|
|
27
|
+
`features.e2e_review.pixel_diff_threshold`,
|
|
28
|
+
`features.e2e_review.require_override_reason`.
|
|
29
|
+
|
|
30
|
+
**Gated by features:** `features.has_e2e_review` — the skill REFUSES to run
|
|
31
|
+
when `false`. The orchestrator (`/new`) silently skips the call when the flag
|
|
32
|
+
is off, preserving backwards compatibility for projects that have not yet
|
|
33
|
+
opted in.
|
|
34
|
+
|
|
35
|
+
**Overlay:** loads `.baldart/overlays/e2e-review.md` if present —
|
|
36
|
+
project-specific verification recipes (custom Gherkin step library, project
|
|
37
|
+
mockup conventions, override approval policy).
|
|
38
|
+
|
|
39
|
+
**On missing/empty keys:** ask the user; do not assume defaults. See
|
|
40
|
+
`framework/agents/project-context.md` § 3.
|
|
41
|
+
|
|
42
|
+
You are the **E2E Review Orchestrator**. Your single job is to determine,
|
|
43
|
+
deterministically, whether an implementation matches its specification — both
|
|
44
|
+
functionally (the user flow works) and visually (the screen matches the
|
|
45
|
+
mockup or the design-system spec). You do not implement. You do not redesign.
|
|
46
|
+
You orchestrate three workers (`coder`, `visual-fidelity-verifier`, the
|
|
47
|
+
underlying Playwright runner) and aggregate their output through a strict
|
|
48
|
+
severity gate.
|
|
49
|
+
|
|
50
|
+
This skill embodies the **three-layer harness pattern** for post-implementation
|
|
51
|
+
verification documented in
|
|
52
|
+
[Effective Harnesses for Long-Running Agents](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents):
|
|
53
|
+
the implementing agent (`coder`) and the verifying agent
|
|
54
|
+
(`visual-fidelity-verifier`) are distinct, with rule-based aggregation in
|
|
55
|
+
between — no agent grades its own work.
|
|
56
|
+
|
|
57
|
+
---
|
|
58
|
+
|
|
59
|
+
## Invocation Modes
|
|
60
|
+
|
|
61
|
+
| Mode | Trigger | Output |
|
|
62
|
+
|---|---|---|
|
|
63
|
+
| **Manual** | User types `/e2e-review CARD-ID` (or just `/e2e-review` then asks for the card) | Markdown report to the user + JSON sidecar at `.baldart/e2e-review/<CARD-ID>/report.json` |
|
|
64
|
+
| **Programmatic** | `/new` Phase 2.6 invokes the skill with `{card_id, worktree_path, mode: "programmatic"}` | JSON only on stdout (parsed by `/new` for gating); written to `.baldart/e2e-review/<CARD-ID>/report.json` |
|
|
65
|
+
|
|
66
|
+
When invoked from `/new`, you operate inside the worktree. When invoked
|
|
67
|
+
manually, you operate in the user's current directory (which may be a worktree
|
|
68
|
+
or the main repo).
|
|
69
|
+
|
|
70
|
+
---
|
|
71
|
+
|
|
72
|
+
## Pre-flight (run once at skill entry)
|
|
73
|
+
|
|
74
|
+
1. **Gate check**: read `features.has_e2e_review`. If `false`:
|
|
75
|
+
- Manual mode: tell the user "E2E review is disabled in baldart.config.yml.
|
|
76
|
+
Run `npx baldart configure` to enable, or run /design-review for an
|
|
77
|
+
advisory check." STOP.
|
|
78
|
+
- Programmatic mode: return JSON
|
|
79
|
+
`{ "status": "skipped", "reason": "feature_disabled" }` and STOP.
|
|
80
|
+
2. **Card resolution**: parse `CARD-ID` from the user's prompt or the
|
|
81
|
+
orchestrator's payload. If missing in manual mode, ask the user which card.
|
|
82
|
+
Read the card YAML from `${paths.backlog_dir}/<area>/<status>/<CARD-ID>.yml`
|
|
83
|
+
(or `${paths.backlog_dir}/<CARD-ID>.yml` if the project does not use the
|
|
84
|
+
per-area layout). If the card cannot be found, STOP with a clear error.
|
|
85
|
+
3. **Auto-skip detection** (BEFORE Phase 1): the skill is unnecessary on
|
|
86
|
+
pure-backend cards. Skip when ALL these conditions hold:
|
|
87
|
+
- The card has no `links.design` field (no mockup reference).
|
|
88
|
+
- `git diff --name-only` (vs `main` / `develop` per `git.merge_strategy`)
|
|
89
|
+
shows no file under `${paths.app_dir}` AND no file under
|
|
90
|
+
`${paths.components_primitives}` AND no `*.tsx` / `*.css` / `*.scss` /
|
|
91
|
+
`*.svelte` / `*.vue` anywhere.
|
|
92
|
+
- The card type is `backend`, `api`, `db`, `infra`, `docs`, `chore`, or
|
|
93
|
+
`config`.
|
|
94
|
+
When auto-skipping: return JSON `{ "status": "skipped", "reason":
|
|
95
|
+
"backend_only_card", "skipped_at": "pre-flight" }`. The orchestrator does
|
|
96
|
+
NOT treat skip as failure.
|
|
97
|
+
4. **State directory**: create `.baldart/e2e-review/<CARD-ID>/` and
|
|
98
|
+
subdirectories `screenshots/`, `mockups/`, `transcripts/`. The directory
|
|
99
|
+
persists across self-heal iterations; previous-iteration artifacts are
|
|
100
|
+
retained for diff and audit.
|
|
101
|
+
|
|
102
|
+
---
|
|
103
|
+
|
|
104
|
+
## Phase 1 — Verification Plan Extraction
|
|
105
|
+
|
|
106
|
+
**Goal**: turn the card's acceptance criteria + mockup references + design
|
|
107
|
+
system constraints into a machine-readable verification plan.
|
|
108
|
+
|
|
109
|
+
1. **Read card sources** in this order:
|
|
110
|
+
- `card.acceptance_criteria[]` — primary source of "what must work".
|
|
111
|
+
- `card.test_plan` — if present (written by `/prd`), it contains
|
|
112
|
+
pre-planned Gherkin scenarios, test credentials, data prerequisites. The
|
|
113
|
+
contract between `/prd` and this skill is documented in
|
|
114
|
+
`framework/.claude/skills/prd/references/test-plan-contract.md` (see the
|
|
115
|
+
`prd` skill's test-plan output).
|
|
116
|
+
- `card.requirements[]` — additional context for what was implemented.
|
|
117
|
+
- `card.links.design[]` — mockup references (Figma URL, local PNG/JPG/PDF
|
|
118
|
+
path).
|
|
119
|
+
- `card.scope.routes[]` — explicit routes/screens in scope. If absent,
|
|
120
|
+
fall back to deriving them from the diff (see Phase 2 below).
|
|
121
|
+
2. **Derive routes/screens in scope** when not explicit:
|
|
122
|
+
- Run `git diff --name-only` (vs the branch base).
|
|
123
|
+
- Match changed files under `${paths.app_dir}` to route patterns
|
|
124
|
+
(e.g. Next.js `app/.../page.tsx` → route, Remix `routes/...` →
|
|
125
|
+
route, SvelteKit `routes/...` → route). If the framework is unknown,
|
|
126
|
+
ask the user.
|
|
127
|
+
- For each changed component under `${paths.components_primitives}`,
|
|
128
|
+
identify the routes that import it via the `codebase-architect` agent
|
|
129
|
+
(fallback to grep when LSP is unavailable).
|
|
130
|
+
3. **Synthesize verification plan** as Gherkin scenarios. Format:
|
|
131
|
+
|
|
132
|
+
```gherkin
|
|
133
|
+
Feature: <card.title>
|
|
134
|
+
Background:
|
|
135
|
+
Given the user is <persona from test_plan.test_credentials.persona>
|
|
136
|
+
And the system is seeded with <data_prerequisites>
|
|
137
|
+
|
|
138
|
+
Scenario: <AC-1 short label>
|
|
139
|
+
Given <pre-state>
|
|
140
|
+
When <user action>
|
|
141
|
+
Then <expected outcome>
|
|
142
|
+
And <screen <route> matches mockup <mockup_ref>>
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
When the card has a `test_plan.test_scenarios[]`, copy those scenarios
|
|
146
|
+
verbatim — do not rephrase them. When the card has only
|
|
147
|
+
`acceptance_criteria[]`, derive one scenario per AC using best-effort
|
|
148
|
+
translation. Flag ACs that resist Gherkin translation (e.g. purely
|
|
149
|
+
non-functional) under a `## Non-Functional Checks` section so they are
|
|
150
|
+
not silently dropped.
|
|
151
|
+
4. **Persist the plan** to `.baldart/e2e-review/<CARD-ID>/plan.json`:
|
|
152
|
+
|
|
153
|
+
```json
|
|
154
|
+
{
|
|
155
|
+
"card_id": "FEAT-XXXX",
|
|
156
|
+
"title": "...",
|
|
157
|
+
"routes": ["/merchant/dashboard", "/merchant/dashboard/edit"],
|
|
158
|
+
"personas": [{"name": "merchant", "auth": "password", "creds": {...}}],
|
|
159
|
+
"data_prerequisites": ["seed merchant 'demo'", "create one paid booking"],
|
|
160
|
+
"scenarios": [
|
|
161
|
+
{
|
|
162
|
+
"id": "AC-1",
|
|
163
|
+
"name": "Merchant can update store hours",
|
|
164
|
+
"gherkin": "Feature: ...",
|
|
165
|
+
"routes_touched": ["/merchant/settings/hours"]
|
|
166
|
+
}
|
|
167
|
+
],
|
|
168
|
+
"non_functional_checks": ["loads in < 2s on slow 3G"],
|
|
169
|
+
"mockup_refs": [{"route": "/merchant/dashboard", "level": "local", "path": "mockups/dashboard.png"}]
|
|
170
|
+
}
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
---
|
|
174
|
+
|
|
175
|
+
## Phase 2 — Mockup Source Cascade (4 levels)
|
|
176
|
+
|
|
177
|
+
**Goal**: for each route in scope, locate the ground-truth visual reference.
|
|
178
|
+
This is graceful by design — never hard-fail because Figma is not configured.
|
|
179
|
+
|
|
180
|
+
For each `route` in the plan, walk the cascade in order and STOP at the first
|
|
181
|
+
hit:
|
|
182
|
+
|
|
183
|
+
### Level (a) — Figma MCP
|
|
184
|
+
|
|
185
|
+
If `card.links.design[]` contains a Figma URL AND the Figma MCP server is
|
|
186
|
+
available (probe `get_design_context` / `get_screenshot` tools at runtime):
|
|
187
|
+
|
|
188
|
+
- Call `get_design_context` for the Figma node to extract tokens / variables /
|
|
189
|
+
primitives referenced in the design.
|
|
190
|
+
- Call `get_screenshot` to fetch the rendered Figma frame at 1440px width;
|
|
191
|
+
save to `.baldart/e2e-review/<CARD-ID>/mockups/<route-slug>.figma.png`.
|
|
192
|
+
- Record `mockup_source.level = "figma"` and the node ID for the audit trail.
|
|
193
|
+
|
|
194
|
+
If the MCP tools error or time out, fall through silently to level (b) — do
|
|
195
|
+
not surface the failure as a blocker.
|
|
196
|
+
|
|
197
|
+
### Level (b) — Local image
|
|
198
|
+
|
|
199
|
+
If `card.links.design[]` contains a local file path (PNG / JPG / PDF) OR the
|
|
200
|
+
card has a `mockups/` directory alongside its YAML:
|
|
201
|
+
|
|
202
|
+
- Resolve the path. For PDFs, extract the first page as PNG (use `pdftoppm`
|
|
203
|
+
or equivalent — degrade gracefully if unavailable).
|
|
204
|
+
- Copy to `.baldart/e2e-review/<CARD-ID>/mockups/<route-slug>.local.png`.
|
|
205
|
+
- Record `mockup_source.level = "local"` and the source path.
|
|
206
|
+
|
|
207
|
+
### Level (c) — Compliance-only (registry)
|
|
208
|
+
|
|
209
|
+
If no mockup image is available BUT `features.has_design_system: true`:
|
|
210
|
+
|
|
211
|
+
- The verifier will operate in compliance-only mode against
|
|
212
|
+
`${paths.design_system}/INDEX.md` + `tokens-reference.md` + per-component
|
|
213
|
+
specs (cascade detailed in
|
|
214
|
+
[`framework/agents/design-system-protocol.md`](../../../agents/design-system-protocol.md)).
|
|
215
|
+
- Identify which primitives the route uses via the `codebase-architect`
|
|
216
|
+
agent (or grep on imports under `${paths.components_primitives}`).
|
|
217
|
+
- Record `mockup_source.level = "compliance-only"` and the list of components
|
|
218
|
+
in scope.
|
|
219
|
+
|
|
220
|
+
### Level (d) — Skip visual
|
|
221
|
+
|
|
222
|
+
If none of (a)–(c) apply (no mockup, no design system):
|
|
223
|
+
|
|
224
|
+
- Record `mockup_source.level = "skip"` and log a warning to the report's
|
|
225
|
+
`gaps[]` field: `"no_mockup_or_design_system — visual fidelity check
|
|
226
|
+
skipped for route <route>"`.
|
|
227
|
+
- The functional E2E (Phase 3) still runs.
|
|
228
|
+
|
|
229
|
+
The cascade output is appended to the plan under `mockup_refs[]`.
|
|
230
|
+
|
|
231
|
+
---
|
|
232
|
+
|
|
233
|
+
## Phase 3 — Functional E2E (browser-based)
|
|
234
|
+
|
|
235
|
+
**Goal**: execute the Gherkin scenarios from Phase 1 in a real browser and
|
|
236
|
+
capture the result + screenshots per route.
|
|
237
|
+
|
|
238
|
+
1. **Pre-conditions**:
|
|
239
|
+
- The project must have a runnable dev server (read `scripts/dev` from
|
|
240
|
+
`package.json` or the equivalent). If the dev server is not already
|
|
241
|
+
running, start it in the background and wait for the health-check URL
|
|
242
|
+
to respond (max 60s). Record the port for cleanup.
|
|
243
|
+
- The project must have `@playwright/test` installed under
|
|
244
|
+
`${paths.e2e_tests_dir}`. If missing, the skill cannot run — log
|
|
245
|
+
`"playwright_not_installed"` in `gaps[]` and surface a setup hint to
|
|
246
|
+
the user (`npm i -D @playwright/test && npx playwright install
|
|
247
|
+
chromium`).
|
|
248
|
+
2. **Spec generation**: spawn a `coder` agent with this contract:
|
|
249
|
+
|
|
250
|
+
```
|
|
251
|
+
Generate a Playwright .spec.ts file for card <CARD-ID> at:
|
|
252
|
+
${paths.e2e_tests_dir}/<CARD-ID-slug>.spec.ts
|
|
253
|
+
|
|
254
|
+
Source of truth: .baldart/e2e-review/<CARD-ID>/plan.json
|
|
255
|
+
(read it — it contains every scenario, persona, prerequisite, and route).
|
|
256
|
+
|
|
257
|
+
## Rules
|
|
258
|
+
- One `test.describe` per scenario.
|
|
259
|
+
- Use `page.getByRole()`, `page.getByText()`, `page.getByTestId()` —
|
|
260
|
+
never CSS selectors that depend on Tailwind class strings.
|
|
261
|
+
- Use `expect(locator)` with auto-waiting.
|
|
262
|
+
- For each route in the scenario, call:
|
|
263
|
+
await page.goto(route);
|
|
264
|
+
await page.waitForLoadState("networkidle");
|
|
265
|
+
await page.screenshot({
|
|
266
|
+
path: ".baldart/e2e-review/<CARD-ID>/screenshots/<route-slug>.png",
|
|
267
|
+
fullPage: true
|
|
268
|
+
});
|
|
269
|
+
The screenshot path is consumed by the visual-fidelity-verifier in Phase 4 — do not change it.
|
|
270
|
+
- For OTP-driven personas (test_plan.test_credentials.auth_method == "otp"),
|
|
271
|
+
use `await page.pause()` between phone-entry and code-entry. Document
|
|
272
|
+
the manual step in a test.step block.
|
|
273
|
+
- Do NOT use chromium.launch() — let Playwright config manage the browser.
|
|
274
|
+
- Use the project's baseURL from playwright.config.ts.
|
|
275
|
+
|
|
276
|
+
## Out of scope
|
|
277
|
+
- Do not modify any file outside ${paths.e2e_tests_dir}/.
|
|
278
|
+
- Do not refactor existing tests.
|
|
279
|
+
- Do not edit playwright.config.ts.
|
|
280
|
+
```
|
|
281
|
+
|
|
282
|
+
This agent runs in `bypassPermissions` mode when called by `/new`.
|
|
283
|
+
3. **Execution**: run the generated spec via Bash:
|
|
284
|
+
|
|
285
|
+
```bash
|
|
286
|
+
npx playwright test ${paths.e2e_tests_dir}/<CARD-ID-slug>.spec.ts \
|
|
287
|
+
--reporter=json \
|
|
288
|
+
--output=.baldart/e2e-review/<CARD-ID>/playwright-results \
|
|
289
|
+
2>&1 | tee .baldart/e2e-review/<CARD-ID>/transcripts/playwright.log
|
|
290
|
+
```
|
|
291
|
+
|
|
292
|
+
Headed mode is reserved for OTP flows in manual invocation; programmatic
|
|
293
|
+
invocation from `/new` always runs headless.
|
|
294
|
+
4. **Result capture**: parse the Playwright JSON output. Record per scenario:
|
|
295
|
+
- `status`: `passed` | `failed` | `flaky` | `skipped`
|
|
296
|
+
- `duration_ms`
|
|
297
|
+
- `failure_reason` (when failed, the first assertion error + stack hint)
|
|
298
|
+
- `screenshot_paths[]` (one per route in the scenario)
|
|
299
|
+
5. **Functional findings**: every `failed` or `flaky` scenario becomes a
|
|
300
|
+
functional finding in the aggregate report (Phase 5). Severity:
|
|
301
|
+
- Auth / payments / data-mutation scenario failed → `critical`
|
|
302
|
+
- Display-only scenario failed → `major`
|
|
303
|
+
- Flaky (passed on retry) → `minor` with `confidence: low`
|
|
304
|
+
|
|
305
|
+
---
|
|
306
|
+
|
|
307
|
+
## Phase 4 — Visual Fidelity (multimodal agent)
|
|
308
|
+
|
|
309
|
+
**Goal**: for each route with a mockup reference at level (a)–(c), invoke the
|
|
310
|
+
`visual-fidelity-verifier` agent and collect its severity-tagged findings.
|
|
311
|
+
|
|
312
|
+
For each `route` in `plan.routes[]`:
|
|
313
|
+
|
|
314
|
+
1. **Pre-filter with pixel-diff** (optimization to control Vision cost):
|
|
315
|
+
- When `mockup_source.level in {figma, local}`, run a pixel-level diff
|
|
316
|
+
(any cheap library: `pixelmatch`, `looks-same`, or even ImageMagick's
|
|
317
|
+
`compare -metric AE`) between the implementation screenshot and the
|
|
318
|
+
mockup. Pass `pixel_diff_threshold` from config (default `0.02` = 2%).
|
|
319
|
+
- If the diff is **below** the threshold, skip the Vision call for this
|
|
320
|
+
route and record an empty findings array with
|
|
321
|
+
`pre_filter: "pixel_diff_below_threshold"`. This is the primary
|
|
322
|
+
latency / cost saver — most routes pass pixel diff cleanly.
|
|
323
|
+
- When `mockup_source.level == "compliance-only"`, skip pre-filter (no
|
|
324
|
+
image to compare against) and proceed directly to step 2.
|
|
325
|
+
- When `mockup_source.level == "skip"`, skip this route entirely.
|
|
326
|
+
2. **Invoke `visual-fidelity-verifier`** with the input contract documented
|
|
327
|
+
in [`visual-fidelity-verifier.md`](../../agents/visual-fidelity-verifier.md):
|
|
328
|
+
|
|
329
|
+
```
|
|
330
|
+
subagent_type: "visual-fidelity-verifier"
|
|
331
|
+
mode: "bypassPermissions" // when called from /new
|
|
332
|
+
prompt: |
|
|
333
|
+
Compare the implementation against the mockup and return JSON per your
|
|
334
|
+
output schema.
|
|
335
|
+
|
|
336
|
+
Input payload:
|
|
337
|
+
{
|
|
338
|
+
"card_id": "FEAT-XXXX",
|
|
339
|
+
"route": "/merchant/dashboard",
|
|
340
|
+
"viewport": { "width": 1440, "height": 900 },
|
|
341
|
+
"implementation_screenshot_path": ".baldart/e2e-review/FEAT-XXXX/screenshots/merchant-dashboard.png",
|
|
342
|
+
"mockup_source": { ... },
|
|
343
|
+
"tolerance": "<from config>",
|
|
344
|
+
"fix_hints_enabled": true
|
|
345
|
+
}
|
|
346
|
+
```
|
|
347
|
+
3. **Parse the agent's JSON response** strictly. If the response is malformed
|
|
348
|
+
or contains prose outside the JSON, log
|
|
349
|
+
`verifier_protocol_violation` in `gaps[]` and treat the route's findings
|
|
350
|
+
as empty (do not invent findings from prose).
|
|
351
|
+
4. **Aggregate** visual findings across all routes into a single
|
|
352
|
+
`visual_findings[]` array, deduplicating by `(route, category, region)`.
|
|
353
|
+
|
|
354
|
+
---
|
|
355
|
+
|
|
356
|
+
## Phase 5 — Aggregation + Severity Gate (STRICT default)
|
|
357
|
+
|
|
358
|
+
**Goal**: combine functional + visual findings into a single verdict using
|
|
359
|
+
the canonical severity taxonomy from `visual-fidelity-verifier.md`.
|
|
360
|
+
|
|
361
|
+
1. **Compose** the `findings[]` array:
|
|
362
|
+
- Functional findings from Phase 3 (`source: "functional"`).
|
|
363
|
+
- Visual findings from Phase 4 (`source: "visual"`).
|
|
364
|
+
- Each finding carries: `severity`, `category`, `source`, `route`,
|
|
365
|
+
`description`, `expected`, `actual`, `fix_hint`, `evidence`,
|
|
366
|
+
`ds_drift_code`, `confidence`.
|
|
367
|
+
2. **Apply tolerance filter** based on
|
|
368
|
+
`features.e2e_review.fidelity_tolerance`:
|
|
369
|
+
- `strict` (DEFAULT): Critical + Major + Minor all gate. Additionally,
|
|
370
|
+
auto-demote `Minor` findings with `confidence: low` to advisory (they
|
|
371
|
+
remain in the report but do not block) — this is the documented
|
|
372
|
+
escape hatch against font-rendering / anti-aliasing false positives.
|
|
373
|
+
- `balanced`: Critical + Major gate. Minor advisory.
|
|
374
|
+
- `lenient`: Critical gates. Major + Minor advisory.
|
|
375
|
+
3. **Compute gate decision**:
|
|
376
|
+
- At least one **gating** finding → `verdict: "blocked"`.
|
|
377
|
+
- No gating findings → `verdict: "passed"`.
|
|
378
|
+
4. **DO NOT report yet** — first run the self-heal loop (Phase 5b).
|
|
379
|
+
|
|
380
|
+
### Phase 5b — Self-heal loop (bounded)
|
|
381
|
+
|
|
382
|
+
Read `features.e2e_review.max_self_heal_iterations` (default `2`). The loop
|
|
383
|
+
applies only when `verdict == "blocked"` AND `iteration < max`.
|
|
384
|
+
|
|
385
|
+
For each iteration:
|
|
386
|
+
|
|
387
|
+
1. **Spawn a `coder` agent** with this contract:
|
|
388
|
+
|
|
389
|
+
```
|
|
390
|
+
The /e2e-review skill found the following gating findings on card
|
|
391
|
+
<CARD-ID>. Implement minimal, surgical fixes — do not refactor.
|
|
392
|
+
|
|
393
|
+
## Gating findings
|
|
394
|
+
[paste the JSON findings filtered to gating severity]
|
|
395
|
+
|
|
396
|
+
## Rules
|
|
397
|
+
- File ownership: stay within the card's allowed file set (see
|
|
398
|
+
/tmp/batch-tracker-<FIRST-CARD-ID>.md ## File Ownership Map).
|
|
399
|
+
- One fix per finding when possible. Reuse existing tokens / primitives
|
|
400
|
+
before inventing new ones (cite ${paths.design_system}/INDEX.md +
|
|
401
|
+
tokens-reference.md).
|
|
402
|
+
- After the fixes, re-run lint and typecheck. Both must pass.
|
|
403
|
+
- Do NOT modify Playwright specs or this skill's state files.
|
|
404
|
+
|
|
405
|
+
## Out of scope
|
|
406
|
+
- Visual polish beyond the findings listed.
|
|
407
|
+
- New features or scope expansion.
|
|
408
|
+
```
|
|
409
|
+
|
|
410
|
+
2. **Re-run Phase 3 (functional) and Phase 4 (visual)** with fresh
|
|
411
|
+
screenshots. Increment `iteration`.
|
|
412
|
+
3. **Re-aggregate and re-decide** (back to Phase 5 step 1).
|
|
413
|
+
4. **STOP** when `verdict == "passed"` OR `iteration == max`.
|
|
414
|
+
|
|
415
|
+
When the loop exhausts iterations and findings remain, transition to the
|
|
416
|
+
**override path** below.
|
|
417
|
+
|
|
418
|
+
### Override path (when self-heal fails)
|
|
419
|
+
|
|
420
|
+
When `iteration == max` AND `verdict == "blocked"`:
|
|
421
|
+
|
|
422
|
+
- **Manual mode**: show the user the remaining findings table and ask:
|
|
423
|
+
"Override the gate and proceed anyway? If yes, provide a reason
|
|
424
|
+
(mandatory when `features.e2e_review.require_override_reason: true`)."
|
|
425
|
+
If the user provides a reason → set `verdict: "overridden"` and record
|
|
426
|
+
`override_reason` in the report. If the user declines → keep
|
|
427
|
+
`verdict: "blocked"`.
|
|
428
|
+
- **Programmatic mode (from `/new`)**: do NOT prompt. Return
|
|
429
|
+
`verdict: "blocked"` with the findings. `/new` decides whether to escalate
|
|
430
|
+
to the user.
|
|
431
|
+
|
|
432
|
+
---
|
|
433
|
+
|
|
434
|
+
## Phase 6 — Return Protocol
|
|
435
|
+
|
|
436
|
+
Write the final report to `.baldart/e2e-review/<CARD-ID>/report.json`:
|
|
437
|
+
|
|
438
|
+
```json
|
|
439
|
+
{
|
|
440
|
+
"status": "passed" | "blocked" | "overridden" | "skipped" | "error",
|
|
441
|
+
"card_id": "FEAT-XXXX",
|
|
442
|
+
"timestamp": "2026-05-24T12:34:56Z",
|
|
443
|
+
"iterations": 2,
|
|
444
|
+
"tolerance": "strict",
|
|
445
|
+
"mockup_source_summary": {
|
|
446
|
+
"figma": 0,
|
|
447
|
+
"local": 2,
|
|
448
|
+
"compliance-only": 1,
|
|
449
|
+
"skip": 0
|
|
450
|
+
},
|
|
451
|
+
"functional": {
|
|
452
|
+
"scenarios_total": 5,
|
|
453
|
+
"passed": 5,
|
|
454
|
+
"failed": 0,
|
|
455
|
+
"flaky": 0,
|
|
456
|
+
"skipped": 0
|
|
457
|
+
},
|
|
458
|
+
"visual": {
|
|
459
|
+
"routes_checked": 3,
|
|
460
|
+
"routes_pre_filtered": 1,
|
|
461
|
+
"compliance_score_avg": 0.94
|
|
462
|
+
},
|
|
463
|
+
"findings": [ ... ],
|
|
464
|
+
"ds_drift_codes": ["DS_TOKENS_DRIFT"],
|
|
465
|
+
"gaps": [ ... ],
|
|
466
|
+
"override_reason": null,
|
|
467
|
+
"transcript_paths": [
|
|
468
|
+
".baldart/e2e-review/FEAT-XXXX/transcripts/playwright.log",
|
|
469
|
+
".baldart/e2e-review/FEAT-XXXX/transcripts/visual-verifier.md"
|
|
470
|
+
]
|
|
471
|
+
}
|
|
472
|
+
```
|
|
473
|
+
|
|
474
|
+
Then:
|
|
475
|
+
|
|
476
|
+
- **Manual mode**: render a concise Markdown summary to the user with the
|
|
477
|
+
verdict, per-route findings table, and links to the full JSON + transcripts.
|
|
478
|
+
Use boxed UI sparingly — this is conversational output.
|
|
479
|
+
- **Programmatic mode**: emit the JSON object as the final message (no prose
|
|
480
|
+
preamble) so `/new` can parse it directly.
|
|
481
|
+
|
|
482
|
+
---
|
|
483
|
+
|
|
484
|
+
## Re-run Trigger (when called from `/new`)
|
|
485
|
+
|
|
486
|
+
When invoked by `/new`, the skill exposes a re-run hook: if `/new` Phase 3.7
|
|
487
|
+
(`/codexreview` gate) modifies a file under `${paths.app_dir}`,
|
|
488
|
+
`${paths.components_primitives}`, or any `*.css` / `*.scss` after the initial
|
|
489
|
+
`/e2e-review` run, the orchestrator MUST re-invoke `/e2e-review` before the
|
|
490
|
+
final commit. This protects against code-review changes that silently
|
|
491
|
+
introduce visual drift.
|
|
492
|
+
|
|
493
|
+
The re-run reuses the same `.baldart/e2e-review/<CARD-ID>/` state directory
|
|
494
|
+
and starts from Phase 3 (skipping plan extraction and mockup cascade).
|
|
495
|
+
|
|
496
|
+
---
|
|
497
|
+
|
|
498
|
+
## Interaction with Other Skills / Agents
|
|
499
|
+
|
|
500
|
+
- **`/new` Phase 2.6** — primary caller. See
|
|
501
|
+
[`framework/.claude/skills/new/SKILL.md`](../new/SKILL.md) Phase 2.6 for the
|
|
502
|
+
invocation contract.
|
|
503
|
+
- **`/prd` test-plan output** — primary input to Phase 1. The skill expects
|
|
504
|
+
the schema documented in
|
|
505
|
+
[`framework/.claude/skills/prd/references/test-plan-contract.md`](../prd/references/test-plan-contract.md).
|
|
506
|
+
- **`/design-review` command** — when invoked from this skill with
|
|
507
|
+
`mode: "programmatic"`, returns JSON instead of Markdown for gating.
|
|
508
|
+
See [`framework/.claude/commands/design-review.md`](../../commands/design-review.md).
|
|
509
|
+
- **`playwright-skill`** — tooling layer for spec execution. See its
|
|
510
|
+
"E2E Review Integration" section.
|
|
511
|
+
- **`webapp-testing`** — used in compliance-only mode for runtime DOM
|
|
512
|
+
inspection (read computed styles, focus order). See its "E2E Review
|
|
513
|
+
Integration" section.
|
|
514
|
+
- **`ds-drift` routine** — weekly safety net. This skill is the per-card
|
|
515
|
+
first line; the routine catches what slips through (e.g. direct human
|
|
516
|
+
edits that bypass `/new`). The two are complementary, not redundant.
|
|
517
|
+
|
|
518
|
+
---
|
|
519
|
+
|
|
520
|
+
## What This Skill Is NOT
|
|
521
|
+
|
|
522
|
+
- **Not a code reviewer** — that is `/codexreview` + `code-reviewer`.
|
|
523
|
+
- **Not a design generator** — that is `ui-design` / `frontend-design`.
|
|
524
|
+
- **Not a mockup creator** — when no mockup exists, the skill degrades to
|
|
525
|
+
compliance-only or skip; it does not invent one.
|
|
526
|
+
- **Not a unit-test runner** — that is `qa-sentinel`.
|
|
527
|
+
- **Not an exploratory tester** — the verification plan is bounded by the
|
|
528
|
+
card's acceptance criteria; the skill does not invent new scenarios.
|
|
529
|
+
|
|
530
|
+
---
|
|
531
|
+
|
|
532
|
+
## Failure Modes & Diagnostics
|
|
533
|
+
|
|
534
|
+
| Symptom | Likely cause | Action |
|
|
535
|
+
|---|---|---|
|
|
536
|
+
| Skill refuses to run | `features.has_e2e_review: false` | Run `npx baldart configure` and enable it |
|
|
537
|
+
| Pixel-diff always 100% | Implementation screenshot dimensions ≠ mockup dimensions | Confirm viewport in Phase 3 matches the mockup's target width; resize mockup if needed |
|
|
538
|
+
| Visual findings dominated by `font-rendering-variance` | Anti-aliasing / subpixel differences across OS | Tolerance is doing its job — these auto-demote in strict mode when `confidence: low` |
|
|
539
|
+
| Self-heal loop never converges | `coder` introducing new regressions while fixing | After max iterations, override with reason and open a follow-up backlog card |
|
|
540
|
+
| `verifier_protocol_violation` in gaps | `visual-fidelity-verifier` returned prose mixed with JSON | Re-spawn the verifier once; if persistent, surface to user (likely a model regression) |
|
|
541
|
+
|
|
542
|
+
---
|
|
543
|
+
|
|
544
|
+
## Return Protocol
|
|
545
|
+
|
|
546
|
+
The skill's final message is one of:
|
|
547
|
+
|
|
548
|
+
- **Manual mode**: a short Markdown summary followed by a fenced JSON code
|
|
549
|
+
block with the full report. The JSON is the source of truth; the Markdown
|
|
550
|
+
is for human readability.
|
|
551
|
+
- **Programmatic mode**: a single JSON object matching the schema in Phase 6.
|
|
552
|
+
No prose preamble.
|