mandrel 1.82.0 → 1.83.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/README.md +41 -0
- package/.agents/docs/SDLC.md +4 -2
- package/.agents/docs/agentrc-reference.json +10 -2
- package/.agents/docs/execution-reference.md +52 -0
- package/.agents/instructions.md +76 -38
- package/.agents/schemas/agentrc.schema.json +31 -3
- package/.agents/schemas/qa-ledger.schema.json +2 -2
- package/.agents/scripts/epic-deliver-prepare.js +41 -1
- package/.agents/scripts/lib/config/explain.js +4 -1
- package/.agents/scripts/lib/config-settings-schema.js +25 -1
- package/.agents/scripts/lib/epic-body-sections.js +88 -0
- package/.agents/scripts/lib/findings/promote-finding.js +3 -3
- package/.agents/scripts/lib/findings/severity.js +5 -6
- package/.agents/scripts/lib/orchestration/check-baselines/phases/evaluate.js +65 -2
- package/.agents/scripts/lib/orchestration/context-hydration-engine.js +96 -11
- package/.agents/scripts/lib/orchestration/doc-reader.js +29 -0
- package/.agents/scripts/lib/orchestration/docs-digest.js +134 -0
- package/.agents/scripts/lib/orchestration/story-close/baseline-attribution/phases/refresh-commit.js +15 -1
- package/.agents/scripts/lib/qa/console-allowlist.js +5 -4
- package/.agents/scripts/lib/qa/resolve-qa-contract.js +144 -8
- package/.agents/skills/core/epic-plan-consolidate/SKILL.md +7 -5
- package/.agents/skills/core/epic-plan-consolidate/examples.md +51 -0
- package/.agents/skills/core/epic-plan-decompose-author/SKILL.md +4 -22
- package/.agents/skills/core/epic-plan-decompose-author/examples.md +47 -0
- package/.agents/skills/core/epic-plan-premortem/SKILL.md +9 -8
- package/.agents/skills/core/epic-plan-premortem/examples.md +53 -0
- package/.agents/skills/core/epic-plan-spec-author/SKILL.md +21 -81
- package/.agents/skills/core/epic-plan-spec-author/examples.md +91 -0
- package/.agents/skills/skills.index.json +3 -3
- package/.agents/skills/stack/qa/qa-explore-driving/SKILL.md +52 -38
- package/.agents/workflows/helpers/deliver-epic-reference.md +514 -0
- package/.agents/workflows/helpers/deliver-epic.md +164 -469
- package/.agents/workflows/helpers/epic-deliver-story.md +35 -11
- package/.agents/workflows/helpers/plan-epic-reference.md +136 -0
- package/.agents/workflows/helpers/plan-epic.md +56 -186
- package/.agents/workflows/helpers/plan-story.md +31 -61
- package/.agents/workflows/helpers/qa-run-scenario.md +194 -0
- package/.agents/workflows/helpers/scope-triage-gate.md +97 -0
- package/.agents/workflows/helpers/single-story-deliver-reference.md +423 -0
- package/.agents/workflows/helpers/single-story-deliver.md +128 -392
- package/.agents/workflows/qa-explore.md +63 -32
- package/.agents/workflows/qa-run.md +293 -130
- package/docs/CHANGELOG.md +14 -0
- package/package.json +1 -1
- package/.agents/schemas/qa-finding.schema.json +0 -133
|
@@ -7,18 +7,26 @@ description: Drive Gherkin scenarios through a real browser as an agent-driven Q
|
|
|
7
7
|
Execute a consumer's Gherkin `.feature` scenarios through a **real browser**
|
|
8
8
|
(the chrome-devtools MCP surface), with the agent acting as the step executor
|
|
9
9
|
and a human observing. The harness resolves the consumer's `qa` contract,
|
|
10
|
-
selects a concrete scenario set, signs in
|
|
11
|
-
|
|
10
|
+
**resolves a target environment**, selects a concrete scenario set, signs in
|
|
11
|
+
via the environment's configured seam, then delegates each scenario to
|
|
12
|
+
[`helpers/qa-run-scenario.md`](helpers/qa-run-scenario.md) — which navigates
|
|
13
|
+
**from a root** to drive each `Given/When/Then` and asserts `Then` outcomes
|
|
12
14
|
**semantically** against the accessibility snapshot. Per-surface console and
|
|
13
|
-
network are instrumented into structured findings; findings are
|
|
14
|
-
|
|
15
|
-
|
|
15
|
+
network are instrumented into structured findings; those findings are recorded
|
|
16
|
+
as `QaLedgerItem`s on the shared session ledger under `temp/qa/` and routed —
|
|
17
|
+
after the operator sign-off gate — through the same
|
|
18
|
+
classify/route/dedup/promote core `/qa-explore` and `/qa-assist` use, so re-run
|
|
19
|
+
sweeps dedup previously-filed findings instead of re-drafting them. The harness
|
|
20
|
+
never files tickets autonomously.
|
|
16
21
|
|
|
17
22
|
This workflow is the agent-driven successor to the framework's earlier
|
|
18
23
|
headless BDD runner. It is a **prose workflow**, not a Node orchestrator: the host LLM
|
|
19
24
|
executes the procedure; deterministic Node helpers under
|
|
20
|
-
`.agents/scripts/lib/qa/` do
|
|
21
|
-
console filtering
|
|
25
|
+
`.agents/scripts/lib/qa/` do contract resolution, environment resolution,
|
|
26
|
+
scenario selection, console filtering, evidence redaction, and session/ledger
|
|
27
|
+
resolution, and the shared findings core under `.agents/scripts/lib/findings/`
|
|
28
|
+
owns classification, dedup/route, and cluster/size/promote. The agent never
|
|
29
|
+
invents those decisions in prose.
|
|
22
30
|
|
|
23
31
|
> **When to run**: During sprint testing to exercise a targeted slice of the
|
|
24
32
|
> acceptance suite (a feature, a tag expression, or a domain), for regression
|
|
@@ -31,14 +39,24 @@ console filtering.
|
|
|
31
39
|
## Slash Command
|
|
32
40
|
|
|
33
41
|
```text
|
|
34
|
-
/qa-run <selector>
|
|
42
|
+
/qa-run [<env>] [<selector>]
|
|
35
43
|
```
|
|
36
44
|
|
|
37
45
|
### Arguments
|
|
38
46
|
|
|
47
|
+
Both arguments are **optional**. A bare `/qa-run` runs the interactive
|
|
48
|
+
env-then-scope flow (Step 0.5); supplying the arguments skips the corresponding
|
|
49
|
+
prompt.
|
|
50
|
+
|
|
39
51
|
| Name | Required | Shape / Example | Notes |
|
|
40
52
|
| ---------- | -------- | -------------------------------------------- | -------------------------------------------------------------------------------------------------- |
|
|
41
|
-
| `
|
|
53
|
+
| `env` | no | `local`, `staging`, `https://staging.app.example` | Selects one of the contract's `environments` (Step 0.5a). Omit to be **prompted** for the environment. A raw URL resolves by origin match. |
|
|
54
|
+
| `selector` | no | `feature:login`, `tag:@smoke and not @wip`, `domain:billing` | Scopes the sweep to a concrete scenario set. Omit to be **prompted** for scope (Step 0.5b). One of three kinds — see below. |
|
|
55
|
+
|
|
56
|
+
- **Bare `/qa-run`** → prompt for the environment (Step 0.5a), then prompt for
|
|
57
|
+
scope (Step 0.5b).
|
|
58
|
+
- **`/qa-run <env> <selector>`** → **skip both prompts**; resolve `<env>`
|
|
59
|
+
directly and resolve `<selector>` directly.
|
|
42
60
|
|
|
43
61
|
The selector is resolved by
|
|
44
62
|
[`resolve-selection.js`](../scripts/lib/qa/resolve-selection.js) into a
|
|
@@ -58,9 +76,11 @@ deterministic, `(file, line)`-sorted scenario set under the contract's
|
|
|
58
76
|
### Examples
|
|
59
77
|
|
|
60
78
|
```text
|
|
61
|
-
/qa-run
|
|
62
|
-
/qa-run
|
|
63
|
-
/qa-run
|
|
79
|
+
/qa-run # interactive: prompt env, then scope
|
|
80
|
+
/qa-run staging # env pinned, prompt for scope
|
|
81
|
+
/qa-run local feature:login # both pinned, no prompts
|
|
82
|
+
/qa-run staging "tag:@smoke and not @wip"
|
|
83
|
+
/qa-run https://staging.app.example domain:billing
|
|
64
84
|
```
|
|
65
85
|
|
|
66
86
|
The canonical tag taxonomy — `@smoke`, `@risk-high`, `@platform-web`,
|
|
@@ -83,14 +103,15 @@ node -e "import('./.agents/scripts/lib/qa/resolve-qa-contract.js').then(async (m
|
|
|
83
103
|
seam is `resolveQaContract(config)`.) The resolver returns the normalized
|
|
84
104
|
contract:
|
|
85
105
|
|
|
86
|
-
| Field
|
|
87
|
-
|
|
|
88
|
-
| `featureRoot`
|
|
89
|
-
| `fixturesManifest`
|
|
90
|
-
| `
|
|
91
|
-
| `
|
|
92
|
-
| `
|
|
93
|
-
| `
|
|
106
|
+
| Field | Use |
|
|
107
|
+
| -------------------- | ------------------------------------------------------------------------- |
|
|
108
|
+
| `featureRoot` | Root passed to `resolve-selection.js` for scenario discovery. |
|
|
109
|
+
| `fixturesManifest` | Persona → seed binding loaded before sign-in. |
|
|
110
|
+
| `environments` | Environment-keyed map (`{ baseUrl, signInSeam, allowWrites? }` per name). Resolved to a single target in Step 0.5a. |
|
|
111
|
+
| `defaultEnvironment` | Environment name used when no `<env>` argument is supplied and the operator accepts the default. |
|
|
112
|
+
| `personas` | Canonical object map keyed by persona name (`personaNames` lists the names). Authored as a plain name array under a `urlTemplate` seam, or as a per-persona credential/skill map under a `skill` (or credential) seam — see Step 2. |
|
|
113
|
+
| `consoleAllowlist` | Inline benign-console patterns (default `[]`) — see Step 4. |
|
|
114
|
+
| `designTokens` | Pointer to the token/style source for visual inspection (default `null`). |
|
|
94
115
|
|
|
95
116
|
### Loud-failure path (no `qa` block)
|
|
96
117
|
|
|
@@ -100,7 +121,7 @@ auto-detection — in three cases:
|
|
|
100
121
|
- **Block absent** (no `qa` key, or an empty `qa: {}` with no harness-required
|
|
101
122
|
fields): the error reads
|
|
102
123
|
_"qa: this project has not bound the QA harness — add a `qa` block to
|
|
103
|
-
.agentrc.json (featureRoot, fixturesManifest,
|
|
124
|
+
.agentrc.json (featureRoot, fixturesManifest, environments, personas) before
|
|
104
125
|
invoking the QA harness."_
|
|
105
126
|
- **Malformed shape** (wrong-typed field, unknown field): the error names the
|
|
106
127
|
offending field, e.g. `qa.featureRoot must be a string`.
|
|
@@ -123,9 +144,87 @@ clear error ("the chrome-devtools MCP server is unavailable; the QA harness
|
|
|
123
144
|
requires a live browser surface") and stop. Do not attempt a headless
|
|
124
145
|
fallback.
|
|
125
146
|
|
|
126
|
-
|
|
147
|
+
### Session & ledger (temp/qa/)
|
|
148
|
+
|
|
149
|
+
Resolve the session and its ledger path **once**, up front, via
|
|
150
|
+
[`qa-session.js`](../scripts/lib/qa/qa-session.js) — the same seam
|
|
151
|
+
`/qa-explore` and `/qa-assist` use:
|
|
127
152
|
|
|
128
|
-
|
|
153
|
+
```js
|
|
154
|
+
import { resolveQaSession } from '../scripts/lib/qa/qa-session.js';
|
|
155
|
+
const { sessionId, ledgerPath, reused, untriaged } = resolveQaSession({ config });
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
- The ledger is always written under **`temp/qa/<sessionId>.ndjson`**
|
|
159
|
+
(`<tempRoot>/qa/`, resolved from `project.paths.tempRoot`). It is one
|
|
160
|
+
`QaLedgerItem` per line (ndjson) validated against
|
|
161
|
+
[`qa-ledger.schema.json`](../schemas/qa-ledger.schema.json). **Never** write
|
|
162
|
+
the ledger anywhere else, and never commit it — `temp/` is gitignored per
|
|
163
|
+
[`.agents/instructions.md` § 6](../instructions.md).
|
|
164
|
+
- When `reused` is `true`, a prior session of the same id exists: **append**,
|
|
165
|
+
never overwrite, and carry the `untriaged` items forward as the rolling
|
|
166
|
+
backlog. Pass `--session-id <id>` (or `QA_SESSION_ID`) to resume a named
|
|
167
|
+
session.
|
|
168
|
+
|
|
169
|
+
The sweep's `F#` findings (Step 4) are recorded as `QaLedgerItem`s on this
|
|
170
|
+
ledger, and Step 5 routes the ledger through the shared
|
|
171
|
+
classify/route/dedup/promote core — there is no separate `/qa-run` finding
|
|
172
|
+
schema or draft-bundle path.
|
|
173
|
+
|
|
174
|
+
## Step 0.5 — Resolve the environment, then the scope (interactive when unargued)
|
|
175
|
+
|
|
176
|
+
### Step 0.5a — Resolve the target environment
|
|
177
|
+
|
|
178
|
+
Resolve which of the contract's `environments` this sweep runs against through
|
|
179
|
+
[`resolveQaEnvironment`](../scripts/lib/qa/resolve-qa-contract.js). It returns
|
|
180
|
+
the resolved target `{ name, baseUrl, signInSeam, allowWrites }` (with
|
|
181
|
+
`allowWrites` defaulted to an explicit boolean — `true` only for the
|
|
182
|
+
conventional `local` environment, `false` for every other target unless the
|
|
183
|
+
consumer opts in).
|
|
184
|
+
|
|
185
|
+
- **`<env>` argument supplied** → pass it straight to `resolveQaEnvironment`.
|
|
186
|
+
An exact environment name wins; a raw URL resolves by **origin match**
|
|
187
|
+
against each environment's `baseUrl`. An unknown name or an unmatched URL
|
|
188
|
+
**throws** — relay the resolver's message (it names the known environments)
|
|
189
|
+
and stop; do **not** guess an environment.
|
|
190
|
+
- **No `<env>` argument (bare `/qa-run`)** → **prompt the operator**. Enumerate
|
|
191
|
+
every environment as `name → baseUrl` (marking `defaultEnvironment`), and ask
|
|
192
|
+
which to run against. The operator may answer with an environment **name** or
|
|
193
|
+
paste a **raw URL** — resolve either through `resolveQaEnvironment` (a raw
|
|
194
|
+
URL matches by origin). If the answer resolves to nothing, **fail loudly**
|
|
195
|
+
with the resolver's known-environments message; do not silently fall back to
|
|
196
|
+
the default.
|
|
197
|
+
|
|
198
|
+
Carry the resolved `{ name, baseUrl, allowWrites }` forward — `baseUrl` is the
|
|
199
|
+
navigation root, and `allowWrites` drives the write guard (Step 1) and is part
|
|
200
|
+
of the per-scenario helper's input contract (Step 3).
|
|
201
|
+
|
|
202
|
+
### Step 0.5b — Select the scope
|
|
203
|
+
|
|
204
|
+
- **`<selector>` argument supplied** → parse it into the resolver's selector
|
|
205
|
+
shape and proceed to Step 1.
|
|
206
|
+
- **No `<selector>` argument (bare `/qa-run`)** → **prompt the operator** for
|
|
207
|
+
scope. Enumerate the selectable scope under `featureRoot` via
|
|
208
|
+
[`resolve-selection.js`](../scripts/lib/qa/resolve-selection.js): the
|
|
209
|
+
available **domains** (first-level `featureRoot` subdirectories), **feature
|
|
210
|
+
stems** (`.feature` path stems), and **canonical tags** (the tag atoms the
|
|
211
|
+
scanner collected across the tree, per the `.agents/rules/gherkin-standards.md`
|
|
212
|
+
taxonomy). Present them as a **multi-select** — the operator may pick any
|
|
213
|
+
combination of domains, feature stems, and tags, which compose into the
|
|
214
|
+
selector set. Always include an explicit final option:
|
|
215
|
+
|
|
216
|
+
> **No coverage here → hand off to `/qa-explore`.** When the surface the
|
|
217
|
+
> operator wants to test has no authored `.feature` coverage, choosing this
|
|
218
|
+
> option ends the sweep and hands off to [`/qa-explore`](qa-explore.md) for
|
|
219
|
+
> agent-led exploratory QA, rather than running an empty selection.
|
|
220
|
+
|
|
221
|
+
If the operator picks the hand-off option, stop the sweep and route to
|
|
222
|
+
`/qa-explore`. Otherwise resolve the multi-select into the concrete selector
|
|
223
|
+
set and proceed to Step 1.
|
|
224
|
+
|
|
225
|
+
## Step 1 — Select the scenario set (and apply the write guard)
|
|
226
|
+
|
|
227
|
+
Pass the resolved selector(s) and the contract's `featureRoot` to
|
|
129
228
|
[`resolveSelection`](../scripts/lib/qa/resolve-selection.js). It returns
|
|
130
229
|
`{ kind, featureRoot, files, scenarios }` where `scenarios` is the
|
|
131
230
|
`(file, line)`-sorted set the sweep will execute. Determinism is load-bearing:
|
|
@@ -137,150 +236,214 @@ sign-in. If the selection is empty, report "no scenarios matched
|
|
|
137
236
|
`<selector>`" and stop — an empty selection is operator error (a typo'd
|
|
138
237
|
feature id or domain), not a passing sweep.
|
|
139
238
|
|
|
140
|
-
|
|
239
|
+
### `allowWrites` guardrail (non-local safety)
|
|
240
|
+
|
|
241
|
+
When the resolved environment has **`allowWrites: false`**, exclude any
|
|
242
|
+
scenario judged **mutating** from the selection before driving. Judge a
|
|
243
|
+
scenario mutating from its **`When` steps** — a `When` that creates, updates,
|
|
244
|
+
or deletes persisted state (submits a form that writes, deletes a record,
|
|
245
|
+
changes a setting) is mutating; a read-only navigation/inspection `When` is
|
|
246
|
+
not. Report the **exclusion count** ("N mutating scenarios excluded on
|
|
247
|
+
read-only `<env>`") alongside the resolved scenario count so the operator sees
|
|
248
|
+
what was skipped and why. The exclusion is overridable **only** by an explicit
|
|
249
|
+
in-session operator confirmation (the operator affirms, in this session, that
|
|
250
|
+
writes to `<env>` are acceptable) — never silently include mutating scenarios
|
|
251
|
+
on a read-only target, and never widen `allowWrites` by editing the contract
|
|
252
|
+
mid-sweep.
|
|
141
253
|
|
|
142
|
-
|
|
143
|
-
|
|
254
|
+
## Step 2 — Sign in via the environment's `signInSeam`
|
|
255
|
+
|
|
256
|
+
Sign in **once per persona** before driving that persona's scenarios, using the
|
|
257
|
+
resolved environment's discriminated-union seam
|
|
258
|
+
(`environment.signInSeam`, anchored on `environment.baseUrl`):
|
|
144
259
|
|
|
145
260
|
- **`kind: 'url'`** — substitute `{persona}` into `template` (e.g.
|
|
146
261
|
`/dev/sign-in-as/{persona}` → `/dev/sign-in-as/admin`) and `navigate_page`
|
|
147
|
-
to the resulting dev seam URL.
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
is authored as a plain name array (`personas: ["athlete", "coach"]`).
|
|
262
|
+
to the resulting dev seam URL. The persona **name** (a `personaNames` entry)
|
|
263
|
+
is the **sole input** the seam consumes — per-persona auth material is
|
|
264
|
+
neither needed nor read here, so under a `urlTemplate` seam the contract is
|
|
265
|
+
authored as a plain name array (`personas: ["athlete", "coach"]`).
|
|
152
266
|
- **`kind: 'skill'`** — invoke the named consumer skill for procedural
|
|
153
267
|
(multi-step or non-URL) sign-in. Read the skill's `SKILL.md` and follow it.
|
|
154
268
|
|
|
155
|
-
###
|
|
269
|
+
### Credentials under a skill seam (bounded rule)
|
|
270
|
+
|
|
271
|
+
Under a `skill` seam, real sign-in is permitted but **bounded**:
|
|
272
|
+
|
|
273
|
+
- Real auth uses **only `credentialRef`-indirected material** — the persona's
|
|
274
|
+
`credentialRef` names a stored credential the skill dereferences; raw
|
|
275
|
+
passwords, tokens, or API keys are never inlined into the contract, the
|
|
276
|
+
workflow, or chat.
|
|
277
|
+
- **Secrets are never echoed** into chat, findings, or the ledger — do not
|
|
278
|
+
print a credential, a session token, or a cookie value at any point.
|
|
279
|
+
- **Captured evidence passes `redact-evidence.js`** (`redactEvidence`) before
|
|
280
|
+
persistence, so any secret that leaks into console/network capture is
|
|
281
|
+
scrubbed before it reaches a finding (see Step 4 and the per-scenario
|
|
282
|
+
helper).
|
|
156
283
|
|
|
157
284
|
Per-persona auth material (`credentialRef` / `signInSkill`, authored via the
|
|
158
285
|
object-map `personas` shape) is consulted **only** under a `skill` or
|
|
159
|
-
credential seam
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
normalizes both authored shapes to one canonical object map keyed by persona
|
|
165
|
-
name; a name-only persona resolves to an empty record (no auth material).
|
|
286
|
+
credential seam. Under a `urlTemplate` dev-impersonation seam the persona name
|
|
287
|
+
is the only input, so the material is never read — author name-only personas
|
|
288
|
+
there rather than fabricating `credentialRef`/`signInSkill` values the harness
|
|
289
|
+
ignores. The resolver normalizes both authored shapes to one canonical object
|
|
290
|
+
map keyed by persona name; a name-only persona resolves to an empty record.
|
|
166
291
|
|
|
167
292
|
After sign-in, confirm the authenticated state with a `take_snapshot`
|
|
168
293
|
(e.g. the user menu or persona badge is present) before driving any scenario.
|
|
294
|
+
This confirmed authenticated session is the precondition the per-scenario
|
|
295
|
+
helper's input contract requires (Step 3).
|
|
296
|
+
|
|
297
|
+
## Step 3 — Drive each scenario via the per-scenario helper
|
|
298
|
+
|
|
299
|
+
For each scenario in selection order, delegate driving, analysis, and reporting
|
|
300
|
+
to [`helpers/qa-run-scenario.md`](helpers/qa-run-scenario.md). Pass its input
|
|
301
|
+
contract:
|
|
169
302
|
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
1. **
|
|
226
|
-
`
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
[`
|
|
249
|
-
|
|
250
|
-
`
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
|
|
303
|
+
- **`environment`** — the resolved `{ name, baseUrl, allowWrites }` from
|
|
304
|
+
Step 0.5a.
|
|
305
|
+
- **`persona`** — the persona name **plus** the confirmed authenticated-session
|
|
306
|
+
precondition established in Step 2 (the helper re-verifies it on entry).
|
|
307
|
+
- **`scenario`** — the scenario ref (`.feature` file path and `(file, line)`
|
|
308
|
+
locator).
|
|
309
|
+
- **`consoleAllowlist`** and **`designTokens`** — from the resolved contract.
|
|
310
|
+
|
|
311
|
+
The helper owns the navigation-first / never-URL-jump rule, the semantic-`Then`
|
|
312
|
+
assertion against the accessibility snapshot, the per-`When` write guard under
|
|
313
|
+
`allowWrites: false`, and mandatory evidence redaction. It returns **one
|
|
314
|
+
structured per-scenario result** — `{ scenario, intent, verdict (pass | fail |
|
|
315
|
+
blocked), surface, findings[] }`. Collect one result per scenario; the sweep
|
|
316
|
+
report shape (per-scenario `intent + verdict` lines plus totals, Step 6) is
|
|
317
|
+
unchanged from the inlined procedure.
|
|
318
|
+
|
|
319
|
+
## Step 4 — Instrument & record findings onto the ledger
|
|
320
|
+
|
|
321
|
+
The per-scenario helper captures console and network per surface and turns
|
|
322
|
+
genuine problems into structured findings, applying the contract's
|
|
323
|
+
`consoleAllowlist` via
|
|
324
|
+
[`filterConsoleMessages`](../scripts/lib/qa/console-allowlist.js) (each
|
|
325
|
+
non-allowlisted console **error** becomes one `F#` finding; allowlisted
|
|
326
|
+
patterns and non-error levels are suppressed) and, when `designTokens` is set,
|
|
327
|
+
spot-checking the surface against the token source. The allowlist is a **noise
|
|
328
|
+
filter, not a security control** — never expand it to silence a genuine error
|
|
329
|
+
signal.
|
|
330
|
+
|
|
331
|
+
Each returned `F#` finding — the console/network-derived shape
|
|
332
|
+
`{ id, classification, surface, symptom, likelyRootCause, disposition,
|
|
333
|
+
acceptance, evidence: { console[], network[] } }`, with evidence already
|
|
334
|
+
scrubbed of tokens, session cookies, and PII via
|
|
335
|
+
[`redact-evidence.js`](../scripts/lib/qa/redact-evidence.js) per
|
|
336
|
+
`.agents/rules/security-baseline.md` — is **recorded as a `QaLedgerItem`** on
|
|
337
|
+
the session ledger resolved in Step 0 (`temp/qa/<sessionId>.ndjson`,
|
|
338
|
+
[`qa-ledger.schema.json`](../schemas/qa-ledger.schema.json)). Map the finding
|
|
339
|
+
onto the ledger shape: a stable `id` (`L1`, `L2`, … in append order), the
|
|
340
|
+
finding's `symptom` as the scrubbed `evidence`, the finding's `surface` as
|
|
341
|
+
`coverage`, a `class` (map the finding's classification onto the
|
|
342
|
+
ledger `class` enum — a product defect is `product-bug`, a tooling gap is
|
|
343
|
+
`tooling-dx`, etc.) and `severity`, `missingTest` (`null` when no test gap
|
|
344
|
+
applies), and `disposition` left untriaged. **Append** to the ledger, never
|
|
345
|
+
overwrite; a re-run appends to the same session. This is the single findings
|
|
346
|
+
channel — there is no separate `/qa-run` finding schema or draft bundle.
|
|
347
|
+
|
|
348
|
+
## Step 5 — Triage the ledger (operator sign-off required)
|
|
349
|
+
|
|
350
|
+
After the sweep completes, route the ledger through the same
|
|
351
|
+
classify/route/dedup/promote core `/qa-explore` Triage and `/qa-assist` use.
|
|
352
|
+
The **operator sign-off gate is preserved**: the harness MUST NOT create
|
|
353
|
+
tickets autonomously — present the routed dispositions and confirm each
|
|
354
|
+
`file` / `defer` / `dismiss` with the operator before any write.
|
|
355
|
+
|
|
356
|
+
For each untriaged ledger item:
|
|
357
|
+
|
|
358
|
+
1. **Classify** it via
|
|
359
|
+
[`classify-finding.js`](../scripts/lib/findings/classify-finding.js). The
|
|
360
|
+
item's `class` resolves to the focus/meta label set Triage applies when
|
|
361
|
+
promoting it. The helper **throws** on an absent/unknown class — fix the
|
|
362
|
+
ledger item's class rather than defaulting.
|
|
363
|
+
2. **Dedup / route** it against existing GitHub Issues via
|
|
364
|
+
[`route-finding.js`](../scripts/lib/findings/route-finding.js):
|
|
365
|
+
|
|
366
|
+
```js
|
|
367
|
+
import { routeFinding, fingerprintFooter } from '../scripts/lib/findings/route-finding.js';
|
|
368
|
+
const { decision, matchedIssue, fingerprint } =
|
|
369
|
+
await routeFinding(finding, { searchIssues });
|
|
370
|
+
```
|
|
371
|
+
|
|
372
|
+
`decision` is one of `new` / `update-existing` / `duplicate` /
|
|
373
|
+
`regression-of-closed`. Wire the `searchIssues` port to the GitHub provider,
|
|
374
|
+
querying **both open and closed** Issues, and stamp the
|
|
375
|
+
`fingerprintFooter(sha)` marker into any Issue body so future sweeps dedup
|
|
376
|
+
against it. This is the **single** dedup implementation shared with
|
|
377
|
+
`/qa-explore`, `/qa-assist`, and `audit-to-stories`.
|
|
378
|
+
3. **Decide the disposition** with the operator (`file` / `defer` / `dismiss`)
|
|
379
|
+
and record it back onto the ledger item.
|
|
380
|
+
4. **Promote the `file`-dispositioned findings through `/plan`** via
|
|
381
|
+
[`promote-finding.js`](../scripts/lib/findings/promote-finding.js) — the
|
|
382
|
+
same cluster/size/route/file path `/qa-explore` and `/audit-to-stories`
|
|
383
|
+
consume (`clusterLedgerItems` + `targetForCluster`: a cluster spanning ≤2
|
|
384
|
+
coverage surfaces routes to `createStory` via `/plan --from-notes`, >2 to
|
|
385
|
+
`createEpic` via `/plan --idea`, with the cluster's `fingerprintFooter(sha)`
|
|
386
|
+
stamped verbatim into the seed). A `file` disposition never opens a raw
|
|
387
|
+
GitHub Issue; only `defer` and `dismiss` skip the `/plan` handoff.
|
|
388
|
+
5. **Gate:** any ticket-filing, seed write, `/plan` invocation, or label
|
|
389
|
+
mutation is a write — confirm each with the operator before it happens.
|
|
390
|
+
|
|
391
|
+
If the run was triggered from an Epic-testing context, hand the approved,
|
|
392
|
+
promoted findings to the Epic-testing helper for attachment to the Epic's QA
|
|
255
393
|
evidence ticket.
|
|
256
394
|
|
|
257
395
|
## Step 6 — Report
|
|
258
396
|
|
|
259
397
|
Summarize the sweep in chat with:
|
|
260
398
|
|
|
261
|
-
-
|
|
399
|
+
- The resolved **environment** (`name → baseUrl`, and whether it is read-only)
|
|
400
|
+
and the selector applied.
|
|
401
|
+
- The resolved scenario count, plus the **`allowWrites` exclusion count** when
|
|
402
|
+
mutating scenarios were skipped on a read-only environment.
|
|
262
403
|
- Scenario totals: passed / failed / blocked.
|
|
263
|
-
- Findings totals by classification and
|
|
404
|
+
- Findings totals by classification, and the ledger routes
|
|
405
|
+
(`new` / `update-existing` / `duplicate` / `regression-of-closed`).
|
|
264
406
|
- A per-scenario line pairing each scenario's plain-English intent with its
|
|
265
407
|
verdict (pass / fail / blocked), grouped by feature file or domain — so the
|
|
266
408
|
digest reads as "what was checked → what happened", not a tag list.
|
|
267
409
|
- For each failure, the scenario name, file path, the surface it ended on, and
|
|
268
410
|
a one-line user-visible symptom.
|
|
269
|
-
-
|
|
411
|
+
- The ledger path (under `temp/qa/`) and a pointer to the routed dispositions
|
|
412
|
+
awaiting operator sign-off (if any).
|
|
270
413
|
|
|
271
414
|
## Constraints
|
|
272
415
|
|
|
273
416
|
- **Always** resolve the `qa` contract first and **fail loudly** when it is
|
|
274
417
|
absent or malformed. There is no auto-detection fallback.
|
|
418
|
+
- **Always** resolve a target environment (Step 0.5a) — prompt for it when no
|
|
419
|
+
`<env>` argument is supplied — and **fail loudly** on an unknown name or an
|
|
420
|
+
unmatched URL. Never silently fall back to the default on a bad answer.
|
|
421
|
+
- **Always** apply the `allowWrites` guardrail: on a read-only environment,
|
|
422
|
+
exclude mutating scenarios and report the exclusion count; include them only
|
|
423
|
+
on explicit in-session operator confirmation.
|
|
275
424
|
- **Always** navigate from a root via UI affordances. **Never** URL-jump to a
|
|
276
425
|
deep link to set up a scenario.
|
|
277
426
|
- **Always** assert `Then` outcomes semantically against the accessibility
|
|
278
427
|
snapshot. **Never** assert via DOM/CSS/XPath selectors, HTTP status codes,
|
|
279
428
|
response bodies, or DB rows inside a scenario — push those to the contract
|
|
280
429
|
tier per `.agents/rules/testing-standards.md`.
|
|
281
|
-
- **
|
|
282
|
-
|
|
283
|
-
|
|
430
|
+
- **Under a skill seam**, real sign-in uses only `credentialRef`-indirected
|
|
431
|
+
material; secrets are never echoed into chat, findings, or the ledger; and
|
|
432
|
+
captured evidence passes `redact-evidence.js` before persistence.
|
|
433
|
+
- **Always** record findings as `QaLedgerItem`s on the shared session ledger
|
|
434
|
+
under `temp/qa/` ([`qa-ledger.schema.json`](../schemas/qa-ledger.schema.json))
|
|
435
|
+
and route them through the shared classify/route/dedup/promote core — there
|
|
436
|
+
is no separate `/qa-run` finding schema or draft-bundle path. Never commit
|
|
437
|
+
the ledger.
|
|
438
|
+
- **Delegate Triage decisions to the helpers.** Classification
|
|
439
|
+
([`classify-finding.js`](../scripts/lib/findings/classify-finding.js)),
|
|
440
|
+
dedup/route ([`route-finding.js`](../scripts/lib/findings/route-finding.js),
|
|
441
|
+
fingerprint-footer against open + closed issues), and cluster/size/promote
|
|
442
|
+
([`promote-finding.js`](../scripts/lib/findings/promote-finding.js)) are
|
|
443
|
+
deterministic — never re-derive them in prose.
|
|
444
|
+
- **Never** file follow-up tickets autonomously; promote `file` findings
|
|
445
|
+
through `/plan` only after operator sign-off, and never open a raw GitHub
|
|
446
|
+
Issue for a `file` finding.
|
|
284
447
|
- **Never** expand `consoleAllowlist` to suppress genuine error signal — it is
|
|
285
448
|
a benign-noise filter, not a security control.
|
|
286
449
|
- **Always** scrub captured evidence of secrets and PII before rendering a
|
package/docs/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,20 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to this project will be documented in this file.
|
|
4
4
|
|
|
5
|
+
## [1.83.0](https://github.com/dsj1984/mandrel/compare/mandrel-v1.82.0...mandrel-v1.83.0) (2026-07-05)
|
|
6
|
+
|
|
7
|
+
|
|
8
|
+
### Added
|
|
9
|
+
|
|
10
|
+
* Epic [#4326](https://github.com/dsj1984/mandrel/issues/4326) ([#4343](https://github.com/dsj1984/mandrel/issues/4343)) ([b2c0d84](https://github.com/dsj1984/mandrel/commit/b2c0d8475615f92f1a4e52debe9cd38d7c4dc27c))
|
|
11
|
+
* Epic [#4337](https://github.com/dsj1984/mandrel/issues/4337) ([#4344](https://github.com/dsj1984/mandrel/issues/4344)) ([a9e1180](https://github.com/dsj1984/mandrel/commit/a9e11808634a8b03f889654e80da97f6efbe3939))
|
|
12
|
+
|
|
13
|
+
|
|
14
|
+
### Fixed
|
|
15
|
+
|
|
16
|
+
* **baselines:** honor maintainability ignoreGlobs in story-close auto-refresh + floor check ([#4347](https://github.com/dsj1984/mandrel/issues/4347)) ([10a4bd8](https://github.com/dsj1984/mandrel/commit/10a4bd86f182fc4aa22800f09173dcd0ebd2ebaa))
|
|
17
|
+
* **hydration:** gate AC/verify strip on each dedicated section separately ([#4346](https://github.com/dsj1984/mandrel/issues/4346)) ([9896588](https://github.com/dsj1984/mandrel/commit/9896588c3b8c5808d2fbd74c695138c9fb657b0c))
|
|
18
|
+
|
|
5
19
|
## [1.82.0](https://github.com/dsj1984/mandrel/compare/mandrel-v1.81.0...mandrel-v1.82.0) (2026-07-04)
|
|
6
20
|
|
|
7
21
|
|
package/package.json
CHANGED