mandrel 2.7.0 → 2.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/README.md +104 -330
- package/.agents/agents/auditor.md +135 -0
- package/.agents/agents/plan-critic.md +80 -0
- package/.agents/audit-checklists/dependencies.md +7 -0
- package/.agents/audit-checklists/documentation.md +1 -0
- package/.agents/docs/SDLC.md +69 -162
- package/.agents/docs/configuration.md +159 -499
- package/.agents/docs/quality-gates.md +59 -180
- package/.agents/instructions.md +170 -295
- package/.agents/rules/changelog-style.md +8 -66
- package/.agents/rules/ci-remediation.md +65 -124
- package/.agents/rules/gherkin-standards.md +10 -31
- package/.agents/rules/git-conventions-reference.md +28 -61
- package/.agents/rules/git-conventions.md +1 -1
- package/.agents/rules/orchestration-error-handling.md +5 -15
- package/.agents/rules/security-baseline.md +7 -13
- package/.agents/rules/shell-conventions.md +4 -13
- package/.agents/rules/test-seams.md +2 -2
- package/.agents/rules/testing-standards.md +7 -17
- package/.agents/schemas/agentrc.schema.json +21 -0
- package/.agents/scripts/audit-to-stories.js +51 -0
- package/.agents/scripts/lib/audit-to-stories/dedupe-against-github.js +120 -55
- package/.agents/scripts/lib/config-settings-schema.js +32 -0
- package/.agents/scripts/lib/findings/semantic-issue-search.js +43 -5
- package/.agents/scripts/lib/observability/terse-result.js +114 -0
- package/.agents/scripts/lib/orchestration/complexity-gate.js +207 -0
- package/.agents/scripts/lib/orchestration/plan-context.js +3 -0
- package/.agents/scripts/lib/orchestration/single-story-close/phases/auto-merge.js +221 -8
- package/.agents/scripts/lib/orchestration/single-story-close/runner.js +55 -14
- package/.agents/scripts/lib/orchestration/story-close/emit-blocked.js +9 -3
- package/.agents/scripts/lib/orchestration/story-deliver-terminal.js +4 -1
- package/.agents/scripts/lib/orchestration/task-body-validator.js +13 -40
- package/.agents/scripts/lib/story-body/body-format-lints.js +215 -0
- package/.agents/scripts/lib/story-body/story-body.js +18 -2
- package/.agents/scripts/lib/templates/decomposer-prompts.js +16 -0
- package/.agents/scripts/providers/github/issues.js +54 -7
- package/.agents/scripts/providers/github/search-budget.js +124 -0
- package/.agents/scripts/providers/github/search-query.js +71 -0
- package/.agents/scripts/single-story-confirm-merge.js +14 -5
- package/.agents/scripts/single-story-init.js +19 -3
- package/.agents/scripts/sync-branch-from-base.js +9 -3
- package/.agents/skills/core/api-and-interface-design/SKILL.md +23 -297
- package/.agents/skills/core/api-and-interface-design/reference.md +76 -0
- package/.agents/skills/core/browser-testing-with-devtools/SKILL.md +20 -327
- package/.agents/skills/core/browser-testing-with-devtools/reference.md +74 -0
- package/.agents/skills/core/code-review-and-quality/reference.md +16 -398
- package/.agents/skills/core/debugging-and-error-recovery/reference.md +14 -281
- package/.agents/skills/core/documentation-and-adrs/reference.md +42 -338
- package/.agents/skills/core/git-workflow-and-versioning/SKILL.md +28 -326
- package/.agents/skills/core/idea-refinement/SKILL.md +67 -193
- package/.agents/skills/core/security-and-hardening/SKILL.md +15 -31
- package/.agents/skills/core/security-and-hardening/reference.md +15 -273
- package/.agents/skills/skills.index.json +5 -5
- package/.agents/skills/stack/qa/gherkin-authoring/SKILL.md +16 -222
- package/.agents/skills/stack/qa/gherkin-authoring/reference.md +157 -0
- package/.agents/skills/stack/qa/playwright/SKILL.md +0 -29
- package/.agents/skills/stack/qa/qa-explore-driving/SKILL.md +19 -23
- package/.agents/skills/stack/qa/qa-harness/SKILL.md +35 -53
- package/.agents/skills/stack/qa/vitest/SKILL.md +0 -29
- package/.agents/workflows/audit-accessibility.md +34 -108
- package/.agents/workflows/audit-architecture.md +35 -123
- package/.agents/workflows/audit-clean-code.md +19 -78
- package/.agents/workflows/audit-data-model.md +32 -100
- package/.agents/workflows/audit-dependencies.md +47 -111
- package/.agents/workflows/audit-devops.md +16 -83
- package/.agents/workflows/audit-documentation.md +46 -93
- package/.agents/workflows/audit-navigability.md +26 -80
- package/.agents/workflows/audit-performance.md +40 -106
- package/.agents/workflows/audit-privacy.md +17 -80
- package/.agents/workflows/audit-quality.md +35 -99
- package/.agents/workflows/audit-security.md +20 -78
- package/.agents/workflows/audit-seo.md +20 -98
- package/.agents/workflows/audit-sre.md +20 -88
- package/.agents/workflows/audit-to-stories.md +1 -8
- package/.agents/workflows/audit-ux-ui.md +17 -80
- package/.agents/workflows/deliver.md +54 -9
- package/.agents/workflows/git-cleanup.md +50 -275
- package/.agents/workflows/helpers/audit-lens-core.md +230 -0
- package/.agents/workflows/helpers/code-review.md +11 -23
- package/.agents/workflows/helpers/deliver-story-reference.md +114 -17
- package/.agents/workflows/helpers/deliver-story.md +36 -186
- package/.agents/workflows/helpers/qa-core.md +174 -0
- package/.agents/workflows/helpers/qa-run-scenario-reference.md +35 -0
- package/.agents/workflows/helpers/qa-run-scenario.md +11 -25
- package/.agents/workflows/helpers/worktree-lifecycle.md +6 -67
- package/.agents/workflows/mandrel-update.md +7 -13
- package/.agents/workflows/plan.md +44 -17
- package/.agents/workflows/qa-assist.md +140 -269
- package/.agents/workflows/qa-explore.md +125 -316
- package/.agents/workflows/qa-run.md +180 -380
- package/docs/CHANGELOG.md +30 -0
- package/package.json +1 -1
- package/.agents/workflows/helpers/audit-dual-path.md +0 -59
- package/.agents/workflows/helpers/audit-self-check.md +0 -70
- package/.agents/workflows/helpers/audit-severity-scale.md +0 -19
|
@@ -6,35 +6,31 @@ description: Drive Gherkin scenarios through a real browser as an agent-driven Q
|
|
|
6
6
|
|
|
7
7
|
Execute a consumer's Gherkin `.feature` scenarios through a **real browser**
|
|
8
8
|
(the chrome-devtools MCP surface), with the agent acting as the step executor
|
|
9
|
-
and a human observing. The
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
[`helpers/qa-run-scenario.md`](helpers/qa-run-scenario.md)
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
owns classification, dedup/route, and cluster/size/promote. The agent never
|
|
29
|
-
invents those decisions in prose.
|
|
30
|
-
|
|
31
|
-
> **When to run**: During sprint testing to exercise a targeted slice of the
|
|
9
|
+
and a human observing. The sweep resolves a **run envelope** — a target
|
|
10
|
+
environment, a concrete scenario set, and an authenticated persona session —
|
|
11
|
+
then delegates each scenario to
|
|
12
|
+
[`helpers/qa-run-scenario.md`](helpers/qa-run-scenario.md), which drives it
|
|
13
|
+
navigation-first and asserts `Then` outcomes semantically against the
|
|
14
|
+
accessibility snapshot. Per-surface console and network are captured as
|
|
15
|
+
structured findings, recorded as `QaLedgerItem`s on the shared session ledger,
|
|
16
|
+
and triaged — after operator sign-off — through the shared classify/route/
|
|
17
|
+
dedup/promote core. The harness never files tickets autonomously.
|
|
18
|
+
|
|
19
|
+
The shared machinery — contract resolution + loud failure, the session & ledger
|
|
20
|
+
contract, redact-first, the `QaLedgerItem` shape, the triage procedure, and the
|
|
21
|
+
HITL write gate — lives once in [`helpers/qa-core.md`](helpers/qa-core.md); this
|
|
22
|
+
workflow states only the `/qa-run`-specific phases (env → scope → sign-in →
|
|
23
|
+
drive) plus a Constraints delta. Deterministic Node helpers under
|
|
24
|
+
`.agents/scripts/lib/qa/` own contract, environment, and scenario resolution;
|
|
25
|
+
the agent never invents those decisions in prose.
|
|
26
|
+
|
|
27
|
+
> **When to run**: during sprint testing to exercise a targeted slice of the
|
|
32
28
|
> acceptance suite (a feature, a tag expression, or a domain), for regression
|
|
33
29
|
> passes before `/deliver`, or on demand while debugging a Story's
|
|
34
30
|
> user-visible behavior in a live browser.
|
|
35
31
|
>
|
|
36
|
-
> **
|
|
37
|
-
>
|
|
32
|
+
> **Skills**: `stack/qa/gherkin-authoring`, `stack/qa/playwright-bdd`
|
|
33
|
+
> (authoring reference; this harness owns execution)
|
|
38
34
|
|
|
39
35
|
## Slash Command
|
|
40
36
|
|
|
@@ -45,33 +41,26 @@ invents those decisions in prose.
|
|
|
45
41
|
### Arguments
|
|
46
42
|
|
|
47
43
|
Both arguments are **optional**. A bare `/qa-run` runs the interactive
|
|
48
|
-
env-then-scope flow
|
|
49
|
-
prompt.
|
|
44
|
+
env-then-scope flow; supplying an argument skips the corresponding prompt.
|
|
50
45
|
|
|
51
46
|
| Name | Required | Shape / Example | Notes |
|
|
52
47
|
| ---------- | -------- | -------------------------------------------- | -------------------------------------------------------------------------------------------------- |
|
|
53
|
-
| `env` | no | `local`, `staging`, `https://staging.app.example` | Selects one of the contract's `environments
|
|
54
|
-
| `selector` | no | `feature:login`, `tag:@smoke and not @wip`, `domain:billing` | Scopes the sweep to a concrete scenario set. Omit to be **prompted** for scope
|
|
55
|
-
|
|
56
|
-
- **Bare `/qa-run`** → prompt for the environment (Step 0.5a), then prompt for
|
|
57
|
-
scope (Step 0.5b).
|
|
58
|
-
- **`/qa-run <env> <selector>`** → **skip both prompts**; resolve `<env>`
|
|
59
|
-
directly and resolve `<selector>` directly.
|
|
48
|
+
| `env` | no | `local`, `staging`, `https://staging.app.example` | Selects one of the contract's `environments`. Omit to be **prompted**. A raw URL resolves by origin match. |
|
|
49
|
+
| `selector` | no | `feature:login`, `tag:@smoke and not @wip`, `domain:billing` | Scopes the sweep to a concrete scenario set. Omit to be **prompted** for scope. One of three kinds — see below. |
|
|
60
50
|
|
|
61
51
|
The selector is resolved by
|
|
62
52
|
[`resolve-selection.js`](../scripts/lib/qa/resolve-selection.js) into a
|
|
63
|
-
deterministic, `(file, line)`-sorted scenario set under
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
- **`feature:<id>`**
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
- **`tag:<expression>`**
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
`featureRoot`-relative subdirectory `name`.
|
|
53
|
+
deterministic, `(file, line)`-sorted scenario set under `featureRoot`. Its
|
|
54
|
+
three kinds:
|
|
55
|
+
|
|
56
|
+
- **`feature:<id>`** — the single `.feature` file whose `featureRoot`-relative
|
|
57
|
+
path stem (or basename) equals the id (case-insensitive; ambiguous ids throw
|
|
58
|
+
— qualify with a relative path).
|
|
59
|
+
- **`tag:<expression>`** — the scenario set whose tags satisfy the cucumber
|
|
60
|
+
boolean expression (`@tag` atoms with `and` / `or` / `not` and parentheses;
|
|
61
|
+
quote expressions containing spaces).
|
|
62
|
+
- **`domain:<name>`** — every scenario under the `featureRoot`-relative
|
|
63
|
+
subdirectory `name`.
|
|
75
64
|
|
|
76
65
|
### Examples
|
|
77
66
|
|
|
@@ -83,365 +72,176 @@ deterministic, `(file, line)`-sorted scenario set under the contract's
|
|
|
83
72
|
/qa-run https://staging.app.example domain:billing
|
|
84
73
|
```
|
|
85
74
|
|
|
86
|
-
The canonical tag taxonomy
|
|
87
|
-
`@
|
|
88
|
-
|
|
89
|
-
file; add new tags to the rule first.
|
|
75
|
+
The canonical tag taxonomy (`@smoke`, `@risk-high`, `@platform-*`,
|
|
76
|
+
`@domain-*`, and the allowed extension syntax) is defined in
|
|
77
|
+
[`.agents/rules/gherkin-standards.md`](../rules/gherkin-standards.md). Do not
|
|
78
|
+
invent tags inside a feature file; add new tags to the rule first.
|
|
90
79
|
|
|
91
|
-
## Step 0 — Resolve the
|
|
80
|
+
## Step 0 — Resolve the run envelope
|
|
92
81
|
|
|
93
|
-
The
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
82
|
+
The outcome of Steps 0–2 is a single resolved envelope — **`{ environment,
|
|
83
|
+
scenario set, authenticated persona session }`** — that the per-scenario driver
|
|
84
|
+
(Step 3) consumes. The deterministic resolvers own every decision; the agent
|
|
85
|
+
narrates none of their internal return shapes. Every resolver **throws loudly**
|
|
86
|
+
on bad input, and the terminal behavior is the same in every case: **relay the
|
|
87
|
+
resolver's verbatim message and STOP** — never guess an environment, a
|
|
88
|
+
`featureRoot`, or a sign-in seam.
|
|
97
89
|
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
90
|
+
First resolve the `qa` contract and the session per
|
|
91
|
+
[`helpers/qa-core.md`](helpers/qa-core.md) (contract resolution + loud failure;
|
|
92
|
+
session & ledger under `temp/qa/`). Then build the rest of the envelope:
|
|
101
93
|
|
|
102
|
-
|
|
103
|
-
seam is `resolveQaContract(config)`.) The resolver returns the normalized
|
|
104
|
-
contract:
|
|
105
|
-
|
|
106
|
-
| Field | Use |
|
|
107
|
-
| -------------------- | ------------------------------------------------------------------------- |
|
|
108
|
-
| `featureRoot` | Root passed to `resolve-selection.js` for scenario discovery. |
|
|
109
|
-
| `fixturesManifest` | Persona → seed binding loaded before sign-in. |
|
|
110
|
-
| `environments` | Environment-keyed map (`{ baseUrl, signInSeam, allowWrites? }` per name). Resolved to a single target in Step 0.5a. |
|
|
111
|
-
| `defaultEnvironment` | Environment name used when no `<env>` argument is supplied and the operator accepts the default. |
|
|
112
|
-
| `personas` | Canonical object map keyed by persona name (`personaNames` lists the names). Authored as a plain name array under a `urlTemplate` seam, or as a per-persona credential/skill map under a `skill` (or credential) seam — see Step 2. |
|
|
113
|
-
| `consoleAllowlist` | Inline benign-console patterns (default `[]`) — see Step 4. |
|
|
114
|
-
| `designTokens` | Pointer to the token/style source for visual inspection (default `null`). |
|
|
115
|
-
|
|
116
|
-
### Loud-failure path (no `qa` block)
|
|
117
|
-
|
|
118
|
-
`resolveQaContract` **throws** — there is no silent fallback to
|
|
119
|
-
auto-detection — in three cases:
|
|
120
|
-
|
|
121
|
-
- **Block absent** (no `qa` key, or an empty `qa: {}` with no harness-required
|
|
122
|
-
fields): the error reads
|
|
123
|
-
_"qa: this project has not bound the QA harness — add a `qa` block to
|
|
124
|
-
.agentrc.json (featureRoot, fixturesManifest, environments, personas) before
|
|
125
|
-
invoking the QA harness."_
|
|
126
|
-
- **Malformed shape** (wrong-typed field, unknown field): the error names the
|
|
127
|
-
offending field, e.g. `qa.featureRoot must be a string`.
|
|
128
|
-
- **Missing required field**: the error names the first missing field.
|
|
129
|
-
|
|
130
|
-
When you hit any of these, **STOP immediately**. Relay the resolver's
|
|
131
|
-
verbatim message to the operator as the harness's terminal output and do not
|
|
132
|
-
proceed to browser execution. Do not invent a `featureRoot`, do not guess a
|
|
133
|
-
sign-in seam, and do not fall back to any retired headless BDD runner. The
|
|
134
|
-
loud failure is the contract: a consumer that has not bound the harness has
|
|
135
|
-
not opted into it.
|
|
136
|
-
|
|
137
|
-
### MCP availability check
|
|
94
|
+
### The chrome-devtools MCP surface must be available
|
|
138
95
|
|
|
139
96
|
The chrome-devtools MCP surface (`navigate_page`, `take_snapshot`, `click`,
|
|
140
97
|
`fill_form`, `evaluate_script`, `wait_for`, `list_console_messages`,
|
|
141
|
-
`list_network_requests`) is **host-provided** —
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
[`
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
`
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
and stop; do **not** guess an environment.
|
|
190
|
-
- **No `<env>` argument (bare `/qa-run`)** → **prompt the operator**. Enumerate
|
|
191
|
-
every environment as `name → baseUrl` (marking `defaultEnvironment`), and ask
|
|
192
|
-
which to run against. The operator may answer with an environment **name** or
|
|
193
|
-
paste a **raw URL** — resolve either through `resolveQaEnvironment` (a raw
|
|
194
|
-
URL matches by origin). If the answer resolves to nothing, **fail loudly**
|
|
195
|
-
with the resolver's known-environments message; do not silently fall back to
|
|
196
|
-
the default.
|
|
197
|
-
|
|
198
|
-
Carry the resolved `{ name, baseUrl, allowWrites }` forward — `baseUrl` is the
|
|
199
|
-
navigation root, and `allowWrites` drives the write guard (Step 1) and is part
|
|
200
|
-
of the per-scenario helper's input contract (Step 3).
|
|
201
|
-
|
|
202
|
-
### Step 0.5b — Select the scope
|
|
203
|
-
|
|
204
|
-
- **`<selector>` argument supplied** → parse it into the resolver's selector
|
|
205
|
-
shape and proceed to Step 1.
|
|
206
|
-
- **No `<selector>` argument (bare `/qa-run`)** → **prompt the operator** for
|
|
207
|
-
scope. Enumerate the selectable scope under `featureRoot` via
|
|
208
|
-
[`resolve-selection.js`](../scripts/lib/qa/resolve-selection.js): the
|
|
209
|
-
available **domains** (first-level `featureRoot` subdirectories), **feature
|
|
210
|
-
stems** (`.feature` path stems), and **canonical tags** (the tag atoms the
|
|
211
|
-
scanner collected across the tree, per the `.agents/rules/gherkin-standards.md`
|
|
212
|
-
taxonomy). Present them as a **multi-select** — the operator may pick any
|
|
213
|
-
combination of domains, feature stems, and tags, which compose into the
|
|
214
|
-
selector set. Always include an explicit final option:
|
|
215
|
-
|
|
216
|
-
> **No coverage here → hand off to `/qa-explore`.** When the surface the
|
|
217
|
-
> operator wants to test has no authored `.feature` coverage, choosing this
|
|
218
|
-
> option ends the sweep and hands off to [`/qa-explore`](qa-explore.md) for
|
|
219
|
-
> agent-led exploratory QA, rather than running an empty selection.
|
|
220
|
-
|
|
221
|
-
If the operator picks the hand-off option, stop the sweep and route to
|
|
222
|
-
`/qa-explore`. Otherwise resolve the multi-select into the concrete selector
|
|
223
|
-
set and proceed to Step 1.
|
|
224
|
-
|
|
225
|
-
## Step 1 — Select the scenario set (and apply the write guard)
|
|
226
|
-
|
|
227
|
-
Pass the resolved selector(s) and the contract's `featureRoot` to
|
|
228
|
-
[`resolveSelection`](../scripts/lib/qa/resolve-selection.js). It returns
|
|
229
|
-
`{ kind, featureRoot, files, scenarios }` where `scenarios` is the
|
|
230
|
-
`(file, line)`-sorted set the sweep will execute. Determinism is load-bearing:
|
|
231
|
-
re-running the same selector across sweeps scopes the identical set, so the
|
|
232
|
-
evidence stays diffable.
|
|
233
|
-
|
|
234
|
-
Load the `fixturesManifest` to resolve each persona's seed data before
|
|
235
|
-
sign-in. If the selection is empty, report "no scenarios matched
|
|
236
|
-
`<selector>`" and stop — an empty selection is operator error (a typo'd
|
|
237
|
-
feature id or domain), not a passing sweep.
|
|
238
|
-
|
|
239
|
-
### `allowWrites` guardrail (non-local safety)
|
|
240
|
-
|
|
241
|
-
When the resolved environment has **`allowWrites: false`**, exclude any
|
|
242
|
-
scenario judged **mutating** from the selection before driving. Judge a
|
|
243
|
-
scenario mutating from its **`When` steps** — a `When` that creates, updates,
|
|
244
|
-
or deletes persisted state (submits a form that writes, deletes a record,
|
|
245
|
-
changes a setting) is mutating; a read-only navigation/inspection `When` is
|
|
246
|
-
not. Report the **exclusion count** ("N mutating scenarios excluded on
|
|
247
|
-
read-only `<env>`") alongside the resolved scenario count so the operator sees
|
|
248
|
-
what was skipped and why. The exclusion is overridable **only** by an explicit
|
|
249
|
-
in-session operator confirmation (the operator affirms, in this session, that
|
|
250
|
-
writes to `<env>` are acceptable) — never silently include mutating scenarios
|
|
251
|
-
on a read-only target, and never widen `allowWrites` by editing the contract
|
|
252
|
-
mid-sweep.
|
|
253
|
-
|
|
254
|
-
## Step 2 — Sign in via the environment's `signInSeam`
|
|
98
|
+
`list_network_requests`) is **host-provided** — an external runtime dependency,
|
|
99
|
+
not in-repo code. If the host does not expose it, degrade with a clear error
|
|
100
|
+
("the chrome-devtools MCP server is unavailable; the QA harness requires a live
|
|
101
|
+
browser surface") and stop. Never attempt a headless fallback.
|
|
102
|
+
|
|
103
|
+
### Step 1 — Resolve the environment, then the scope
|
|
104
|
+
|
|
105
|
+
**Environment** — resolve which of the contract's `environments` this sweep
|
|
106
|
+
runs against via
|
|
107
|
+
[`resolveQaEnvironment`](../scripts/lib/qa/resolve-qa-contract.js), yielding
|
|
108
|
+
`{ name, baseUrl, signInSeam, allowWrites }` (`allowWrites` defaults to an
|
|
109
|
+
explicit boolean — `true` only for the conventional `local` environment). When
|
|
110
|
+
`<env>` is supplied, pass it straight through (an exact name wins; a raw URL
|
|
111
|
+
matches by origin). When it is omitted (bare `/qa-run`), **prompt** the
|
|
112
|
+
operator, enumerating every environment as `name → baseUrl` (marking
|
|
113
|
+
`defaultEnvironment`); they answer with a name or a raw URL. On an unknown name
|
|
114
|
+
or unmatched URL the resolver throws (naming the known environments) — relay it
|
|
115
|
+
and stop; never silently fall back to the default.
|
|
116
|
+
|
|
117
|
+
**Scope** — when `<selector>` is supplied, parse it into the resolver's
|
|
118
|
+
selector shape. When omitted, **prompt** the operator: enumerate the selectable
|
|
119
|
+
scope under `featureRoot` via `resolve-selection.js` (domains, feature stems,
|
|
120
|
+
canonical tags) as a **multi-select** the operator composes into a selector
|
|
121
|
+
set. Always include a final **"No coverage here → hand off to `/qa-explore`"**
|
|
122
|
+
option: when the surface has no authored `.feature` coverage, choosing it ends
|
|
123
|
+
the sweep and routes to [`/qa-explore`](qa-explore.md) rather than running an
|
|
124
|
+
empty selection.
|
|
125
|
+
|
|
126
|
+
**Scenario set + write guard** — pass the resolved selector(s) and `featureRoot`
|
|
127
|
+
to [`resolveSelection`](../scripts/lib/qa/resolve-selection.js), which returns
|
|
128
|
+
the `(file, line)`-sorted scenario set (determinism is load-bearing: the same
|
|
129
|
+
selector scopes the identical set across sweeps). Load `fixturesManifest` to
|
|
130
|
+
resolve each persona's seed before sign-in. An empty selection is operator
|
|
131
|
+
error (a typo'd id or domain), not a passing sweep — report "no scenarios
|
|
132
|
+
matched `<selector>`" and stop.
|
|
133
|
+
|
|
134
|
+
> **`allowWrites` guardrail (non-local safety).** When the resolved environment
|
|
135
|
+
> has `allowWrites: false`, **exclude any mutating scenario** from the selection
|
|
136
|
+
> before driving — judge a scenario mutating from its `When` steps (a `When`
|
|
137
|
+
> that creates, updates, or deletes persisted state is mutating; a read-only
|
|
138
|
+
> navigation/inspection `When` is not). Report the **exclusion count** ("N
|
|
139
|
+
> mutating scenarios excluded on read-only `<env>`") alongside the resolved
|
|
140
|
+
> count. The exclusion is overridable **only** by an explicit in-session
|
|
141
|
+
> operator confirmation — never silently include mutating scenarios on a
|
|
142
|
+
> read-only target, and never widen `allowWrites` by editing the contract
|
|
143
|
+
> mid-sweep.
|
|
144
|
+
|
|
145
|
+
### Step 2 — Sign in via the environment's `signInSeam`
|
|
255
146
|
|
|
256
147
|
Sign in **once per persona** before driving that persona's scenarios, using the
|
|
257
|
-
resolved environment's discriminated-union seam
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
-
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
- Real auth uses **only `credentialRef`-indirected material** — the persona's
|
|
274
|
-
`credentialRef` names a stored credential the skill dereferences; raw
|
|
275
|
-
passwords, tokens, or API keys are never inlined into the contract, the
|
|
276
|
-
workflow, or chat.
|
|
277
|
-
- **Secrets are never echoed** into chat, findings, or the ledger — do not
|
|
278
|
-
print a credential, a session token, or a cookie value at any point.
|
|
279
|
-
- **Captured evidence passes `redact-evidence.js`** (`redactEvidence`) before
|
|
280
|
-
persistence, so any secret that leaks into console/network capture is
|
|
281
|
-
scrubbed before it reaches a finding (see Step 4 and the per-scenario
|
|
282
|
-
helper).
|
|
283
|
-
|
|
284
|
-
Per-persona auth material (`credentialRef` / `signInSkill`, authored via the
|
|
285
|
-
object-map `personas` shape) is consulted **only** under a `skill` or
|
|
286
|
-
credential seam. Under a `urlTemplate` dev-impersonation seam the persona name
|
|
287
|
-
is the only input, so the material is never read — author name-only personas
|
|
288
|
-
there rather than fabricating `credentialRef`/`signInSkill` values the harness
|
|
289
|
-
ignores. The resolver normalizes both authored shapes to one canonical object
|
|
290
|
-
map keyed by persona name; a name-only persona resolves to an empty record.
|
|
291
|
-
|
|
292
|
-
After sign-in, confirm the authenticated state with a `take_snapshot`
|
|
293
|
-
(e.g. the user menu or persona badge is present) before driving any scenario.
|
|
294
|
-
This confirmed authenticated session is the precondition the per-scenario
|
|
295
|
-
helper's input contract requires (Step 3).
|
|
148
|
+
resolved environment's discriminated-union seam (anchored on `baseUrl`):
|
|
149
|
+
|
|
150
|
+
- **`kind: 'url'`** — substitute the persona **name** into `template` (e.g.
|
|
151
|
+
`/dev/sign-in-as/{persona}`) and `navigate_page` there. The name is the sole
|
|
152
|
+
input; under a `urlTemplate` seam the contract is authored as a plain name
|
|
153
|
+
array and no per-persona auth material is read.
|
|
154
|
+
- **`kind: 'skill'`** — invoke the named consumer sign-in skill (procedural /
|
|
155
|
+
non-URL sign-in). Real auth uses **only `credentialRef`-indirected material**
|
|
156
|
+
the skill dereferences; raw passwords, tokens, or API keys are never inlined
|
|
157
|
+
into the contract, the workflow, or chat, and captured evidence is redacted
|
|
158
|
+
per [`helpers/qa-core.md`](helpers/qa-core.md) before persistence.
|
|
159
|
+
|
|
160
|
+
**Verification (the envelope's proof).** After sign-in, confirm the
|
|
161
|
+
authenticated state with a `take_snapshot` showing the persona badge (the user
|
|
162
|
+
menu / persona badge is present) before driving any scenario. This confirmed
|
|
163
|
+
session is the precondition the per-scenario helper re-verifies on entry.
|
|
296
164
|
|
|
297
165
|
## Step 3 — Drive each scenario via the per-scenario helper
|
|
298
166
|
|
|
299
167
|
For each scenario in selection order, delegate driving, analysis, and reporting
|
|
300
|
-
to [`helpers/qa-run-scenario.md`](helpers/qa-run-scenario.md)
|
|
301
|
-
|
|
168
|
+
to [`helpers/qa-run-scenario.md`](helpers/qa-run-scenario.md) — the **one prose
|
|
169
|
+
home** for the driving rules (navigation-first / never URL-jump, semantic
|
|
170
|
+
`Then` assertion, the per-`When` write guard, mandatory evidence redaction, and
|
|
171
|
+
the sequential-only browser rule). Do not restate those rules here. Pass the
|
|
172
|
+
helper its input contract:
|
|
302
173
|
|
|
303
|
-
- **`environment`** — the resolved `{ name, baseUrl, allowWrites }`
|
|
304
|
-
Step 0.5a.
|
|
174
|
+
- **`environment`** — the resolved `{ name, baseUrl, allowWrites }` (Step 1).
|
|
305
175
|
- **`persona`** — the persona name **plus** the confirmed authenticated-session
|
|
306
|
-
precondition
|
|
307
|
-
- **`scenario`** — the scenario ref (`.feature` file path and `(file, line)`
|
|
308
|
-
locator).
|
|
176
|
+
precondition from Step 2 (the helper re-verifies it on entry).
|
|
177
|
+
- **`scenario`** — the scenario ref (`.feature` file path and `(file, line)`).
|
|
309
178
|
- **`consoleAllowlist`** and **`designTokens`** — from the resolved contract.
|
|
310
179
|
|
|
311
|
-
The helper
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
structured per-scenario result** — `{ scenario, intent, verdict (pass | fail |
|
|
315
|
-
blocked), surface, findings[] }`. Collect one result per scenario; the sweep
|
|
316
|
-
report shape (per-scenario `intent + verdict` lines plus totals, Step 6) is
|
|
317
|
-
unchanged from the inlined procedure.
|
|
180
|
+
The helper returns **one structured per-scenario result** —
|
|
181
|
+
`{ scenario, intent, verdict (pass | fail | blocked), surface, findings[] }`.
|
|
182
|
+
Collect one result per scenario for the sweep report (Step 5).
|
|
318
183
|
|
|
319
|
-
## Step 4 —
|
|
184
|
+
## Step 4 — Record findings onto the ledger
|
|
320
185
|
|
|
321
186
|
The per-scenario helper captures console and network per surface and turns
|
|
322
|
-
genuine problems into structured findings, applying the contract's
|
|
187
|
+
genuine problems into structured `F#` findings, applying the contract's
|
|
323
188
|
`consoleAllowlist` via
|
|
324
189
|
[`filterConsoleMessages`](../scripts/lib/qa/console-allowlist.js) (each
|
|
325
|
-
non-allowlisted console **error** becomes one
|
|
326
|
-
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
`
|
|
333
|
-
|
|
334
|
-
|
|
335
|
-
|
|
336
|
-
|
|
337
|
-
|
|
338
|
-
|
|
339
|
-
|
|
340
|
-
|
|
341
|
-
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
|
|
353
|
-
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
|
|
358
|
-
|
|
359
|
-
|
|
360
|
-
item's `class` resolves to the focus/meta label set Triage applies when
|
|
361
|
-
promoting it. The helper **throws** on an absent/unknown class — fix the
|
|
362
|
-
ledger item's class rather than defaulting.
|
|
363
|
-
2. **Dedup / route** it against existing GitHub Issues via
|
|
364
|
-
[`route-finding.js`](../scripts/lib/findings/route-finding.js):
|
|
365
|
-
|
|
366
|
-
```js
|
|
367
|
-
import { routeFinding, fingerprintFooter } from '../scripts/lib/findings/route-finding.js';
|
|
368
|
-
const { decision, matchedIssue, fingerprint } =
|
|
369
|
-
await routeFinding(finding, { searchIssues });
|
|
370
|
-
```
|
|
371
|
-
|
|
372
|
-
`decision` is one of `new` / `update-existing` / `duplicate` /
|
|
373
|
-
`regression-of-closed`. Wire the `searchIssues` port to the GitHub provider,
|
|
374
|
-
querying **both open and closed** Issues, and stamp the
|
|
375
|
-
`fingerprintFooter(sha)` marker into any Issue body so future sweeps dedup
|
|
376
|
-
against it. This is the **single** dedup implementation shared with
|
|
377
|
-
`/qa-explore`, `/qa-assist`, and `audit-to-stories`.
|
|
378
|
-
3. **Decide the disposition** with the operator (`file` / `defer` / `dismiss`)
|
|
379
|
-
and record it back onto the ledger item.
|
|
380
|
-
4. **Promote the `file`-dispositioned findings through `/plan`** via
|
|
381
|
-
[`promote-finding.js`](../scripts/lib/findings/promote-finding.js) — the
|
|
382
|
-
same cluster/size/route/file path `/qa-explore` and `/audit-to-stories`
|
|
383
|
-
consume (`clusterLedgerItems` + `targetForCluster`: a cluster spanning ≤2
|
|
384
|
-
coverage surfaces routes to `createStory` via `/plan --seed-file`, >2 to
|
|
385
|
-
`createPlanSeed` via `/plan --seed`, with the cluster's `fingerprintFooter(sha)`
|
|
386
|
-
stamped verbatim into the seed). A `file` disposition never opens a raw
|
|
387
|
-
GitHub Issue; only `defer` and `dismiss` skip the `/plan` handoff.
|
|
388
|
-
5. **Gate:** any ticket-filing, seed write, `/plan` invocation, or label
|
|
389
|
-
mutation is a write — confirm each with the operator before it happens.
|
|
390
|
-
|
|
391
|
-
## Step 6 — Report
|
|
392
|
-
|
|
393
|
-
Summarize the sweep in chat with:
|
|
394
|
-
|
|
395
|
-
- The resolved **environment** (`name → baseUrl`, and whether it is read-only)
|
|
396
|
-
and the selector applied.
|
|
397
|
-
- The resolved scenario count, plus the **`allowWrites` exclusion count** when
|
|
398
|
-
mutating scenarios were skipped on a read-only environment.
|
|
399
|
-
- Scenario totals: passed / failed / blocked.
|
|
400
|
-
- Findings totals by classification, and the ledger routes
|
|
401
|
-
(`new` / `update-existing` / `duplicate` / `regression-of-closed`).
|
|
402
|
-
- A per-scenario line pairing each scenario's plain-English intent with its
|
|
403
|
-
verdict (pass / fail / blocked), grouped by feature file or domain — so the
|
|
404
|
-
digest reads as "what was checked → what happened", not a tag list.
|
|
405
|
-
- For each failure, the scenario name, file path, the surface it ended on, and
|
|
406
|
-
a one-line user-visible symptom.
|
|
407
|
-
- The ledger path (under `temp/qa/`) and a pointer to the routed dispositions
|
|
408
|
-
awaiting operator sign-off (if any).
|
|
190
|
+
non-allowlisted console **error** becomes one finding; allowlisted patterns and
|
|
191
|
+
non-error levels are suppressed) and spot-checking against `designTokens` when
|
|
192
|
+
set. The allowlist is a **noise filter, not a security control** — never expand
|
|
193
|
+
it to silence a genuine error signal.
|
|
194
|
+
|
|
195
|
+
Record each returned `F#` finding as a `QaLedgerItem` on the session ledger
|
|
196
|
+
(shape and append-never-overwrite rule per [`helpers/qa-core.md`](helpers/qa-core.md)):
|
|
197
|
+
map the finding's `symptom` to the redacted `evidence`, its `surface` to
|
|
198
|
+
`coverage`, its `classification` onto the ledger `class` enum (a product defect
|
|
199
|
+
is `product-bug`, a tooling gap is `tooling-dx`, …) plus a `severity`,
|
|
200
|
+
`missingTest` (`null` when no gap applies), and `disposition` left untriaged.
|
|
201
|
+
|
|
202
|
+
## Step 5 — Triage the ledger, then report
|
|
203
|
+
|
|
204
|
+
Route the ledger through the shared classify → route → disposition → promote
|
|
205
|
+
procedure in [`helpers/qa-core.md`](helpers/qa-core.md), under its HITL write
|
|
206
|
+
gate: the harness MUST NOT create tickets autonomously — present the routed
|
|
207
|
+
dispositions and confirm each `file` / `defer` / `dismiss` before any write.
|
|
208
|
+
|
|
209
|
+
Then summarize the sweep in chat with:
|
|
210
|
+
|
|
211
|
+
- the resolved **environment** (`name → baseUrl`, and whether it is read-only)
|
|
212
|
+
and the selector applied;
|
|
213
|
+
- the resolved scenario count, plus the **`allowWrites` exclusion count** when
|
|
214
|
+
mutating scenarios were skipped on a read-only environment;
|
|
215
|
+
- scenario totals (passed / failed / blocked);
|
|
216
|
+
- findings totals by classification and the ledger routes
|
|
217
|
+
(`new` / `update-existing` / `duplicate` / `regression-of-closed`);
|
|
218
|
+
- a per-scenario line pairing each scenario's plain-English intent with its
|
|
219
|
+
verdict, grouped by feature file or domain — "what was checked → what
|
|
220
|
+
happened", not a tag list;
|
|
221
|
+
- for each failure: the scenario name, file path, the surface it ended on, and
|
|
222
|
+
a one-line user-visible symptom;
|
|
223
|
+
- the ledger path (under `temp/qa/`) and a pointer to any routed dispositions
|
|
224
|
+
awaiting operator sign-off.
|
|
409
225
|
|
|
410
226
|
## Constraints
|
|
411
227
|
|
|
412
|
-
|
|
413
|
-
|
|
414
|
-
|
|
415
|
-
|
|
416
|
-
|
|
228
|
+
Beyond the shared core ([`helpers/qa-core.md`](helpers/qa-core.md): contract +
|
|
229
|
+
loud failure, session/ledger, redact-first, QaLedgerItem, triage, HITL gate)
|
|
230
|
+
and the driving rules ([`helpers/qa-run-scenario.md`](helpers/qa-run-scenario.md):
|
|
231
|
+
navigation-first, semantic `Then`, redaction, sequential-only), the
|
|
232
|
+
`/qa-run`-specific deltas are:
|
|
233
|
+
|
|
234
|
+
- **Always** resolve a target environment (Step 1) — prompt when no `<env>` is
|
|
235
|
+
supplied — and **fail loudly** on an unknown name or unmatched URL; never
|
|
236
|
+
silently fall back to the default.
|
|
417
237
|
- **Always** apply the `allowWrites` guardrail: on a read-only environment,
|
|
418
238
|
exclude mutating scenarios and report the exclusion count; include them only
|
|
419
239
|
on explicit in-session operator confirmation.
|
|
420
|
-
- **Always**
|
|
421
|
-
|
|
422
|
-
|
|
423
|
-
|
|
424
|
-
|
|
425
|
-
|
|
426
|
-
- **
|
|
427
|
-
|
|
428
|
-
captured evidence passes `redact-evidence.js` before persistence.
|
|
429
|
-
- **Always** record findings as `QaLedgerItem`s on the shared session ledger
|
|
430
|
-
under `temp/qa/` ([`qa-ledger.schema.json`](../schemas/qa-ledger.schema.json))
|
|
431
|
-
and route them through the shared classify/route/dedup/promote core — there
|
|
432
|
-
is no separate `/qa-run` finding schema or draft-bundle path. Never commit
|
|
433
|
-
the ledger.
|
|
434
|
-
- **Delegate Triage decisions to the helpers.** Classification
|
|
435
|
-
([`classify-finding.js`](../scripts/lib/findings/classify-finding.js)),
|
|
436
|
-
dedup/route ([`route-finding.js`](../scripts/lib/findings/route-finding.js),
|
|
437
|
-
fingerprint-footer against open + closed issues), and cluster/size/promote
|
|
438
|
-
([`promote-finding.js`](../scripts/lib/findings/promote-finding.js)) are
|
|
439
|
-
deterministic — never re-derive them in prose.
|
|
440
|
-
- **Never** file follow-up tickets autonomously; promote `file` findings
|
|
441
|
-
through `/plan` only after operator sign-off, and never open a raw GitHub
|
|
442
|
-
Issue for a `file` finding.
|
|
443
|
-
- **Never** expand `consoleAllowlist` to suppress genuine error signal — it is
|
|
444
|
-
a benign-noise filter, not a security control.
|
|
445
|
-
- **Always** scrub captured evidence of secrets and PII before rendering a
|
|
446
|
-
finding.
|
|
447
|
-
- **Never** fall back to a retired headless BDD-runner workflow.
|
|
240
|
+
- **Always** sign in per persona through the environment's `signInSeam` and
|
|
241
|
+
confirm the authenticated state with a post-sign-in `take_snapshot` before
|
|
242
|
+
driving.
|
|
243
|
+
- An **empty selection is operator error**, not a passing sweep — report it and
|
|
244
|
+
stop.
|
|
245
|
+
- **Never** expand `consoleAllowlist` to suppress genuine error signal.
|
|
246
|
+
- **Never** file follow-up tickets autonomously, and **never** fall back to a
|
|
247
|
+
retired headless BDD runner.
|