mandrel 2.39.0 → 2.41.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/README.md +8 -5
- package/.agents/agents/auditor.md +5 -0
- package/.agents/docs/SDLC.md +21 -12
- package/.agents/instructions.md +22 -22
- package/.agents/rules/api-conventions.md +43 -7
- package/.agents/rules/ci-remediation.md +3 -14
- package/.agents/rules/gherkin-standards.md +21 -6
- package/.agents/rules/git-conventions.md +6 -5
- package/.agents/rules/security-baseline.md +6 -7
- package/.agents/rules/testing-standards.md +75 -198
- package/.agents/scripts/audit-to-stories.js +510 -66
- package/.agents/scripts/install-matrix-assert.js +2 -2
- package/.agents/scripts/lib/audit-to-stories/epic-grouping-directive.js +45 -0
- package/.agents/scripts/lib/audit-to-stories/ledger-commit.js +290 -0
- package/.agents/scripts/lib/audit-to-stories/parse-audit-md.js +94 -3
- package/.agents/scripts/lib/audit-to-stories/seed-from-findings.js +10 -0
- package/.agents/scripts/lib/label-constants.js +18 -0
- package/.agents/scripts/lib/label-taxonomy.js +18 -5
- package/.agents/scripts/lib/orchestration/dependency-candidates.js +139 -0
- package/.agents/scripts/lib/orchestration/epic-candidates.js +159 -0
- package/.agents/scripts/lib/orchestration/epic-checklist.js +103 -0
- package/.agents/scripts/lib/orchestration/epic-container.js +202 -0
- package/.agents/scripts/lib/orchestration/epic-expansion.js +148 -0
- package/.agents/scripts/lib/orchestration/plan-context.js +97 -36
- package/.agents/scripts/lib/orchestration/plan-persist/cross-plan-links.js +80 -0
- package/.agents/scripts/lib/orchestration/plan-persist/epic-adoption.js +192 -0
- package/.agents/scripts/lib/orchestration/plan-persist/epic-ops.js +324 -0
- package/.agents/scripts/lib/orchestration/plan-persist/external-deps.js +164 -0
- package/.agents/scripts/lib/orchestration/plan-persist/run-plan-persist.js +30 -0
- package/.agents/scripts/lib/orchestration/plan-persist/story-ops.js +26 -5
- package/.agents/scripts/lib/orchestration/run-epilogue.js +130 -1
- package/.agents/scripts/lib/orchestration/ticket-validator.js +11 -1
- package/.agents/scripts/plan-persist.js +98 -1
- package/.agents/scripts/providers/github/sub-issue-add.js +218 -0
- package/.agents/scripts/resolve-stories.js +42 -2
- package/.agents/skills/core/browser-testing-with-devtools/SKILL.md +5 -10
- package/.agents/skills/core/browser-testing-with-devtools/reference.md +7 -46
- package/.agents/skills/core/code-review-and-quality/SKILL.md +0 -5
- package/.agents/skills/core/documentation-and-adrs/SKILL.md +0 -3
- package/.agents/skills/core/gates-and-baselines/SKILL.md +10 -137
- package/.agents/skills/core/gates-and-baselines/reference.md +103 -0
- package/.agents/skills/core/idea-refinement/SKILL.md +2 -18
- package/.agents/skills/core/security-and-hardening/SKILL.md +2 -4
- package/.agents/skills/core/security-and-hardening/reference.md +0 -70
- package/.agents/skills/skills.index.json +10 -70
- package/.agents/skills/stack/qa/gherkin-authoring/SKILL.md +4 -10
- package/.agents/skills/stack/qa/gherkin-authoring/reference.md +9 -113
- package/.agents/skills/stack/qa/playwright-bdd/SKILL.md +29 -154
- package/.agents/skills/stack/qa/qa-harness/SKILL.md +157 -98
- package/.agents/templates/docs/audit-sweep-runbook.md +169 -0
- package/.agents/workflows/audit-to-stories.md +85 -7
- package/.agents/workflows/git-cleanup.md +3 -2
- package/.agents/workflows/git-deliver.md +3 -2
- package/.agents/workflows/helpers/audit-lens-core.md +24 -4
- package/.agents/workflows/helpers/deliver-reference.md +8 -0
- package/.agents/workflows/helpers/plan-reference.md +109 -1
- package/.agents/workflows/mandrel-deliver.md +47 -43
- package/.agents/workflows/mandrel-plan.md +61 -55
- package/.agents/workflows/qa-assist.md +20 -17
- package/.agents/workflows/qa-explore.md +30 -29
- package/.agents/workflows/qa-run.md +2 -1
- package/docs/CHANGELOG.md +30 -0
- package/package.json +1 -1
- package/.agents/rules/changelog-style.md +0 -180
- package/.agents/rules/shell-conventions.md +0 -61
- package/.agents/scripts/lib/qa/coverage-verdict.js +0 -214
- package/.agents/skills/core/api-and-interface-design/SKILL.md +0 -55
- package/.agents/skills/core/api-and-interface-design/reference.md +0 -76
- package/.agents/skills/core/debugging-and-error-recovery/SKILL.md +0 -45
- package/.agents/skills/core/debugging-and-error-recovery/reference.md +0 -56
- package/.agents/skills/core/git-workflow-and-versioning/SKILL.md +0 -54
- package/.agents/skills/core/idea-refinement/refinement-criteria.md +0 -155
- package/.agents/skills/core/idea-refinement/scripts/idea-refine.sh +0 -15
- package/.agents/skills/core/qa-coverage-mapping/SKILL.md +0 -105
- package/.agents/skills/stack/qa/qa-explore-driving/SKILL.md +0 -152
- package/.agents/skills/stack/qa/vitest/SKILL.md +0 -22
|
@@ -1,13 +1,15 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: qa-harness
|
|
3
3
|
description:
|
|
4
|
-
Conventions for
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
4
|
+
Conventions for agent-driven QA driving — the one driving-conventions
|
|
5
|
+
reference behind both `/qa-run` (the known-scenario sweep through a real
|
|
6
|
+
browser) and `/qa-explore` (agent-led exploratory driving). Use when
|
|
7
|
+
instrumenting a live surface — covers navigation-first execution,
|
|
8
|
+
per-surface console and network capture, design-token visual checks, the
|
|
9
|
+
`F#` finding shape, per-environment resolution, and static driving as the
|
|
10
|
+
documented interim. The run procedures live in `.agents/workflows/qa-run.md`
|
|
11
|
+
and `.agents/workflows/qa-explore.md`; this skill is the conventions
|
|
12
|
+
reference they lean on.
|
|
11
13
|
---
|
|
12
14
|
|
|
13
15
|
# Skill: qa-harness
|
|
@@ -15,47 +17,110 @@ description:
|
|
|
15
17
|
## Policy Capsule
|
|
16
18
|
|
|
17
19
|
- Driving rules (navigation-first, semantic `Then`, mandatory redaction, sequential-only) live in one prose home — [`qa-run-scenario.md`](../../../../workflows/helpers/qa-run-scenario.md); apply them, do not restate them.
|
|
20
|
+
- Resolve the `qa` contract first and fail loudly when it is absent or malformed; there is no auto-detection fallback and no headless degrade.
|
|
21
|
+
- Resolve the target **environment** before driving (via `resolveQaEnvironment`); each environment keys its own `baseUrl`, `signInSeam`, and `allowWrites`, and its name is recorded on the ledger.
|
|
22
|
+
- Sign in only through the resolved `signInSeam` (`kind: 'url'` dev impersonation or `kind: 'skill'` with a stored `credentialRef`). **Never** type real credentials inline and never fabricate a session.
|
|
23
|
+
- Hold the read-only capture invariant absolutely: no source edits, no product mutations — the only write is appending to the `temp/qa/<sessionId>` ledger.
|
|
24
|
+
- Scrub captured console, network, and headers of tokens, session cookies, and PII before any finding reaches disk or GitHub; findings are posted to GitHub at approval time.
|
|
18
25
|
- Capture console and network per surface; turn each non-allowlisted console error and each failed/error-status request into one structured `F#` finding.
|
|
19
26
|
- Filter console through `qa.consoleAllowlist` via `filterConsoleMessages`; treat the allowlist as a benign-noise filter, never as a security control to silence genuine errors.
|
|
20
27
|
- Spot-check surfaces against `qa.designTokens` when set; flag gross token violations (off-palette colors, off-scale spacing/typography) as findings.
|
|
21
|
-
-
|
|
22
|
-
- Record findings as `QaLedgerItem`s and route them through the shared classify/route/promote core ([`qa-core.md`](../../../../workflows/helpers/qa-core.md)); the harness never files tickets autonomously.
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
[
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
28
|
+
- Choose **static driving** only at Plan time and only where no seam resolves — the documented interim, never a silent fallback — and record the partial coverage.
|
|
29
|
+
- Record findings as `QaLedgerItem`s and route them through the shared classify/route/promote core ([`qa-core.md`](../../../../workflows/helpers/qa-core.md)); the harness never files tickets autonomously, and every phase transition is HITL-gated.
|
|
30
|
+
|
|
31
|
+
Guidance for driving a live surface through a real browser (the chrome-devtools
|
|
32
|
+
MCP surface). Two workflows lean on this one skill: the **known-scenario
|
|
33
|
+
sweep** ([`qa-run.md`](../../../../workflows/qa-run.md)) walks a resolved
|
|
34
|
+
Gherkin scenario set, and **exploratory driving**
|
|
35
|
+
([`qa-explore.md`](../../../../workflows/qa-explore.md)) walks a named surface
|
|
36
|
+
the agent has no script for. Each **procedure** — argument parsing, phase
|
|
37
|
+
gates, contract resolution sequence — is the SSOT in its own workflow; this
|
|
38
|
+
skill shows **how** to apply the driving and instrumentation conventions both
|
|
39
|
+
depend on. Assertion-tier rules live in
|
|
40
|
+
[`testing-standards.md`](../../../../rules/testing-standards.md)
|
|
32
41
|
(§ Assertion Placement); scenario prose conventions live in
|
|
33
42
|
[`gherkin-authoring`](../gherkin-authoring/SKILL.md); browser-locator
|
|
34
|
-
discipline is shared with [`playwright`](../playwright/SKILL.md)
|
|
35
|
-
|
|
43
|
+
discipline is shared with [`playwright`](../playwright/SKILL.md); browser
|
|
44
|
+
instrumentation lives in
|
|
45
|
+
[`browser-testing-with-devtools`](../../../core/browser-testing-with-devtools/SKILL.md).
|
|
46
|
+
Read this skill before instrumenting a live surface; read the workflow for the
|
|
47
|
+
run order.
|
|
36
48
|
|
|
37
49
|
## 1. Driving Rules Live in One Prose Home
|
|
38
50
|
|
|
39
|
-
The driving rules
|
|
40
|
-
**semantic `Then
|
|
41
|
-
browser-action mapping, the per-`When` write guard, mandatory
|
|
42
|
-
redaction, and the **sequential-only** browser rule — are stated once
|
|
43
|
-
[`qa-run-scenario.md`](../../../../workflows/helpers/qa-run-scenario.md)
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
+
The driving rules both modes depend on — **navigation-first / never
|
|
52
|
+
URL-jump**, **semantic `Then`** assertion against the accessibility snapshot,
|
|
53
|
+
the Gherkin → browser-action mapping, the per-`When` write guard, mandatory
|
|
54
|
+
evidence redaction, and the **sequential-only** browser rule — are stated once
|
|
55
|
+
in [`qa-run-scenario.md`](../../../../workflows/helpers/qa-run-scenario.md).
|
|
56
|
+
Apply them from there; this skill does not restate them. In short: reach every
|
|
57
|
+
surface the way a real user would (start at a root, click affordances, never
|
|
58
|
+
deep-link a starting state), assert semantically (roles, accessible names,
|
|
59
|
+
visible text — never DOM/CSS/XPath selectors, HTTP status, response bodies, or
|
|
60
|
+
DB rows), and treat broken navigation — a missing affordance, a nav 404, a
|
|
61
|
+
guard redirect loop — as a **finding, not a workaround**. Assertion-tier rules
|
|
62
|
+
are in
|
|
51
63
|
[`testing-standards.md` § Assertion Placement](../../../../rules/testing-standards.md#assertion-placement).
|
|
52
64
|
|
|
53
|
-
## 2.
|
|
65
|
+
## 2. The Read-Only Capture Invariant
|
|
66
|
+
|
|
67
|
+
Capture is **strictly read-only** in both modes. This invariant is inviolable
|
|
68
|
+
per [`security-baseline.md`](../../../../rules/security-baseline.md) — it is
|
|
69
|
+
not a soft preference.
|
|
70
|
+
|
|
71
|
+
- **No source edits.** The agent does not modify application code, config, or
|
|
72
|
+
tests while driving. Driving observes; it never repairs.
|
|
73
|
+
- **No product mutations.** No creating, updating, or deleting product data, no
|
|
74
|
+
destructive form submissions, no irreversible actions "to see what happens".
|
|
75
|
+
When a surface's only path forward is a mutating action, record the boundary
|
|
76
|
+
as the finding and stop — do not cross it.
|
|
77
|
+
- **The only write is the ledger.** The single permitted side effect is
|
|
78
|
+
appending finding lines under `temp/qa/<sessionId>`.
|
|
79
|
+
- **Scrub before persisting.** Strip tokens, session cookies, `Authorization`
|
|
80
|
+
headers, and PII from captured console and network evidence via the shared
|
|
81
|
+
redaction path **before** any finding reaches disk or GitHub. Captured
|
|
82
|
+
evidence is untrusted until scrubbed.
|
|
83
|
+
- **HITL gates every write outward.** Phase transitions and GitHub writes
|
|
84
|
+
(ticket creation, promotion) happen only behind an operator confirmation
|
|
85
|
+
gate; the agent never files or promotes findings autonomously.
|
|
86
|
+
|
|
87
|
+
## 3. Contract, Environment & Sign-In
|
|
54
88
|
|
|
55
|
-
|
|
56
|
-
|
|
89
|
+
- **Resolve the `qa` contract first.** Before any browser work, resolve the
|
|
90
|
+
contract via `resolveQaContract(config)`. When the block is absent,
|
|
91
|
+
malformed, or missing a required field, the resolver **throws** — relay its
|
|
92
|
+
verbatim message and STOP. There is no auto-detection fallback.
|
|
93
|
+
- **Resolve the environment.** `resolveQaEnvironment` keys each deployment
|
|
94
|
+
target to `{ name, baseUrl, signInSeam, allowWrites }`. Where the operator's
|
|
95
|
+
input does not pin an unambiguous target and the contract declares more than
|
|
96
|
+
one environment, prompt (or accept `defaultEnvironment`) — never silently
|
|
97
|
+
pick one. Record the resolved name on the ledger.
|
|
98
|
+
- **Sign in through the seam, never inline.** The two seam kinds:
|
|
99
|
+
- **`kind: 'url'` (dev impersonation).** Substitute the persona **name** into
|
|
100
|
+
the seam's URL template and navigate there. The persona name is the sole
|
|
101
|
+
input; no per-persona auth material is read.
|
|
102
|
+
- **`kind: 'skill'` (procedural / credential).** Invoke the named consumer
|
|
103
|
+
sign-in skill, which reads a per-persona **`credentialRef`** — an indirect
|
|
104
|
+
handle to a stored credential, never an inline secret. Read that skill's
|
|
105
|
+
`SKILL.md` and follow it.
|
|
106
|
+
|
|
107
|
+
The agent MUST NOT type real usernames, passwords, or tokens, and MUST NOT
|
|
108
|
+
fabricate or forge a session. This is a hard security boundary, not a
|
|
109
|
+
convenience to work around. Confirm authenticated state with a
|
|
110
|
+
`take_snapshot` before driving.
|
|
111
|
+
- **No headless fallback.** The chrome-devtools MCP surface is a host-provided
|
|
112
|
+
runtime dependency. If it is unavailable, degrade with a clear error and stop
|
|
113
|
+
— never fall back to the retired headless BDD runner.
|
|
57
114
|
|
|
58
|
-
|
|
115
|
+
## 4. Mode — Known-Scenario Sweep (`/qa-run`)
|
|
116
|
+
|
|
117
|
+
The sweep drives a resolved scenario set and instruments each surface the
|
|
118
|
+
moment it lands on it, before moving on. Capture is **per surface** so evidence
|
|
119
|
+
is attributable to a concrete user-reachable state. Record each scenario's
|
|
120
|
+
result (pass / fail / blocked), the surface it ended on, and a one-line symptom
|
|
121
|
+
for any failure.
|
|
122
|
+
|
|
123
|
+
### 4.1 Console
|
|
59
124
|
|
|
60
125
|
1. Capture with `list_console_messages` on the current surface.
|
|
61
126
|
2. Filter through the contract's `consoleAllowlist` using
|
|
@@ -75,13 +140,13 @@ Allowlist matching is case-sensitive substring matching, so patterns stay
|
|
|
75
140
|
readable in `.agentrc.json` without regex escaping; a blank pattern is ignored
|
|
76
141
|
rather than matching everything.
|
|
77
142
|
|
|
78
|
-
###
|
|
143
|
+
### 4.2 Network
|
|
79
144
|
|
|
80
145
|
Capture with `list_network_requests` on the surface. Failed requests and
|
|
81
146
|
error-status responses (4xx / 5xx) become findings alongside the
|
|
82
147
|
console-derived set, sharing the same `F#` numbering across the surface.
|
|
83
148
|
|
|
84
|
-
###
|
|
149
|
+
### 4.3 Design-token visual check
|
|
85
150
|
|
|
86
151
|
When the contract's `designTokens` pointer is set (it defaults to `null`),
|
|
87
152
|
spot-check the rendered surface against the token source. Flag **gross** token
|
|
@@ -98,10 +163,10 @@ A gross violation is one a designer would call a regression on sight; subtle
|
|
|
98
163
|
sub-pixel differences are not harness findings. When `designTokens` is `null`,
|
|
99
164
|
skip this check entirely — do not invent a token source.
|
|
100
165
|
|
|
101
|
-
|
|
166
|
+
### 4.4 The `F#` finding shape
|
|
102
167
|
|
|
103
|
-
Every captured problem is normalized into the structured `F#`
|
|
104
|
-
|
|
168
|
+
Every captured problem is normalized into the structured `F#` shape so the
|
|
169
|
+
sweep can record it onto the shared ledger (each `F#` finding becomes one
|
|
105
170
|
`QaLedgerItem` — see [`qa-core.md`](../../../../workflows/helpers/qa-core.md))
|
|
106
171
|
and the schema validates:
|
|
107
172
|
|
|
@@ -111,7 +176,7 @@ and the schema validates:
|
|
|
111
176
|
"classification": "console-error", // console-error | network-error | visual-token | ...
|
|
112
177
|
"surface": "/invoices", // the user-reachable surface, not a deep link
|
|
113
178
|
"symptom": "...", // one-line user-visible / captured symptom
|
|
114
|
-
"likelyRootCause": null, //
|
|
179
|
+
"likelyRootCause": null, // null unless the cause is evidenced, not guessed
|
|
115
180
|
"disposition": "follow-up", // blocker | follow-up
|
|
116
181
|
"acceptance": null, // AC this folds into, when known
|
|
117
182
|
"evidence": {
|
|
@@ -124,77 +189,71 @@ and the schema validates:
|
|
|
124
189
|
- **Determinism is load-bearing.** Re-running the same selector over the same
|
|
125
190
|
captured console with the same allowlist yields the same findings in the same
|
|
126
191
|
order. Do not reorder or renumber findings between sweeps.
|
|
127
|
-
- **Scrub before rendering.** Before any finding's `evidence` is rendered or
|
|
128
|
-
drafted, strip tokens, session cookies, Authorization headers, and PII from
|
|
129
|
-
the captured console and network per
|
|
130
|
-
[`security-baseline.md`](../../../../rules/security-baseline.md). Findings are
|
|
131
|
-
posted to GitHub at approval time — captured evidence is untrusted until
|
|
132
|
-
scrubbed.
|
|
133
|
-
|
|
134
|
-
## 4. Framework-Generic Heuristic Cards
|
|
135
|
-
|
|
136
|
-
The harness ships **framework-generic** root-cause heuristics — they reason
|
|
137
|
-
about symptoms, not about any one frontend framework. Use a card to populate
|
|
138
|
-
`likelyRootCause` and to set `disposition`. The cards are guidance, not a
|
|
139
|
-
classifier: when a symptom matches none cleanly, leave `likelyRootCause: null`
|
|
140
|
-
and let the operator triage from the symptom.
|
|
141
|
-
|
|
142
|
-
| Symptom pattern | Likely root cause | Default disposition |
|
|
143
|
-
| --- | --- | --- |
|
|
144
|
-
| `404` / `Not Found` on a navigation or asset request | Dead route, broken link, or missing build artifact | follow-up (blocker if it breaks the scenario path) |
|
|
145
|
-
| `401` / `403` reaching a surface the persona should see | Missing or over-tight authorization check; guard misconfig | blocker |
|
|
146
|
-
| `500` / `502` / `503` on a user action | Server-side fault behind the action | blocker |
|
|
147
|
-
| Uncaught `TypeError` / `ReferenceError` in console | Null/undefined dereference or missing binding in client code | blocker when it breaks the surface, else follow-up |
|
|
148
|
-
| `Failed to fetch` / `NetworkError` / CORS-rejected request | Misconfigured CORS allowlist, wrong origin, or a downed dependency | follow-up |
|
|
149
|
-
| Hydration / mismatch warning escalated to error | Server/client render divergence | follow-up |
|
|
150
|
-
| Off-palette color, off-scale spacing/typography | Design-token drift — hard-coded value bypassing the token | follow-up |
|
|
151
|
-
| Repeated identical console error across many surfaces | A shared component or global bootstrap fault | record once; the shared route/dedup core collapses duplicates at triage |
|
|
152
|
-
|
|
153
|
-
Heuristics for working the cards:
|
|
154
|
-
|
|
155
|
-
- **Record once, let dedup collapse.** When the same error fires on many
|
|
156
|
-
surfaces, record it once rather than filing N copies; the shared
|
|
157
|
-
classify/route/dedup core ([`qa-core.md`](../../../../workflows/helpers/qa-core.md))
|
|
158
|
-
collapses duplicates at triage against the fingerprint footer.
|
|
159
192
|
- **Blocker vs. follow-up.** A finding is a **blocker** when it breaks the
|
|
160
193
|
scenario's user-visible outcome or exposes an authorization gap. Everything
|
|
161
|
-
else
|
|
194
|
+
else — noise that does not break the journey, cosmetic token drift — is a
|
|
162
195
|
**follow-up**.
|
|
163
196
|
- **Symptom over diagnosis.** When unsure of the root cause, record the precise
|
|
164
197
|
symptom and leave `likelyRootCause: null`. A wrong guess is worse than an
|
|
165
198
|
honest "unknown" the operator can triage.
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
199
|
+
- **Record once, let dedup collapse.** When the same error fires on many
|
|
200
|
+
surfaces, record it once rather than filing N copies; the shared
|
|
201
|
+
classify/route/dedup core collapses duplicates at triage against the
|
|
202
|
+
fingerprint footer.
|
|
203
|
+
|
|
204
|
+
## 5. Mode — Exploratory Driving (`/qa-explore`)
|
|
205
|
+
|
|
206
|
+
Exploratory driving walks a named surface with no scenario script, capturing
|
|
207
|
+
what it observes. It is the agent-led half of exploratory QA; its human-led
|
|
208
|
+
sibling is `/qa-assist` (the human drives, the agent scribes). Everything in
|
|
209
|
+
§§ 1–3 applies unchanged; the deltas are the method choice and the per-surface
|
|
210
|
+
boundary.
|
|
211
|
+
|
|
212
|
+
- **Pick the driving method explicitly at Plan time** — drive (default) vs.
|
|
213
|
+
static — and record it in the ledger. Do not switch methods mid-surface
|
|
214
|
+
without a new Plan note.
|
|
215
|
+
- **Drive is the default.** When the resolved environment carries a
|
|
216
|
+
`signInSeam`, authenticated surfaces — including deployed hosts — are
|
|
217
|
+
**driven** through that seam, not statically deferred.
|
|
218
|
+
- **Static driving is the documented interim**, chosen **only where no seam
|
|
219
|
+
resolves** for the target environment. It walks the surface from source,
|
|
220
|
+
route definitions, and rendered markup rather than a running browser:
|
|
221
|
+
- **Never a silent fallback.** Static is a deliberate Plan-phase decision
|
|
222
|
+
recorded with its reason ("environment: preview, method: static, reason: no
|
|
223
|
+
seam resolves"), not something the agent slips into when the browser MCP
|
|
224
|
+
hiccups.
|
|
225
|
+
- **Interim, not equivalent.** Static driving cannot exercise real
|
|
226
|
+
authorization, routing guards, or runtime console/network signal. Treat its
|
|
227
|
+
coverage as partial and say so in the ledger; a static pass does not close
|
|
228
|
+
the same coverage a driven pass would.
|
|
229
|
+
- **Same read-only invariant** (§ 2) applies identically.
|
|
230
|
+
- **Promote to driving when a seam lands.** Re-run the surface driven rather
|
|
231
|
+
than leaving it permanently static.
|
|
232
|
+
- **A surface that could not be driven is itself a signal.** Where the
|
|
233
|
+
environment resolves no seam, drive the unauthenticated surface or fall back
|
|
234
|
+
to static and record the gap — never a silent skip.
|
|
235
|
+
- **Observe, do not fabricate.** Never script the runtime to manufacture an
|
|
236
|
+
outcome the exploration is meant to discover.
|
|
237
|
+
|
|
238
|
+
## 6. Record onto the Ledger & Triage (Never File Autonomously)
|
|
239
|
+
|
|
240
|
+
Record each finding as a `QaLedgerItem` on the shared session ledger under
|
|
170
241
|
`temp/qa/`, then route the ledger through the shared classify → route →
|
|
171
242
|
disposition → promote core — both stated once in
|
|
172
243
|
[`qa-core.md`](../../../../workflows/helpers/qa-core.md). The harness **MUST
|
|
173
|
-
NOT** create tickets autonomously: findings are promoted through
|
|
174
|
-
after the operator confirms each disposition at the HITL
|
|
175
|
-
is the safety boundary against spurious filing.
|
|
176
|
-
|
|
177
|
-
## 6. Sign-In & Contract Discipline
|
|
178
|
-
|
|
179
|
-
- **Resolve the `qa` contract first.** Before any browser work, resolve the
|
|
180
|
-
contract via `resolveQaContract(config)`. When the block is absent,
|
|
181
|
-
malformed, or missing a required field, the resolver **throws** — relay its
|
|
182
|
-
verbatim message and STOP. There is no auto-detection fallback.
|
|
183
|
-
- **Dev seam only.** Sign in once per persona via the contract's `signInSeam`
|
|
184
|
-
(`kind: 'url'` dev seam or `kind: 'skill'`). **Never** enter real
|
|
185
|
-
credentials. Confirm authenticated state with a `take_snapshot` before
|
|
186
|
-
driving any scenario.
|
|
187
|
-
- **No headless fallback.** The chrome-devtools MCP surface is a host-provided
|
|
188
|
-
runtime dependency. If it is unavailable, degrade with a clear error and stop
|
|
189
|
-
— never fall back to the retired headless BDD runner.
|
|
244
|
+
NOT** create tickets autonomously: findings are promoted through
|
|
245
|
+
`/mandrel-plan` only after the operator confirms each disposition at the HITL
|
|
246
|
+
write gate. That gate is the safety boundary against spurious filing.
|
|
190
247
|
|
|
191
248
|
## 7. Cross-References
|
|
192
249
|
|
|
193
|
-
-
|
|
250
|
+
- Sweep procedure (SSOT): [`qa-run.md`](../../../../workflows/qa-run.md).
|
|
251
|
+
- Exploration procedure (SSOT): [`qa-explore.md`](../../../../workflows/qa-explore.md).
|
|
194
252
|
- Driving rules (one prose home): [`qa-run-scenario.md`](../../../../workflows/helpers/qa-run-scenario.md).
|
|
195
253
|
- Shared QA core (contract/session/redaction/ledger/triage/HITL): [`qa-core.md`](../../../../workflows/helpers/qa-core.md).
|
|
196
254
|
- Console filter module: [`console-allowlist.js`](../../../../scripts/lib/qa/console-allowlist.js).
|
|
197
255
|
- Assertion-tier rules: [`testing-standards.md`](../../../../rules/testing-standards.md).
|
|
198
256
|
- Scenario prose: [`gherkin-authoring`](../gherkin-authoring/SKILL.md).
|
|
199
257
|
- Browser-locator discipline: [`playwright`](../playwright/SKILL.md).
|
|
200
|
-
-
|
|
258
|
+
- Browser instrumentation: [`browser-testing-with-devtools`](../../../core/browser-testing-with-devtools/SKILL.md).
|
|
259
|
+
- Evidence scrubbing / read-only boundary: [`security-baseline.md`](../../../../rules/security-baseline.md).
|
|
@@ -0,0 +1,169 @@
|
|
|
1
|
+
# Audit sweep runbook
|
|
2
|
+
|
|
3
|
+
> **Template generated by Mandrel.** Copy it into your own docs tree (e.g.
|
|
4
|
+
> `docs/audit-sweep-runbook.md`), then localise every bracketed placeholder:
|
|
5
|
+
> the cadence, the lens list, who reviews the ledger PR, and the label
|
|
6
|
+
> conventions your repository actually uses. The steps themselves are the
|
|
7
|
+
> contract — the values around them are yours.
|
|
8
|
+
>
|
|
9
|
+
> The workflow this runbook drives is
|
|
10
|
+
> [`/audit-to-stories`](../../workflows/audit-to-stories.md); the CLI it calls
|
|
11
|
+
> is [`audit-to-stories.js`](../../scripts/audit-to-stories.js). Run the CLI
|
|
12
|
+
> with `--help` for the authoritative flag list.
|
|
13
|
+
|
|
14
|
+
## What this runbook is for
|
|
15
|
+
|
|
16
|
+
A maintenance **sweep** runs the `audit-*` lenses full-scope, folds their
|
|
17
|
+
findings onto the cross-run ledger, and turns what is genuinely new into
|
|
18
|
+
Stories. It is the unattended sibling of an interactive `/audit-to-stories`
|
|
19
|
+
run: no HITL gates, so every judgement call the interactive path asks a human
|
|
20
|
+
has to be settled here instead.
|
|
21
|
+
|
|
22
|
+
| Setting | Value for this repository |
|
|
23
|
+
| --- | --- |
|
|
24
|
+
| Cadence | _e.g. weekly, Sunday 02:00_ |
|
|
25
|
+
| Lenses in scope | _e.g. security, clean-code, quality, dependencies_ |
|
|
26
|
+
| Severity floor | _`delivery.auditToStories.severityFloor`, default `high`_ |
|
|
27
|
+
| Ledger reviewer | _e.g. @your-handle_ |
|
|
28
|
+
| Story triage owner | _e.g. the on-call maintainer_ |
|
|
29
|
+
|
|
30
|
+
## Step 1 — Run the lenses full-scope
|
|
31
|
+
|
|
32
|
+
Run each `audit-*` workflow with **no** `--paths` and no change-set filter, so
|
|
33
|
+
the whole target-set union is audited rather than whatever a recent branch
|
|
34
|
+
happened to touch. Each lens writes its report to
|
|
35
|
+
`temp/audits/audit-<lens>-results.md`.
|
|
36
|
+
|
|
37
|
+
A sweep scoped to a change set is not a sweep: it re-reports the same recent
|
|
38
|
+
files every cycle and never reaches the code nobody has touched in a year,
|
|
39
|
+
which is exactly where audit findings accumulate.
|
|
40
|
+
|
|
41
|
+
## Step 2 — Cross-check the severity tally
|
|
42
|
+
|
|
43
|
+
Every lens report ends its executive summary with a machine-readable line:
|
|
44
|
+
|
|
45
|
+
```text
|
|
46
|
+
Severity tally: Critical 0 / High 3 / Medium 7 / Low 2
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
The CLI re-counts the `### Finding` blocks it actually parsed and compares them
|
|
50
|
+
to that declared tally. A mismatch — or a missing tally line — means the report
|
|
51
|
+
is not trustworthy: a finding was malformed, a severity did not resolve onto the
|
|
52
|
+
canonical scale, or the lens truncated its own output.
|
|
53
|
+
|
|
54
|
+
`--auto` **fails closed** on any such failure. It exits non-zero having opened
|
|
55
|
+
no Issue and written no ledger, and names the offending report in
|
|
56
|
+
`summary.reportFailures[]`. `--allow-missing-tally` is a `--scan` affordance
|
|
57
|
+
only; `--auto` ignores it by design, because an unattended run has no operator
|
|
58
|
+
to read a warning.
|
|
59
|
+
|
|
60
|
+
**When the sweep goes red here, re-run the lens.** Do not reach for
|
|
61
|
+
`--allow-missing-tally` and do not hand-edit the report to make the numbers
|
|
62
|
+
agree — the tally is the only signal that the parse saw what the lens wrote.
|
|
63
|
+
|
|
64
|
+
## Step 3 — Dry-run the first cycles
|
|
65
|
+
|
|
66
|
+
Start every new sweep in report-only mode, and stay there until the tallies
|
|
67
|
+
stop surprising you:
|
|
68
|
+
|
|
69
|
+
```bash
|
|
70
|
+
node .agents/scripts/audit-to-stories.js --auto --dry-run
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
`--dry-run` performs zero GitHub writes and skips the ledger write, printing
|
|
74
|
+
only the run summary. Read `totals.create` before you let the sweep file
|
|
75
|
+
anything: a first full-scope run over an un-audited repository can propose more
|
|
76
|
+
Stories than your team can triage in a quarter. Raise `--severity` (or
|
|
77
|
+
`delivery.auditToStories.severityFloor`) until the create count is a batch you
|
|
78
|
+
would actually take on, then go live.
|
|
79
|
+
|
|
80
|
+
## Step 4 — Go live, and persist the ledger
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
node .agents/scripts/audit-to-stories.js --auto --ledger-commit
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
The cross-run ledger (`baselines/audit-ledger.json`) is **consumer state, not
|
|
87
|
+
scratch output**. It is what lets the next sweep tell a re-detection from a
|
|
88
|
+
fresh finding, and a deliberately-rejected finding from an unseen one. A
|
|
89
|
+
scheduled job normally runs on an ephemeral checkout, so unless the ledger is
|
|
90
|
+
committed back it dies with the clone and every later sweep starts amnesiac —
|
|
91
|
+
re-proposing findings already filed and re-surfacing findings a human already
|
|
92
|
+
rejected.
|
|
93
|
+
|
|
94
|
+
`--ledger-commit` closes that loop. After the run summary has printed, and only
|
|
95
|
+
when the ledger actually changed, it:
|
|
96
|
+
|
|
97
|
+
1. creates `chore/audit-ledger-<YYYY-MM-DD>` from the current HEAD,
|
|
98
|
+
2. commits **only** the ledger file, subject
|
|
99
|
+
`chore(audit): reconcile audit ledger <date>`,
|
|
100
|
+
3. pushes the branch, and
|
|
101
|
+
4. opens a PR against your base branch.
|
|
102
|
+
|
|
103
|
+
**Auto-merge is never requested.** The ledger records machine-derived lifecycle
|
|
104
|
+
state, so a human glance before it lands is the point — nominate that reviewer
|
|
105
|
+
in the table above. Review the PR for entries flipping to `accepted-risk` or
|
|
106
|
+
`regressed`; those two are the ledger telling you something about your backlog,
|
|
107
|
+
not about itself.
|
|
108
|
+
|
|
109
|
+
Any git or `gh` failure in that sequence is fatal and names the step it broke
|
|
110
|
+
on — but it happens _after_ the summary is printed, so a broken remote never
|
|
111
|
+
costs you the sweep's findings.
|
|
112
|
+
|
|
113
|
+
Without the flag, a run whose ledger changed on a checkout that cannot persist
|
|
114
|
+
it — no `origin` remote, or HEAD parked off the base branch — sets
|
|
115
|
+
`ledger.unpersisted: true` in the summary and warns on stderr naming the file.
|
|
116
|
+
Treat that warning as a red sweep: the findings are fine, but the memory is
|
|
117
|
+
about to be thrown away.
|
|
118
|
+
|
|
119
|
+
## Step 5 — Enrich before you deliver
|
|
120
|
+
|
|
121
|
+
`--emit-stories` renders `{ title, body, labels }` payloads from audit findings.
|
|
122
|
+
Those bodies are **audit prose**, not delivery-ready Specs: they describe a
|
|
123
|
+
symptom and a recommendation, not a scoped change with acceptance criteria a
|
|
124
|
+
worker can verify against.
|
|
125
|
+
|
|
126
|
+
Do not point `/mandrel-deliver` at a freshly-filed audit Story. Route it through
|
|
127
|
+
`/mandrel-plan` first — the planning pass is where the finding becomes a
|
|
128
|
+
capability slice with a `## Spec`, real `acceptance[]` items and runnable
|
|
129
|
+
`verify[]` lines. Planning is deliberately not automated here: deciding what a
|
|
130
|
+
finding is worth, and how far the fix should reach, is the judgement the sweep
|
|
131
|
+
exists to surface rather than to make.
|
|
132
|
+
|
|
133
|
+
The wiring pass is the exception — it _is_ mechanical and it is **required**.
|
|
134
|
+
After opening the Issues, replay their numbers through `--wire-edges` so the
|
|
135
|
+
cohort's declared ordering exists as `blocked by #N` footers and native
|
|
136
|
+
`blocked_by` relations. An unwired cohort is genuinely unordered, and
|
|
137
|
+
`/mandrel-deliver` will co-dispatch Stories the edges say must follow one
|
|
138
|
+
another.
|
|
139
|
+
|
|
140
|
+
## Step 6 — Label convention
|
|
141
|
+
|
|
142
|
+
Audit-sourced Stories carry a closed label set, so they can be filtered out of
|
|
143
|
+
(or into) ordinary planning at a glance:
|
|
144
|
+
|
|
145
|
+
| Label | Meaning |
|
|
146
|
+
| --- | --- |
|
|
147
|
+
| `type::story` | Every emitted Story — the sweep never opens Epics or tasks. |
|
|
148
|
+
| `agent::ready` | Filed and available for pickup. |
|
|
149
|
+
| `audit::<lens>` | One per lens represented in the group; a cross-audit merge carries several. |
|
|
150
|
+
| `risk::high` | Added when any finding in the group is Critical. |
|
|
151
|
+
|
|
152
|
+
The lens labels are a **closed taxonomy**: only the canonical `audit::<lens>`
|
|
153
|
+
names are valid, and the filer refuses to emit a label the repository has never
|
|
154
|
+
created. Create them once with `audit-labels-bootstrap.js` before the first live
|
|
155
|
+
sweep — a generated label that does not exist makes every Issue create fail.
|
|
156
|
+
|
|
157
|
+
Do not invent per-finding labels. If you need another axis (a team, a
|
|
158
|
+
component), add it in triage on top of this set rather than teaching the sweep
|
|
159
|
+
to mint labels no taxonomy defines.
|
|
160
|
+
|
|
161
|
+
## Failure triage
|
|
162
|
+
|
|
163
|
+
| Symptom | Cause | Action |
|
|
164
|
+
| --- | --- | --- |
|
|
165
|
+
| Non-zero exit, `summary.reportFailures[]` populated | A lens report's tally is missing or disagrees with the parse | Re-run that lens; never downgrade with `--allow-missing-tally` |
|
|
166
|
+
| `ledger.unpersisted: true` in the summary | No `origin`, or HEAD off the base branch | Re-run with `--ledger-commit`, or commit the ledger by hand |
|
|
167
|
+
| `--ledger-commit failed at step "..."` | git or `gh` failed at the named step | Fix the remote/auth and re-run; the summary above it is still valid |
|
|
168
|
+
| Same findings re-proposed every cycle | The ledger is not being committed | Adopt Step 4 |
|
|
169
|
+
| `totals.create` far larger than the team can absorb | Severity floor too low for a first full-scope run | Raise `--severity` and re-dry-run |
|