mandrel 2.4.0 → 2.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/audit-checklists/accessibility.md +29 -0
- package/.agents/audit-checklists/architecture.md +4 -5
- package/.agents/audit-checklists/clean-code.md +10 -0
- package/.agents/audit-checklists/data-model.md +22 -0
- package/.agents/audit-checklists/dependencies.md +11 -2
- package/.agents/audit-checklists/devops.md +4 -0
- package/.agents/audit-checklists/navigability.md +3 -0
- package/.agents/audit-checklists/performance.md +8 -11
- package/.agents/audit-checklists/privacy.md +3 -4
- package/.agents/audit-checklists/quality.md +2 -0
- package/.agents/audit-checklists/security.md +4 -5
- package/.agents/audit-checklists/seo.md +7 -1
- package/.agents/audit-checklists/sre.md +14 -12
- package/.agents/audit-checklists/ux-ui.md +4 -0
- package/.agents/docs/configuration.md +3 -0
- package/.agents/docs/workflows.md +4 -3
- package/.agents/schemas/agentrc.schema.json +17 -0
- package/.agents/schemas/audit-rules.json +134 -19
- package/.agents/schemas/audit-rules.schema.json +6 -2
- package/.agents/scripts/audit-labels-bootstrap.js +4 -4
- package/.agents/scripts/audit-to-stories.js +244 -19
- package/.agents/scripts/lib/audit-suite/checklist-threading.js +26 -3
- package/.agents/scripts/lib/audit-suite/dispatch-checklist.js +132 -0
- package/.agents/scripts/lib/audit-suite/index.js +1 -0
- package/.agents/scripts/lib/audit-suite/selector.js +290 -14
- package/.agents/scripts/lib/audit-to-stories/audit-lenses.js +2 -1
- package/.agents/scripts/lib/audit-to-stories/build-story-body.js +5 -1
- package/.agents/scripts/lib/audit-to-stories/dedupe-against-github.js +23 -3
- package/.agents/scripts/lib/audit-to-stories/finding-adapter.js +38 -0
- package/.agents/scripts/lib/audit-to-stories/ledger.js +256 -0
- package/.agents/scripts/lib/audit-to-stories/parse-audit-md.js +41 -7
- package/.agents/scripts/lib/audit-to-stories/seed-from-findings.js +20 -2
- package/.agents/scripts/lib/command-header.js +1 -1
- package/.agents/scripts/lib/config-settings-schema-delivery.js +21 -0
- package/.agents/scripts/lib/dynamic-workflow/performance-report-contract.js +5 -3
- package/.agents/scripts/lib/feedback-loop/audit-results-graduator.js +56 -0
- package/.agents/scripts/lib/findings/route-finding.js +108 -10
- package/.agents/scripts/lib/observability/runtime-friction.js +137 -6
- package/.agents/scripts/lib/orchestration/retro-proposals.js +0 -0
- package/.agents/scripts/lib/orchestration/run-epilogue.js +5 -19
- package/.agents/scripts/lib/orchestration/single-story-close/phases/post-land.js +11 -0
- package/.agents/scripts/lib/orchestration/story-close/phases/local-lens-review.js +81 -1
- package/.agents/scripts/lib/orchestration/story-close/phases/review-core.js +1 -0
- package/.agents/scripts/lib/orchestration/story-follow-ups.js +53 -13
- package/.agents/scripts/nav-registry-diff.js +449 -0
- package/.agents/workflows/audit-accessibility.md +243 -0
- package/.agents/workflows/audit-architecture.md +89 -71
- package/.agents/workflows/audit-clean-code.md +87 -53
- package/.agents/workflows/audit-data-model.md +198 -0
- package/.agents/workflows/audit-dependencies.md +143 -28
- package/.agents/workflows/audit-devops.md +109 -18
- package/.agents/workflows/audit-documentation.md +25 -53
- package/.agents/workflows/audit-navigability.md +78 -22
- package/.agents/workflows/audit-performance.md +207 -103
- package/.agents/workflows/audit-privacy.md +51 -13
- package/.agents/workflows/audit-quality.md +71 -61
- package/.agents/workflows/audit-security.md +94 -71
- package/.agents/workflows/audit-seo.md +80 -25
- package/.agents/workflows/audit-sre.md +99 -66
- package/.agents/workflows/audit-to-stories.md +44 -5
- package/.agents/workflows/audit-ux-ui.md +71 -17
- package/.agents/workflows/helpers/audit-dual-path.md +59 -0
- package/.agents/workflows/helpers/audit-self-check.md +70 -0
- package/.agents/workflows/helpers/audit-severity-scale.md +19 -0
- package/.agents/workflows/helpers/deliver-story.md +25 -0
- package/docs/CHANGELOG.md +23 -0
- package/package.json +1 -1
- package/.agents/audit-checklists/lighthouse.md +0 -15
- package/.agents/schemas/audit-results.schema.json +0 -69
- package/.agents/workflows/audit-lighthouse.md +0 -269
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
---
|
|
2
|
-
description: Audit hot paths,
|
|
2
|
+
description: Audit performance by measuring first — profile hot paths, I/O, memory, and payload against the repo's own numbers — and audit interleaving/partial-failure correctness (TOCTOU, unawaited promises, non-atomic writes) as a first-class dimension.
|
|
3
3
|
---
|
|
4
4
|
|
|
5
5
|
# Performance & Bottleneck Audit
|
|
@@ -10,9 +10,23 @@ Performance Engineer & Systems Architect
|
|
|
10
10
|
|
|
11
11
|
## Context & Objective
|
|
12
12
|
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
13
|
+
Find where a system is slow, wasteful, or unsafe under concurrency — and prove
|
|
14
|
+
it with numbers, not opinions. This lens has three standing commitments that
|
|
15
|
+
separate it from a prose read-through:
|
|
16
|
+
|
|
17
|
+
1. **Measure before you judge.** A performance claim that is not backed by a
|
|
18
|
+
profile, a timing, or a byte count is a hypothesis, not a finding. Step 0
|
|
19
|
+
times the repo's own suite and entry points and produces the evidence every
|
|
20
|
+
finding must cite.
|
|
21
|
+
2. **Adapt to the repo profile.** A CLI/tooling repo has no bundle and no Core
|
|
22
|
+
Web Vitals; a frontend app does. Step 1 detects the target profile and
|
|
23
|
+
activates only the dimensions that apply, declaring the rest inapplicable
|
|
24
|
+
rather than fabricating findings for a surface that does not exist.
|
|
25
|
+
3. **Interleaving correctness is a performance concern.** The most expensive
|
|
26
|
+
defects this repo has shipped were not slow loops — they were races
|
|
27
|
+
(check-then-act on a lease, non-atomic checkout mutation under concurrent
|
|
28
|
+
close, shared-cache poisoning). Step 2 treats interleaving & partial-failure
|
|
29
|
+
correctness as a first-class dimension, statically and repo-observably.
|
|
16
30
|
|
|
17
31
|
## Scope (Story / plan-run mode)
|
|
18
32
|
|
|
@@ -37,88 +51,154 @@ before this section existed.
|
|
|
37
51
|
|
|
38
52
|
## Execution strategy (dual-path)
|
|
39
53
|
|
|
40
|
-
This lens runs along one of two execution paths
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
(
|
|
63
|
-
|
|
64
|
-
`
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
>
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
- **
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
54
|
+
This lens runs along one of two execution paths (orchestrated dynamic-workflow
|
|
55
|
+
or sequential single-pass). Both emit the **identical** Step 4 report contract;
|
|
56
|
+
downstream consumers (`audit-to-stories`) are agnostic to which path produced
|
|
57
|
+
it. See [`helpers/audit-dual-path.md`](helpers/audit-dual-path.md) for strategy
|
|
58
|
+
selection, the forcing flags, and the read-only guarantee — read `audit-<lens>`
|
|
59
|
+
there as this lens's name.
|
|
60
|
+
|
|
61
|
+
> **Measurement is non-mutating, not forbidden.** This lens is read-only with
|
|
62
|
+
> respect to source, but it MUST be allowed to *run* measurements. The
|
|
63
|
+
> orchestrated path grants its measurement agents a `Bash` tool restricted to a
|
|
64
|
+
> **non-mutating command allowlist** (profilers, timers, bundle-stat and
|
|
65
|
+
> file-size probes — never a command that writes source, installs, or mutates
|
|
66
|
+
> git/labels). See the allowlist in
|
|
67
|
+
> [`.claude/workflows/audit-performance.workflow.js`](../../.claude/workflows/audit-performance.workflow.js).
|
|
68
|
+
|
|
69
|
+
## Step 0: Measure before you judge (mandatory)
|
|
70
|
+
|
|
71
|
+
Produce evidence first; every finding in Step 4 carries an **Evidence** field
|
|
72
|
+
that cites a repro command and tags itself `measured` or `estimated`. Run the
|
|
73
|
+
measurements that apply to the repo (Step 1 tells you which), preferring the
|
|
74
|
+
repo's own scripts over invented ones.
|
|
75
|
+
|
|
76
|
+
**Timing (all repos).** Time the project's own test suite and any CLI entry
|
|
77
|
+
points. Prefer `hyperfine` for stable multi-run statistics; fall back to
|
|
78
|
+
`/usr/bin/time -v` (or `time`) when it is absent:
|
|
79
|
+
|
|
80
|
+
```bash
|
|
81
|
+
hyperfine --warmup 1 'npm test' # suite wall-clock + variance
|
|
82
|
+
hyperfine --warmup 1 'node bin/<entry>.js --help' # CLI cold-start cost
|
|
83
|
+
/usr/bin/time -v node bin/<entry>.js <args> 2>&1 | tail -n 20 # fallback + RSS
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
**CPU profile (all repos).** Profile a detected entry script with the V8
|
|
87
|
+
sampling profiler and read the hottest self-time frames:
|
|
88
|
+
|
|
89
|
+
```bash
|
|
90
|
+
node --cpu-prof --cpu-prof-dir=temp/audits/cpuprof bin/<entry>.js <args>
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
Then inspect the emitted `.cpuprofile` (the top self-time nodes) for the real
|
|
94
|
+
hot path.
|
|
95
|
+
|
|
96
|
+
**Payload / bundle (web repos only).** Emit build stats and size the shipped
|
|
97
|
+
payload; do not estimate what the bundler will tell you exactly:
|
|
98
|
+
|
|
99
|
+
```bash
|
|
100
|
+
npx vite build --profile # or the repo's own build script
|
|
101
|
+
du -sh dist/ && find dist -name '*.js' -exec wc -c {} + | sort -n | tail
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
**Evidence discipline.** A finding tagged `measured` names the command above
|
|
105
|
+
whose output produced its number. A finding tagged `estimated` (e.g. a
|
|
106
|
+
Big-O argument read off the code without a runnable repro) says so plainly and
|
|
107
|
+
is graded no higher than **Medium** unless a measurement upgrades it. Attach
|
|
108
|
+
the repro command to the finding so the operator can reproduce it in one paste.
|
|
109
|
+
|
|
110
|
+
### Step 0b: Perf baseline artifact + diff-aware trend
|
|
111
|
+
|
|
112
|
+
Write a per-run baseline capturing the Step 0 numbers to
|
|
113
|
+
`{{auditOutputDir}}/perf-baseline.json` (suite time, per-entry cold-start, RSS,
|
|
114
|
+
bundle bytes where applicable, and the hot-frame list). On every run:
|
|
115
|
+
|
|
116
|
+
- **Diff against the previous baseline** when one exists. A metric that
|
|
117
|
+
regressed past the previous run's value is an **automatic High** finding with
|
|
118
|
+
the delta (old → new, absolute and %) as its Evidence.
|
|
119
|
+
- **Suppress unchanged known findings.** A finding whose measured value is
|
|
120
|
+
within noise of the prior baseline is reported as *unchanged* in a trend
|
|
121
|
+
summary line, not re-litigated as a fresh finding. New or regressed metrics
|
|
122
|
+
are what the run surfaces.
|
|
123
|
+
- Record the baseline path and the trend verdict (`first-run` /
|
|
124
|
+
`improved` / `unchanged` / `regressed`) in the Executive Summary.
|
|
125
|
+
|
|
126
|
+
## Step 1: Repo profile & active dimensions
|
|
127
|
+
|
|
128
|
+
Detect the target profile from repo-observable markers, then activate only the
|
|
129
|
+
dimensions that apply. Declare the inapplicable ones explicitly in the report
|
|
130
|
+
(so a reader knows they were considered and ruled out, not forgotten).
|
|
131
|
+
|
|
132
|
+
Detection signals:
|
|
133
|
+
|
|
134
|
+
- A **CLI / tooling / library** repo — a `bin` field or executable entry
|
|
135
|
+
scripts, no framework marker, no frontend directory: bundle and Core Web
|
|
136
|
+
Vitals are **inapplicable**.
|
|
137
|
+
- A **web / frontend** repo — a framework marker (`react`, `vue`, `svelte`,
|
|
138
|
+
`next`, `vite`, `webpack`) and/or a frontend surface (`src/components/**`,
|
|
139
|
+
`*.html`, `*.css`, a `dist/` build): payload/bundle and CWV **apply**.
|
|
140
|
+
- A **service / backend** repo — server entry, route/controller/API surface:
|
|
141
|
+
I/O and interleaving dominate; bundle is **inapplicable**.
|
|
142
|
+
|
|
143
|
+
The interleaving & partial-failure dimension and the CPU / I/O / memory
|
|
144
|
+
dimensions apply to **every** profile. Only payload/bundle (and its re-homed
|
|
145
|
+
CWV material) is web-gated.
|
|
146
|
+
|
|
147
|
+
## Step 2: Analysis dimensions (orthogonal set)
|
|
148
|
+
|
|
149
|
+
The historically overlapping ten dimensions collapse to four orthogonal
|
|
150
|
+
resource dimensions plus one correctness dimension. Audit each *active*
|
|
151
|
+
dimension (per Step 1) against measured evidence from Step 0.
|
|
152
|
+
|
|
153
|
+
- **CPU & algorithmic hot paths:** super-linear complexity on a path that runs
|
|
154
|
+
under load, redundant recomputation, synchronous work blocking the event
|
|
155
|
+
loop. Ground every claim in a `--cpu-prof` hot frame or a timed repro.
|
|
156
|
+
- **I/O & syscall efficiency:** N+1 queries/reads, unbatched network or
|
|
157
|
+
filesystem round-trips, missing caching, oversized payloads crossing a
|
|
158
|
+
boundary, chatty `fs` calls in a loop. Cite the call site and a timing.
|
|
159
|
+
- **Memory & leaks:** unbounded caches/arrays, retained closures, listeners
|
|
160
|
+
never removed, growth across a repeated operation. Cite RSS from Step 0 or a
|
|
161
|
+
heap delta.
|
|
162
|
+
- **Payload & bundle (web only):** heavy or duplicated dependencies, missing
|
|
163
|
+
code-splitting, unoptimized assets, render-blocking resources. Re-homed from
|
|
164
|
+
the retired lighthouse lens: capture a **per-route Core-Web-Vitals score
|
|
165
|
+
baseline** and measure with a **median-of-3** protocol (three runs, report
|
|
166
|
+
the median LCP/CLS/INP/TBT per route) so single-run variance never drives a
|
|
167
|
+
finding. Inapplicable — and omitted — on non-web repos.
|
|
168
|
+
- **Interleaving & partial-failure correctness:** the concurrency dimension.
|
|
169
|
+
Statically, repo-observably, look for: unawaited / floating promises;
|
|
170
|
+
`Promise.all` over independently-failing branches where `allSettled` is
|
|
171
|
+
required; **check-then-act (TOCTOU)** on files, locks, labels, or uniqueness
|
|
172
|
+
constraints; non-atomic read-modify-write; in-place writes where
|
|
173
|
+
temp-file-plus-rename is required for crash-atomicity; missing
|
|
174
|
+
transaction/compensation around a multi-step write; non-idempotent retried
|
|
175
|
+
side effects; and shared-cache poisoning (one run writing state a concurrent
|
|
176
|
+
run reads). **Speculative races are the known failure mode of this
|
|
177
|
+
dimension** — a claimed race with no concrete interleaving and no
|
|
178
|
+
repo-observable shared-state path is a false positive; lean hard on the
|
|
179
|
+
self-cross-check to drop it.
|
|
180
|
+
|
|
181
|
+
## Step 3: Severity rubric (performance-anchored)
|
|
182
|
+
|
|
183
|
+
Grade every finding on the shared
|
|
184
|
+
[`Critical | High | Medium | Low` scale](helpers/audit-severity-scale.md),
|
|
185
|
+
anchored to performance/correctness cost rather than gut feel:
|
|
186
|
+
|
|
187
|
+
- Grade **Critical** for a guaranteed data-loss or corruption path under normal
|
|
188
|
+
concurrency (e.g. a lost-update TOCTOU on persisted state), or a hang/outage
|
|
189
|
+
under expected load.
|
|
190
|
+
- Grade **High** for a measured regression past the previous baseline (delta is
|
|
191
|
+
the Evidence), or a hot-path cost (frequency × per-call cost) on a path proven
|
|
192
|
+
to run under load, or a race with a concrete losing interleaving.
|
|
193
|
+
- Grade **Medium** for a real but bounded cost: an `estimated`-only algorithmic
|
|
194
|
+
concern, a cold-path inefficiency, or a concurrency smell with a plausible but
|
|
195
|
+
unproven interleaving.
|
|
196
|
+
- Grade **Low** for minor or opportunistic issues; fix when nearby.
|
|
197
|
+
|
|
198
|
+
Latency thresholds, when a user-facing route is in scope, follow the CWV bands
|
|
199
|
+
in the payload/bundle dimension (LCP ≤2.5s good / ≤4.0s needs-improvement).
|
|
200
|
+
|
|
201
|
+
## Step 4: Output Requirements
|
|
122
202
|
|
|
123
203
|
Generate and save a highly structured Markdown audit report to
|
|
124
204
|
`{{auditOutputDir}}/audit-performance-results.md`, using the exact template
|
|
@@ -129,22 +209,32 @@ below.
|
|
|
129
209
|
|
|
130
210
|
## Executive Summary
|
|
131
211
|
|
|
132
|
-
[Overview of performance
|
|
212
|
+
[Overview of performance posture vs the Step 0 measurements. State the repo
|
|
213
|
+
profile detected (Step 1) and which dimensions were inapplicable. State the
|
|
214
|
+
baseline trend verdict (first-run / improved / unchanged / regressed) and the
|
|
215
|
+
`perf-baseline.json` path. Close with the self-cross-check `kept k / dropped d`
|
|
216
|
+
line.]
|
|
133
217
|
|
|
134
218
|
## Detailed Findings
|
|
135
219
|
|
|
136
|
-
[For every bottleneck identified, use the following
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
- **
|
|
142
|
-
- **
|
|
143
|
-
|
|
144
|
-
- **
|
|
145
|
-
|
|
220
|
+
[For every bottleneck or correctness defect identified, use the following
|
|
221
|
+
strict structure. Lead each title with the primary file it lives in:]
|
|
222
|
+
|
|
223
|
+
### `path/to/primary-file.ext` — [Short title of the finding]
|
|
224
|
+
|
|
225
|
+
- **Dimension:** [CPU & algorithmic | I/O | Memory & leaks | Payload & bundle | Interleaving & partial-failure]
|
|
226
|
+
- **Impact:** [Critical | High | Medium | Low]
|
|
227
|
+
- **Location:** `path/to/primary-file.ext:line`
|
|
228
|
+
- **Evidence:** [A repro command from Step 0 (or a quoted code path) and a
|
|
229
|
+
`measured` or `estimated` tag — e.g. "`measured`: `hyperfine 'npm test'` →
|
|
230
|
+
regressed 8.1s → 11.4s (+40%) vs perf-baseline.json"]
|
|
231
|
+
- **Current State:** [Technical explanation of where and why the bottleneck or
|
|
232
|
+
race occurs]
|
|
233
|
+
- **Recommendation & Rationale:** [Specific optimization/fix tactic and the
|
|
234
|
+
expected gain or the interleaving it closes]
|
|
235
|
+
- **Acceptance signal:** [the command or observable that proves this finding is remediated — e.g. a benchmark below the target threshold, a re-run of this lens showing the baseline no longer regressed, or a test exercising the losing interleaving]
|
|
146
236
|
- **Agent Prompt:**
|
|
147
|
-
`[A copy-pasteable, highly specific prompt to execute this
|
|
237
|
+
`[A copy-pasteable, highly specific prompt to execute this fix independently]`
|
|
148
238
|
|
|
149
239
|
## Low-Hanging Fruit
|
|
150
240
|
|
|
@@ -153,8 +243,22 @@ below.
|
|
|
153
243
|
|
|
154
244
|
## Constraint
|
|
155
245
|
|
|
156
|
-
This is a **read-only** audit
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
246
|
+
This is a **read-only** audit **with respect to source**: it does not edit,
|
|
247
|
+
create, or delete application code, dependencies, or configuration. It **does**
|
|
248
|
+
run non-mutating measurements (Step 0) and writes exactly two artifacts — the
|
|
249
|
+
report and `perf-baseline.json`. Note: this lens supersedes the retired
|
|
250
|
+
`audit-lighthouse` lens by folding its measured Core-Web-Vitals material (the
|
|
251
|
+
per-route score baseline and median-of-3 protocol) into the web branch of the
|
|
252
|
+
payload/bundle dimension.
|
|
253
|
+
|
|
254
|
+
## Self-cross-check (mandatory — filter false positives before you finalize)
|
|
255
|
+
|
|
256
|
+
Before you write the report artifact from the previous step, run the shared
|
|
257
|
+
adversarial self-cross-check over your Detailed Findings — see
|
|
258
|
+
[`helpers/audit-self-check.md`](helpers/audit-self-check.md). It defines the
|
|
259
|
+
per-finding evidence bar, the exclusion list, and the final re-open-and-drop
|
|
260
|
+
pass whose `kept <k> / dropped <d>` counts you record in the Executive
|
|
261
|
+
Summary, so the sequential single-pass path filters unverified findings just as
|
|
262
|
+
the orchestrated path's adversarial reviewer does. The **interleaving**
|
|
263
|
+
dimension leans on this pass hardest: drop every claimed race that lacks a
|
|
264
|
+
concrete losing interleaving over a repo-observable shared-state path.
|
|
@@ -36,19 +36,41 @@ before this section existed.
|
|
|
36
36
|
proceed with the full codebase-wide scan defined in the remaining
|
|
37
37
|
steps.
|
|
38
38
|
|
|
39
|
-
## Step 1:
|
|
39
|
+
## Step 1: Sink-First Detection
|
|
40
40
|
|
|
41
41
|
> Apply [`helpers/parallel-tooling.md`](helpers/parallel-tooling.md) when batching the scan below — independent reads belong in one turn, long shells run via `run_in_background` + `Monitor`.
|
|
42
42
|
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
43
|
+
A PII leak is a **source → sink** flow: sensitive data reaching an egress point.
|
|
44
|
+
Enumerate the **sinks** first, then trace which ones receive PII. Report only
|
|
45
|
+
**proven flows** — a sink that never touches a PII source is not a finding.
|
|
46
|
+
|
|
47
|
+
1. **Enumerate the sinks.** Run these verbatim `rg` commands (they anchor every
|
|
48
|
+
finding to a real line):
|
|
49
|
+
|
|
50
|
+
```bash
|
|
51
|
+
# Logging sinks
|
|
52
|
+
rg -n "\b(console\.(log|info|warn|error|debug)|logger\.(info|warn|error|debug|log))\s*\(" --glob '!**/*.test.*'
|
|
53
|
+
# Telemetry / analytics sinks
|
|
54
|
+
rg -n "\b(track|capture|analytics|telemetry|reportEvent|Sentry\.(captureException|captureMessage))\s*\(" --glob '!**/*.test.*'
|
|
55
|
+
# Persistence sinks
|
|
56
|
+
rg -n "\b(localStorage|sessionStorage|\.set\(|db\.(insert|update|save)|prisma\.\w+\.(create|update|upsert))\b" --glob '!**/*.test.*'
|
|
57
|
+
# Outbound-HTTP sinks (PII in URLs / bodies / headers)
|
|
58
|
+
rg -n "\b(fetch|axios|http\.request|got|ky)\s*\(" --glob '!**/*.test.*'
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
2. **Secret scan.** Prefer `gitleaks`; fall back to `rg`:
|
|
62
|
+
|
|
63
|
+
```bash
|
|
64
|
+
command -v gitleaks >/dev/null 2>&1 && gitleaks detect --no-banner --redact -v || \
|
|
65
|
+
rg -n -i "(api[_-]?key|secret|password|token|salt)\s*[:=]\s*['\"][^'\"]{8,}" --glob '!**/*.test.*'
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
3. **Trace PII sources to the enumerated sinks.** PII source tokens to follow:
|
|
69
|
+
`email`, `password`, `token`, `phone`, `address`, `ssn`, `dob`, `ip`,
|
|
70
|
+
`fullName`, `firstName`/`lastName`, `creditCard`, `user` object spreads. For
|
|
71
|
+
each sink from step 1, decide whether a PII source reaches it (directly, or
|
|
72
|
+
via a variable/object logged whole). Only a **proven** source→sink flow
|
|
73
|
+
becomes a finding; cite both the source line and the sink line.
|
|
52
74
|
|
|
53
75
|
## Step 2: Analysis Dimensions
|
|
54
76
|
|
|
@@ -70,6 +92,9 @@ Evaluate the codebase against these privacy pillars:
|
|
|
70
92
|
Generate and save a highly structured Markdown audit report to
|
|
71
93
|
`{{auditOutputDir}}/audit-privacy-results.md`, using the exact template below.
|
|
72
94
|
|
|
95
|
+
> Grade every finding's severity on the shared
|
|
96
|
+
> [`Critical | High | Medium | Low` scale](helpers/audit-severity-scale.md).
|
|
97
|
+
|
|
73
98
|
```markdown
|
|
74
99
|
# Privacy & PII Audit Report
|
|
75
100
|
|
|
@@ -85,15 +110,18 @@ Generate and save a highly structured Markdown audit report to
|
|
|
85
110
|
|
|
86
111
|
## Detailed Findings
|
|
87
112
|
|
|
88
|
-
[For every gap identified, use the following strict structure
|
|
113
|
+
[For every gap identified, use the following strict structure. Lead each title
|
|
114
|
+
with the primary file the finding lives in:]
|
|
89
115
|
|
|
90
|
-
### [Short
|
|
116
|
+
### `path/to/primary-file.ext` — [Short title of the issue]
|
|
91
117
|
|
|
92
|
-
- **
|
|
118
|
+
- **Dimension:** [Leaky Log | Insecure Storage | Data Over-collection]
|
|
93
119
|
- **Impact:** [Critical | High | Medium | Low]
|
|
120
|
+
- **Location:** `path/to/primary-file.ext:line`
|
|
94
121
|
- **Current State:** [The specific file/line/module and why it is problematic]
|
|
95
122
|
- **Recommendation & Rationale:** [How to remediate and why it's necessary for
|
|
96
123
|
compliance]
|
|
124
|
+
- **Acceptance signal:** [the command or observable that proves this finding is remediated — e.g. a grep for the leaky log that now returns empty, or a re-run of this lens]
|
|
97
125
|
- **Agent Prompt:**
|
|
98
126
|
`[A copy-pasteable, highly specific prompt to execute this remediation independently]`
|
|
99
127
|
```
|
|
@@ -102,3 +130,13 @@ Generate and save a highly structured Markdown audit report to
|
|
|
102
130
|
|
|
103
131
|
This is a **read-only** audit. Do not modify any code. Focus on identifying
|
|
104
132
|
risks and providing clear remediation steps.
|
|
133
|
+
|
|
134
|
+
## Self-cross-check (mandatory — filter false positives before you finalize)
|
|
135
|
+
|
|
136
|
+
Before you write the report artifact from the previous step, run the shared
|
|
137
|
+
adversarial self-cross-check over your Detailed Findings — see
|
|
138
|
+
[`helpers/audit-self-check.md`](helpers/audit-self-check.md). It defines the
|
|
139
|
+
per-finding evidence bar, the exclusion list, and the final re-open-and-drop
|
|
140
|
+
pass whose `kept <k> / dropped <d>` counts you record in the Executive
|
|
141
|
+
Summary, so the sequential single-pass path filters unverified findings just as
|
|
142
|
+
the orchestrated path's adversarial reviewer does.
|
|
@@ -47,62 +47,52 @@ before this section existed.
|
|
|
47
47
|
|
|
48
48
|
## Execution strategy (dual-path)
|
|
49
49
|
|
|
50
|
-
This lens runs along one of two execution paths
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
- **
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
**
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
> orchestrated subagents run in `acceptEdits` and inherit the session tool
|
|
97
|
-
> allowlist, but the workflow script grants the analysis agents only
|
|
98
|
-
> read/search tools (`Read`, `Grep`, `Glob`) — no write/edit/shell-mutation
|
|
99
|
-
> tools. The single write in an orchestrated run is the final report artifact.
|
|
100
|
-
|
|
101
|
-
## Step 0 - Project Context
|
|
102
|
-
|
|
103
|
-
1. Read the Story under audit — its `## Goal`, inline `acceptance[]` /
|
|
104
|
-
`verify[]`, and folded `## Spec` — to identify the target features.
|
|
105
|
-
2. Identify the target codebase paths for the audit.
|
|
50
|
+
This lens runs along one of two execution paths (orchestrated dynamic-workflow
|
|
51
|
+
or sequential single-pass). Both emit the **identical** Step 3 report contract;
|
|
52
|
+
downstream consumers (`audit-to-stories`) are agnostic to which path produced
|
|
53
|
+
it. See [`helpers/audit-dual-path.md`](helpers/audit-dual-path.md) for strategy
|
|
54
|
+
selection, the forcing flags, and the read-only guarantee — read `audit-<lens>`
|
|
55
|
+
there as this lens's name.
|
|
56
|
+
|
|
57
|
+
## Step 0 - Mode split + tool-first artifact read (mandatory)
|
|
58
|
+
|
|
59
|
+
**Resolve the mode first**, then read the numbers before judging. The two modes
|
|
60
|
+
do not share a Step 0 — a codebase-wide run must not try to read a Story it was
|
|
61
|
+
never given.
|
|
62
|
+
|
|
63
|
+
- **Story-scoped mode** (the `## Scope` block above is populated with a change
|
|
64
|
+
set): read the Story under audit — its `## Goal`, inline `acceptance[]` /
|
|
65
|
+
`verify[]`, and folded `## Spec` — to identify the target features, and scope
|
|
66
|
+
the audit to the change set and its direct dependencies.
|
|
67
|
+
- **Codebase-wide mode** (the `## Scope` block renders the literal
|
|
68
|
+
`{{changedFiles}}` token): there is **no Story** — do not look for one. Audit
|
|
69
|
+
the whole test surface, ranked (below).
|
|
70
|
+
|
|
71
|
+
**Read the committed test-quality artifacts as evidence** (both modes). This
|
|
72
|
+
lens grounds every coverage/quality claim in the metrics the delivery gates
|
|
73
|
+
already compute and commit, rather than prose-scanning the tests:
|
|
74
|
+
|
|
75
|
+
- `baselines/coverage.json` — per-file line/branch coverage. Cite the covered
|
|
76
|
+
ratio for any file you flag as under-tested.
|
|
77
|
+
- `baselines/crap.json` — the CRAP score (complexity × uncoveredness). A high
|
|
78
|
+
CRAP row is a measured "complex **and** under-tested" hotspot — the single
|
|
79
|
+
strongest coverage-gap signal.
|
|
80
|
+
- `baselines/mutation.json` — mutation-testing survivors where present: tests
|
|
81
|
+
that execute code without asserting on it (coverage without confidence).
|
|
82
|
+
|
|
83
|
+
**Rank churn-by-coverage.** Order candidate findings by **churn × coverage
|
|
84
|
+
gap** — frequently-changed files (`git log --format= --name-only -n 200 | sort
|
|
85
|
+
| uniq -c | sort -rn`) that also score low coverage / high CRAP are the
|
|
86
|
+
highest-value gaps. Lead the report with them; cap the Detailed Findings at the
|
|
87
|
+
top hotspots so the output is an actionable batch, not an exhaustive dump.
|
|
88
|
+
|
|
89
|
+
**Anchor the rubric** to [`rules/testing-standards.md`](../rules/testing-standards.md):
|
|
90
|
+
the three-tier pyramid, assertion-placement, and mocking/isolation MUSTs are the
|
|
91
|
+
standard a finding is measured against — cite the rule the test violates rather
|
|
92
|
+
than asserting a bare opinion.
|
|
93
|
+
|
|
94
|
+
Reading these committed artifacts is **read-only** and explicitly permitted (see
|
|
95
|
+
the Constraint) — it is not "running the suite".
|
|
106
96
|
|
|
107
97
|
## Step 1: Context Gathering (Read-Only Scan)
|
|
108
98
|
|
|
@@ -149,6 +139,9 @@ Evaluate the gathered context against the following test quality dimensions:
|
|
|
149
139
|
Generate and save a highly structured Markdown audit report to
|
|
150
140
|
`{{auditOutputDir}}/audit-quality-results.md`, using the exact template below.
|
|
151
141
|
|
|
142
|
+
> Grade every finding's severity on the shared
|
|
143
|
+
> [`Critical | High | Medium | Low` scale](helpers/audit-severity-scale.md).
|
|
144
|
+
|
|
152
145
|
```markdown
|
|
153
146
|
# Testing & Quality Assurance Audit
|
|
154
147
|
|
|
@@ -168,16 +161,19 @@ primary vulnerabilities, coverage gaps, and areas causing developer friction.]
|
|
|
168
161
|
|
|
169
162
|
## Detailed Findings
|
|
170
163
|
|
|
171
|
-
[For every gap identified, use the following strict structure
|
|
164
|
+
[For every gap identified, use the following strict structure. Lead each title
|
|
165
|
+
with the primary file the finding lives in:]
|
|
172
166
|
|
|
173
|
-
### [Short
|
|
167
|
+
### `path/to/primary-file.ext` — [Short title of the issue]
|
|
174
168
|
|
|
175
169
|
- **Category:** [Flakiness | Coverage | Performance | Mocking | Test Plans]
|
|
176
|
-
- **Impact:** [High | Medium | Low]
|
|
170
|
+
- **Impact:** [Critical | High | Medium | Low]
|
|
171
|
+
- **Location:** `path/to/primary-file.ext:line`
|
|
177
172
|
- **Current State:** [How the tests are currently written and why it's
|
|
178
173
|
problematic]
|
|
179
174
|
- **Recommendation & Rationale:** [The specific testing pattern or refactor
|
|
180
175
|
strategy to fix the issue]
|
|
176
|
+
- **Acceptance signal:** [the command or observable that proves this finding is remediated — e.g. the new test failing before / passing after the fix, a coverage re-check, or a re-run of this lens]
|
|
181
177
|
- **Agent Prompt:**
|
|
182
178
|
`[A copy-pasteable, highly specific prompt to execute this fix independently]`
|
|
183
179
|
```
|
|
@@ -186,5 +182,19 @@ primary vulnerabilities, coverage gaps, and areas causing developer friction.]
|
|
|
186
182
|
|
|
187
183
|
## Constraint
|
|
188
184
|
|
|
189
|
-
Do NOT execute any code modifications, edit files, create branches, or run
|
|
190
|
-
test suite
|
|
185
|
+
Do NOT execute any code modifications, edit files, create branches, or **run**
|
|
186
|
+
the test suite (do not invoke `npm test`, a coverage run, or a mutation run —
|
|
187
|
+
those mutate state and cost minutes). Reading the **committed** coverage / CRAP
|
|
188
|
+
/ mutation artifacts under `baselines/` is explicitly permitted and required
|
|
189
|
+
(Step 0): citing an already-computed metric is read-only analysis, not a suite
|
|
190
|
+
run. Output the report and stop.
|
|
191
|
+
|
|
192
|
+
## Self-cross-check (mandatory — filter false positives before you finalize)
|
|
193
|
+
|
|
194
|
+
Before you write the report artifact from the previous step, run the shared
|
|
195
|
+
adversarial self-cross-check over your Detailed Findings — see
|
|
196
|
+
[`helpers/audit-self-check.md`](helpers/audit-self-check.md). It defines the
|
|
197
|
+
per-finding evidence bar, the exclusion list, and the final re-open-and-drop
|
|
198
|
+
pass whose `kept <k> / dropped <d>` counts you record in the Executive
|
|
199
|
+
Summary, so the sequential single-pass path filters unverified findings just as
|
|
200
|
+
the orchestrated path's adversarial reviewer does.
|